Schema
Define schema fields, data types and validation rules. Map file columns to your fields and control how the importer converts values.
A schema describes the rows that your application accepts. Set it as a
JavaScript property or a React prop. A schema HTML attribute does not work.
import type { ImporterSchema } from "@demuddler/importer";
export const customerSchema: ImporterSchema = { fields: [ { key: "email", label: "Email address", description: "The customer's contact address.", aliases: ["Email", "Contact email"], dtype: "text", rules: [{ type: "required" }, { type: "format", format: "email" }], }, { key: "seats", label: "Seats", dtype: "int64", rules: [{ type: "required" }, { type: "min", value: 1 }], }, { key: "joined", label: "Join date", dtype: "date" }, { key: "active", label: "Active", dtype: "bool" }, ],};Field reference
Section titled “Field reference”| Field | Required | Meaning |
|---|---|---|
key |
Yes | The name of the field in your data. Each key must be different. Delivered rows use the keys in schema order. |
label |
No | The name that the user sees. |
description |
No | A short text that shows under the field name in the field picker. Its words help the importer suggest a column, but they never match a column alone. |
aliases |
No | Other column names that match this field. |
dtype |
No | The type of the delivered value. If your backend needs a type, set it. |
rules |
No | The checks on the values. Without it, the field has no rules. |
Users can change the column matches and the values. They cannot change your
fields or rules. If you do not set schema, the importer makes the fields from
the file, and the user can edit them and
their rules.
The importer does not deliver a column that matches no field. If you need that column, add a
field for it. An optional field with no column delivers null. A required
field with no column blocks completion.
Destination types and values
Section titled “Destination types and values”dtype |
Delivered value | Notes |
|---|---|---|
text |
String or null |
Use it for identifiers and for values whose spelling must not change. If leading zeros matter, also read the column as text. |
int64 |
Number, integer string or null |
An integer outside the safe range of a JavaScript number is a string, so that it keeps all its digits. Check the type before you call Number(). |
float64 |
Number, decimal string or null |
A decimal that a JavaScript number cannot hold exactly, such as 12345678901234567.89, is a string with every digit. Check the type before you call Number(). |
bool |
Boolean or null |
The text values true, false, yes, no, y and n are accepted in any case, and 1 and 0. |
date |
ISO date string, ISO timestamp string or null |
A date has the form YYYY-MM-DD. A timestamp keeps its time. The values are not JavaScript Date objects. |
The file settings control how the
importer reads a column. The dtype controls the delivered value. If the
importer read a column as a number, dtype: "text" cannot restore leading zeros
or formatting.
An empty value is null, not an empty string or zero. The importer delivers a
percentage such as 12.5% as 12.5. The batch marks the cells
that used a percent sign.
If a selected value does not convert to its type, delivery and file export stop.
The importer never replaces such a value with null. Correct the value or the
type, or leave the row out.
Rule reference
Section titled “Rule reference”Each rule is an object in the rules array of a field.
| Rule | Example | Behaviour |
|---|---|---|
| Required | { type: "required" } |
Refuses empty values. A required field with no column blocks completion. |
| Text pattern | { type: "regex", pattern: "^[A-Z]{2}$" } |
Tests each value that is not empty. The pattern can match any part of the value. To match the whole value, start with ^ and end with $. A pattern that is not valid makes the rule unavailable. |
| Minimum text length | { type: "min_len", len: 2 } |
Counts characters, not bytes. |
| Maximum text length | { type: "max_len", len: 80 } |
Counts characters, not bytes. |
| Minimum number | { type: "min", value: 0 } |
The lowest number that passes. The rule reports values that are not numbers as unchecked. |
| Maximum number | { type: "max", value: 100 } |
The highest number that passes. The rule reports values that are not numbers as unchecked. |
| Allowed values | { type: "one_of", values: ["active", "trial"] } |
Ignores case. It does not change the value to the spelling in the list. |
| Unique values | { type: "unique", ignore_case?: boolean } |
Finds repeated values in this file, not in your database. Spaces at the ends of a value do not count. Case counts unless ignore_case is true. Use ignore_case for email addresses. |
| Compare fields | { type: "compare", other: "start", op: "ge" } |
Compares this value with the value of another field in the same row. The operators are lt, le, gt and ge. It compares numbers and dates. |
| Recognised format | { type: "format", format: "email" } |
The formats are email, url, phone and country. They check the form of a value, not whether an address or a number exists. |
| Conditionally required | { type: "required_if", other: "plan", equals: "paid" } |
Requires a value when the other field has the given value. Case and spaces at the ends do not count. |
| Validation hook | { type: "custom", hook: "known-customer" } |
Runs the validation hook with this name on your page. |
Only required makes a field required. A format, range or pattern rule passes
an empty value.
other is the key of another field, not a column name. If that field does not
exist or has no column, the rule is unavailable.
A pattern is a string, not a JavaScript RegExp. Do not add / at the ends. In
JavaScript, a pattern for digits is "^\\d+$". The importer does not support
look-around or backreferences.
Limits and refused schemas
Section titled “Limits and refused schemas”The importer refuses a whole schema that breaks one of these rules. It does not
drop the bad part. An unknown key in a field or a rule is an error. So are an
unknown dtype, rule type, format or op, and a repeated field key. A
key, an alias, a pattern and a hook must not be empty.
| Item | Limit |
|---|---|
| Fields in a schema | 1,000 |
Characters in a key, a label or an other |
200 |
Characters in a description |
1,000 |
| Aliases of a field | 50 |
| Rules of a field | 50 |
Characters in a pattern |
1,000 |
Values in a one_of rule |
1,000 |
Characters in an alias, a one_of value or an equals value |
1,000 |
Characters in a hook |
100 |
len of min_len and max_len |
A whole number from 0 to 4,294,967,295 |
min and max take a finite number.
Refused settings shows what your
page sees when the importer refuses a schema.
Validation does not silently discard rows
Section titled “Validation does not silently discard rows”A rule failure does not remove a row from delivery. The user can correct the
row or leave it out. To leave out rows that break a rule, your handler can pass
skipInvalid: true to batches().
A rule that is unavailable did not pass. It did not run. A finished import does not prove that every row meets your business rules. Your server must check the rows that it accepts. See Deliver rows.
Mapping
Section titled “Mapping”Mapping runs in the browser. It sends no column name to a server or a model. The importer matches a column to a field in this order:
- A match that the user confirmed before, on a page of the same origin with the same field keys
- The
keyorlabelof the field. Case, spaces, punctuation and accents do not count, soPrénommatchesprenom. - One of the
aliasesof the field - A built-in list of common column names
- Similar spelling and shared words
The built-in list covers names, email, phone, postal codes, companies,
addresses, city, region, country, date of birth, amount, quantity, SKU and
identifiers. It also has common names in Spanish, French, German, Portuguese,
Italian, Dutch, Hindi, Japanese and Chinese. For example, Surname matches a
last_name field, and E-Mail-Adresse matches an email field. The whole
column name must match, so Billing Address does not match an address field.
A name with two common meanings, such as nombre, is not in the list.
The importer also matches a column by its values, for example a column of email
addresses. It applies a match without the user only when the match is strong and
no other field is close. The user reviews the other matches. Add aliases for
names that are special to your product. Aliases rank above the built-in list.
When the user adds a file, its columns match by name. A different column order does not move values to a different field.
Remembered mappings
Section titled “Remembered mappings”When a user finishes an import or downloads a file, the importer remembers
which column name went to which field key. The next file on a page of the same
origin, such as https://app.example.com, with the same field keys, starts
with those matches. The user can still change them.
The importer keeps these pairs in the browser storage of the runtime, for each origin and set of field keys. The pairs contain no cell values, and they never leave the browser. A pair expires after 180 days without a new confirmation. The importer keeps at most 200 pairs for each origin and set of field keys, and at most 50 sets in one browser.
If the browser blocks storage, the importer remembers nothing. It does not remember matches for a schema that it made from the file. A change to the field keys starts with no remembered matches.
Existing JSON Schema, Zod or OpenAPI definitions
Section titled “Existing JSON Schema, Zod or OpenAPI definitions”The importer accepts only a ImporterSchema. It cannot read a JSON Schema, an
OpenAPI document or a Zod validator.
Write a field list from your existing definition, with every rule that you need. Then check the delivered rows again on your backend with your existing validator. Nested objects, functions and database checks do not convert to rules. If you write a converter, make it report an error for each part that it cannot convert, so that you lose no rule.