Skip to content
demuddler

Schema

Define schema fields, data types and validation rules. Map file columns to your fields and control how the importer converts values.

A schema describes the rows that your application accepts. Set it as a JavaScript property or a React prop. A schema HTML attribute does not work.

import type { ImporterSchema } from "@demuddler/importer";
export const customerSchema: ImporterSchema = {
fields: [
{
key: "email",
label: "Email address",
description: "The customer's contact address.",
aliases: ["Email", "Contact email"],
dtype: "text",
rules: [{ type: "required" }, { type: "format", format: "email" }],
},
{
key: "seats",
label: "Seats",
dtype: "int64",
rules: [{ type: "required" }, { type: "min", value: 1 }],
},
{ key: "joined", label: "Join date", dtype: "date" },
{ key: "active", label: "Active", dtype: "bool" },
],
};
Field Required Meaning
key Yes The name of the field in your data. Each key must be different. Delivered rows use the keys in schema order.
label No The name that the user sees.
description No A short text that shows under the field name in the field picker. Its words help the importer suggest a column, but they never match a column alone.
aliases No Other column names that match this field.
dtype No The type of the delivered value. If your backend needs a type, set it.
rules No The checks on the values. Without it, the field has no rules.

Users can change the column matches and the values. They cannot change your fields or rules. If you do not set schema, the importer makes the fields from the file, and the user can edit them and their rules.

The importer does not deliver a column that matches no field. If you need that column, add a field for it. An optional field with no column delivers null. A required field with no column blocks completion.

dtype Delivered value Notes
text String or null Use it for identifiers and for values whose spelling must not change. If leading zeros matter, also read the column as text.
int64 Number, integer string or null An integer outside the safe range of a JavaScript number is a string, so that it keeps all its digits. Check the type before you call Number().
float64 Number, decimal string or null A decimal that a JavaScript number cannot hold exactly, such as 12345678901234567.89, is a string with every digit. Check the type before you call Number().
bool Boolean or null The text values true, false, yes, no, y and n are accepted in any case, and 1 and 0.
date ISO date string, ISO timestamp string or null A date has the form YYYY-MM-DD. A timestamp keeps its time. The values are not JavaScript Date objects.

The file settings control how the importer reads a column. The dtype controls the delivered value. If the importer read a column as a number, dtype: "text" cannot restore leading zeros or formatting.

An empty value is null, not an empty string or zero. The importer delivers a percentage such as 12.5% as 12.5. The batch marks the cells that used a percent sign.

If a selected value does not convert to its type, delivery and file export stop. The importer never replaces such a value with null. Correct the value or the type, or leave the row out.

Each rule is an object in the rules array of a field.

Rule Example Behaviour
Required { type: "required" } Refuses empty values. A required field with no column blocks completion.
Text pattern { type: "regex", pattern: "^[A-Z]{2}$" } Tests each value that is not empty. The pattern can match any part of the value. To match the whole value, start with ^ and end with $. A pattern that is not valid makes the rule unavailable.
Minimum text length { type: "min_len", len: 2 } Counts characters, not bytes.
Maximum text length { type: "max_len", len: 80 } Counts characters, not bytes.
Minimum number { type: "min", value: 0 } The lowest number that passes. The rule reports values that are not numbers as unchecked.
Maximum number { type: "max", value: 100 } The highest number that passes. The rule reports values that are not numbers as unchecked.
Allowed values { type: "one_of", values: ["active", "trial"] } Ignores case. It does not change the value to the spelling in the list.
Unique values { type: "unique", ignore_case?: boolean } Finds repeated values in this file, not in your database. Spaces at the ends of a value do not count. Case counts unless ignore_case is true. Use ignore_case for email addresses.
Compare fields { type: "compare", other: "start", op: "ge" } Compares this value with the value of another field in the same row. The operators are lt, le, gt and ge. It compares numbers and dates.
Recognised format { type: "format", format: "email" } The formats are email, url, phone and country. They check the form of a value, not whether an address or a number exists.
Conditionally required { type: "required_if", other: "plan", equals: "paid" } Requires a value when the other field has the given value. Case and spaces at the ends do not count.
Validation hook { type: "custom", hook: "known-customer" } Runs the validation hook with this name on your page.

Only required makes a field required. A format, range or pattern rule passes an empty value.

other is the key of another field, not a column name. If that field does not exist or has no column, the rule is unavailable.

A pattern is a string, not a JavaScript RegExp. Do not add / at the ends. In JavaScript, a pattern for digits is "^\\d+$". The importer does not support look-around or backreferences.

The importer refuses a whole schema that breaks one of these rules. It does not drop the bad part. An unknown key in a field or a rule is an error. So are an unknown dtype, rule type, format or op, and a repeated field key. A key, an alias, a pattern and a hook must not be empty.

Item Limit
Fields in a schema 1,000
Characters in a key, a label or an other 200
Characters in a description 1,000
Aliases of a field 50
Rules of a field 50
Characters in a pattern 1,000
Values in a one_of rule 1,000
Characters in an alias, a one_of value or an equals value 1,000
Characters in a hook 100
len of min_len and max_len A whole number from 0 to 4,294,967,295

min and max take a finite number. Refused settings shows what your page sees when the importer refuses a schema.

A rule failure does not remove a row from delivery. The user can correct the row or leave it out. To leave out rows that break a rule, your handler can pass skipInvalid: true to batches().

A rule that is unavailable did not pass. It did not run. A finished import does not prove that every row meets your business rules. Your server must check the rows that it accepts. See Deliver rows.

Mapping runs in the browser. It sends no column name to a server or a model. The importer matches a column to a field in this order:

  1. A match that the user confirmed before, on a page of the same origin with the same field keys
  2. The key or label of the field. Case, spaces, punctuation and accents do not count, so Prénom matches prenom.
  3. One of the aliases of the field
  4. A built-in list of common column names
  5. Similar spelling and shared words

The built-in list covers names, email, phone, postal codes, companies, addresses, city, region, country, date of birth, amount, quantity, SKU and identifiers. It also has common names in Spanish, French, German, Portuguese, Italian, Dutch, Hindi, Japanese and Chinese. For example, Surname matches a last_name field, and E-Mail-Adresse matches an email field. The whole column name must match, so Billing Address does not match an address field. A name with two common meanings, such as nombre, is not in the list.

The importer also matches a column by its values, for example a column of email addresses. It applies a match without the user only when the match is strong and no other field is close. The user reviews the other matches. Add aliases for names that are special to your product. Aliases rank above the built-in list.

When the user adds a file, its columns match by name. A different column order does not move values to a different field.

When a user finishes an import or downloads a file, the importer remembers which column name went to which field key. The next file on a page of the same origin, such as https://app.example.com, with the same field keys, starts with those matches. The user can still change them.

The importer keeps these pairs in the browser storage of the runtime, for each origin and set of field keys. The pairs contain no cell values, and they never leave the browser. A pair expires after 180 days without a new confirmation. The importer keeps at most 200 pairs for each origin and set of field keys, and at most 50 sets in one browser.

If the browser blocks storage, the importer remembers nothing. It does not remember matches for a schema that it made from the file. A change to the field keys starts with no remembered matches.

Existing JSON Schema, Zod or OpenAPI definitions

Section titled “Existing JSON Schema, Zod or OpenAPI definitions”

The importer accepts only a ImporterSchema. It cannot read a JSON Schema, an OpenAPI document or a Zod validator.

Write a field list from your existing definition, with every rule that you need. Then check the delivered rows again on your backend with your existing validator. Nested objects, functions and database checks do not convert to rules. If you write a converter, make it report an error for each part that it cannot convert, so that you lose no rule.