Skip to content
demuddler

Deliver rows

Read imported rows in batches, choose which rows to include and handle failed saves. Examples show how to send data to your backend.

Set handoff on the custom element and register a complete listener. In React, onComplete turns on handoff for you.

complete does not upload a file or save records. It gives your handler a result for a short time. Your handler reads the batches and sends them where your application needs them.

import "@demuddler/importer";
import type { ImportResult } from "@demuddler/importer";
async function countRows(result: ImportResult): Promise<void> {
let received = 0;
for await (const batch of result.batches({ size: 1000 })) {
received += batch.rows.length;
}
document.querySelector("#import-count")?.replaceChildren(String(received));
}
const importer = document.querySelector("demuddler-importer");
if (importer) {
importer.handoff = true;
importer.addEventListener("complete", ({ detail }) => {
detail.waitUntil(countRows(detail.result));
});
}

Call waitUntil during the event, not later. Put every asynchronous step, including your final server commit, inside the promise that you register. An async event listener alone does not register its work.

In React, return the promise: onComplete={deliver} or onComplete={(result) => deliver(result)}. A callback with braces must return it. onComplete={(result) => { deliver(result); }} finishes too early.

During delivery, nobody can change the data. If the promise rejects, the importer shows an error and lets the person try again. Read the iterator to the end. If you stop early, the importer does not deliver the remaining rows.

After the promise settles, result stops working. Until delivery finishes, do not unmount the component, reset it, or change its runtime or its licence key.

Result member Meaning
importId A random ID for this completed import. Every batch and manifest carries it. When the person selects the finish button again, the new completion has a new ID.
fields Destination field keys in schema order.
files The names of the files that the person read, in the order that the importer combined them.
totalRows Source data rows before exclusions and the plan limit. It is not the number of rows that your backend receives.
batches(options) An async generator of row batches of limited size. Read it with for await.
Batch member Meaning
importId The result’s importId.
deliveryId A random ID for one call of batches(). Together with batchNumber, it identifies the batch, so a server can store a repeated request once.
batchNumber Counts from 1 in each delivery.
checksum The SHA-256 digest of JSON.stringify(rows), in hexadecimal.
manifest On the last batch only: importId, deliveryId, the number of batches and rows, and a checksum of the batch checksums.
fields The keys, in the same order as the values in each row.
rows Arrays of values, not objects. Each value is null, a string, a number or a boolean.
rowNumbers The original data row numbers. They start at 1 and do not count headers. Their order matches rows. After exclusions, the numbers have gaps.
sources One entry for each row, in the order of rows: the position of its file in files, from 0, and its line in that file. For a spreadsheet, the line is the sheet row. Use it to trace a row back to the file that a person sent.
percentMarks Pairs of positions, a zero-based row in the batch and a destination field, for values that used percentage notation. 12.5% stays the number 12.5.
failures Always empty in a batch that you receive. A value that cannot convert stops the delivery before its batch arrives.

For example:

{
"importId": "6f1c2a9e-2d4b-4c1e-9a57-0b8c1d2e3f40",
"deliveryId": "0c9d8e7f-6a5b-4c3d-8e2f-1a0b9c8d7e6f",
"batchNumber": 1,
"checksum": "<SHA-256 of JSON.stringify(rows)>",
"fields": ["email", "seats"],
"rows": [
["alex@example.test", 3],
["sam@example.test", 2]
],
"rowNumbers": [1, 4],
"sources": [
[0, 2],
[0, 5]
],
"percentMarks": [],
"failures": []
}

The last batch of a delivery also has a manifest. An import with no rows still delivers one empty batch, so it has a manifest too:

{
"importId": "6f1c2a9e-2d4b-4c1e-9a57-0b8c1d2e3f40",
"deliveryId": "0c9d8e7f-6a5b-4c3d-8e2f-1a0b9c8d7e6f",
"batches": 3,
"rows": 1204,
"checksum": "<SHA-256 of each batch checksum followed by a newline>"
}

To make objects, pair each row with batch.fields. Do not use the source headers as destination keys. If your backend accepts arrays, keep them, because field names in every row make the request larger.

Dates, very large integers and some decimals arrive as strings. See Destination types and values.

Batch option Default Effect
size 5000 A whole number from 1 to 100,000. It sets the number of source rows in each batch, not the byte size of your HTTP request.
skipInvalid The choice in the Import step, or false Leaves out rows that fail validation.
skipDuplicates The choice in the Import step, or false Leaves out rows that the duplicate scan in the Import step marked, or that your duplicateScan marked. It does not compare rows with your database.
skipExcluded The choice in the Import step, or false Leaves out rows that the person excluded.

If you omit these options, the batches follow the choices in the Import step. Set them only when your application owns that policy. After exclusions, a batch can have fewer rows than size. The iterator continues until the last row.

Filters and column widths change only the display. They do not select export rows or change the output order of the schema. Cell edits, transforms that you choose and schema conversion do change the output values.

Your plan limits the source window. Handle limit or onLimit, and tell your users when the import left out part of the source file. See Plans and limits.

Without handoff, the finish action makes a file. With handoff, your handler receives typed batches. The download formats are csv and xlsx.

CSV writes a header row, and XLSX writes a header in the worksheet. XLSX holds at most 16,384 columns and 1,048,576 worksheet rows, including the header. If the rows pass a format limit, the export fails. It does not leave out the remaining rows.

The export event gives the format and the number of rows in the file. A successful complete handler means only that its registered work finished.

Your backend must authenticate the user and check their permission for the target. It must also check the schema and the values, and limit the size of a request. Validation in the browser gives the person feedback. It does not control access.

A browser error cannot undo a batch that your API accepted. If an import must succeed as a whole, stage the rows and commit them only after the iterator finishes. The example below uses endpoints that you write. They are not Demuddler endpoints:

Endpoint Required behaviour
PUT /api/imports/:id/batches/:delivery/:number Stage a batch under an authenticated owner. Check its checksum. The same delivery and number sent again must not add duplicate rows.
PUT /api/imports/:id/commit Compare the manifest with the staged batches of its delivery: every number from 1 to batches, the row count and the checksum. Then publish those rows in one transaction. A repeated commit must return the result that already exists.
GET /api/imports/:id Return whether an attempt is staged, committed or rejected, so the host can find the result of a lost response.
import type { ImportManifest, ImportResult } from "@demuddler/importer";
export async function deliver(result: ImportResult, importId: string): Promise<void> {
const path = `/api/imports/${encodeURIComponent(importId)}`;
let manifest: ImportManifest | undefined;
async function put(url: string, body: unknown): Promise<void> {
const response = await fetch(url, {
method: "PUT",
credentials: "same-origin",
headers: { "content-type": "application/json" },
body: JSON.stringify(body),
});
if (!response.ok) throw new Error("The rows could not be saved. Try again.");
}
for await (const batch of result.batches({ size: 500 })) {
await put(`${path}/batches/${batch.deliveryId}/${batch.batchNumber}`, batch);
manifest = batch.manifest ?? manifest;
}
await put(`${path}/commit`, { manifest, fields: result.fields });
}

To check a batch on the server, hash its rows as JSON. In JavaScript, JSON.stringify(rows) gives the same text. In another language, hash the rows text as the request carried it. Another JSON writer can write numbers or spaces differently.

The importId argument in the example is your own ID. Make it once for each import that you intend. Keep the same ID when you send a request again. Use it too when the person selects the finish button again with the same data. If the data changes, find the status of the old attempt before you start a new one. A committed ID must never accept different data. A response can be lost after the server commits. If you then make a new ID automatically, the import can happen twice.

Apply the CSRF protection of your application to these endpoints. Give access to staged rows only to their tenant and user. Delete abandoned staged records on the server after a time. The idempotency key is the key that makes a repeated request safe. Do not use a file name, an email address or another customer value as the idempotency key.

The example sends one request at a time. This limits memory and slows the reading to the speed of your server. For wide rows or a small limit on the API body, choose a smaller batch. A row count alone does not make sure that a request fits the byte limit of your server.

Situation What the host must do
A required field has no column Nothing. The person cannot leave Match columns until every required field has a column, so complete does not fire.
A rule fails on a row The importer delivers the row unless the person or skipInvalid leaves it out. Check the row again on your backend.
A value cannot convert Let the iterator reject. Do not commit batches that you staged before.
A later batch or the final count fails Treat the whole attempt as failed. Earlier successful requests can still be in staging.
Your API returns an error Throw from the registered delivery promise. Do not report success.
A response is lost Find the status of the import ID before you start another attempt.
The tab closes Work in the browser stops. Work on the server can continue. When the person returns, find the status of the saved import ID.
The person chooses Cancel before delivery Handle cancel or onCancel, and close or go to another page as your product needs. Cancel does not undo writes on the server.

During delivery, the built-in cancel action is not available. If your host adds its own stop control, stop your own requests and reject the registered promise. A stopped HTTP request does not prove that the server rejected or rolled back the request.