Structured files in Storage

Storage uses one tabular contract for CSV, TSV, and XLSX. Declare it when the file’s expected structure is known, regardless of whether the bytes come from a browser upload or rows generated through uploadPart(). The schema is stored on the upload session and reused when Storage builds the file index. It is not inferred from Excel formatting.

Shared schema contract

const schema = {
  schemaPolicy: {
    profile: 'fallback',
    additionalProperties: 'allow',
  },
  sheets: [
    {
      name: 'Products',
      table: {
        columns: [
          { key: 'sku', header: 'SKU', type: 'string' },
          { key: 'price', header: 'Price', type: 'number' },
          { key: 'createdAt', header: 'Created', type: 'date' },
          { key: 'metadata', header: 'Metadata', type: 'object' },
          { key: 'active', header: 'Active', type: 'boolean' },
        ],
      },
    },
  ],
};

XLSX may declare multiple worksheets.
CSV and TSV accept exactly one logical sheet. Its name identifies the tabular schema; the physical file still has no worksheet tabs.
key is the property returned in each row.
header matches the visible source header and defaults to key when omitted.
column order is stable and shared by row generation and indexed reads.

Column types

Type	Result
`string`	Text is preserved as text
`number`	Compatible numeric values become numbers
`boolean`	Compatible boolean values become booleans
`date`	Compatible date values become typed dates
`object`	JSON object or array text is parsed into an object or array

object deliberately excludes JSON scalar values. A value such as {"segment":"A"} or [1,2] is valid; "A", 12, and malformed JSON are not object-column values. Without a declared type, imported tabular values remain strings. XLSX cell styles, number formats, colors, and formatting never declare the logical type.

Choose validation and extra-column handling

The optional schemaPolicy object separates validation from handling extra source columns. It uses the same profile and additionalProperties property names as the JSON validator, while applying them to tabular imports. If omitted, Storage uses { profile: 'fallback', additionalProperties: 'allow' }. With only profile: 'strict', extra columns default to reject. Storage rejects unknown schemaPolicy property names, so a misspelled option cannot silently select a default.

Option	Behavior
`profile: 'fallback'`	Keep imperfect declared data and report missing sheets or columns and invalid typed values
`profile: 'strict'`	Fail structured indexing for missing declared sheets or columns, invalid typed values, or sheet mismatches
`additionalProperties: 'allow'`	Keep extra source columns in structured rows
`additionalProperties: 'strip'`	Omit extra source columns from structured rows while preserving the raw file
`additionalProperties: 'reject'`	Fail structured indexing when an extra source column appears

Use the default fallback profile for user-provided files and integrations where preserving imperfect structured data matters. Fallback behavior is deterministic:

the raw file remains available;
matching declared columns use their declared types;
missing and extra sheets or columns are reported in fileStats.schema.issues;
extra columns are retained as strings with the default additionalProperties: 'allow';
if any non-null value is incompatible with a declared type, the whole column falls back to strings;
incompatible values are never silently replaced with null.

This avoids a mixed column such as [12.5, "invalid"] and keeps the source evidence intact.

const { session, upload } = await storage.createUploadSession({
  name: 'products.xlsx',
  contentTypeHint: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
  uploadMode: 'direct',
  computeStats: 'sync',
  schemaPolicy: { profile: 'fallback', additionalProperties: 'allow' },
  sheets: [{
    name: 'Products',
    table: {
      columns: [
        { key: 'sku', header: 'SKU', type: 'string' },
        { key: 'price', header: 'Price', type: 'number' },
        { key: 'createdAt', header: 'Created', type: 'date' },
        { key: 'metadata', header: 'Metadata', type: 'object' },
      ],
    },
  }],
});

return {
  storageUploadSessionId: session.storageUploadSessionId,
  upload: {
    url: upload.uploadUrl,
    method: upload.method,
    headers: upload.headers,
  },
};

The client uploads the existing workbook with the returned contract. Storage then applies the declared schema while building the XLSX index. The client does not need to re-create or flatten the workbook.

Strict validation

Use schemaPolicy: { profile: 'strict' } when a mismatch must stop structured processing, for example a controlled financial import or a versioned partner feed. Strict validation requires table.columns when the session is created. A missing or additional sheet, missing declared column, or incompatible typed value makes structured indexing fail with STORAGE_TABULAR_SCHEMA_MISMATCH. Extra columns fail by default because strict uses additionalProperties: 'reject'; choose strip to omit them or allow to retain them while keeping the other strict checks. Duplicate declarations are rejected when the session is created. For example, { profile: 'strict', additionalProperties: 'strip' } accepts a file with extra source columns, omits those columns from structured rows, and still rejects missing declared columns, invalid typed values, and sheet mismatches. strip and reject require declared table.columns even with the fallback profile. An already finalized file is preserved for download and diagnosis when later indexing fails. With synchronous CSV validation, malformed raw input fails before publication. A failed upload session can retain temporary bytes; failure does not always mean they were removed. Downstream processing must require successful structured indexing before it consumes rows.

Strict validates the declared tabular contract, not business rules such as “price must be positive” or “SKU must exist.” Apply those rules after reading typed rows.

CSV and TSV dialect

CSV and TSV use the same column schema plus optional dialect settings:

const { session } = await storage.createUploadSession({
  name: 'products.tsv',
  contentTypeHint: 'text/tab-separated-values',
  uploadMode: 'incremental',
  computeStats: 'sync',
  schemaPolicy: { profile: 'fallback', additionalProperties: 'allow' },
  csv: {
    delimiter: '\t',
    recordDelimiter: '\n',
    encoding: 'utf8',
    headerRow: 1,
  },
  sheets: [{
    name: 'Data',
    table: {
      columns: [
        { key: 'sku', header: 'SKU', type: 'string' },
        { key: 'price', header: 'Price', type: 'number' },
      ],
    },
  }],
});

Declare the delimiter and encoding when the source is known. Automatic detection remains available for ordinary imports, but an explicit dialect prevents ambiguity and is reused by indexed reads. Quoting defaults to "; escape also defaults to ". Quoted fields can contain delimiters, escaped quotes and line breaks. Use quote: false only when the source contract treats quotes and backslashes as literal characters. In that dialect, escape is ignored; a backslash does not protect an unquoted delimiter. Native row generation rejects values containing a delimiter or line break when quoting is disabled. The same dialect must be used for indexing and reading. Upload sessions accept it under csv; getFileStats() and record readers accept read-time dialect options directly:

const csv = {
  delimiter: '|',
  recordDelimiter: '\r\n',
  quote: '"',
  escape: '"',
};

await storage.walkFileData(
  storageEntryId,
  csv,
  async (rows) => {
    await processRows(rows);
  },
  { fullScan: true, batchSize: 100_000, readBatchSize: 100_000 },
);

Set encoding when known. Otherwise, native detection reads the source encoding and BOM. A read does not rewrite the file’s encoding or line endings. Empty or whitespace-only source lines outside quoted fields are skipped as records; quoted empty fields and blank lines inside a multiline field remain part of the CSV data. Optional maxRecordBytes is a positive integer limiting the source bytes in one logical CSV record, including multiline quoted fields. The default is 64 MiB. It is independent of file size, batchSize and readBatchSize; raise it explicitly only when the source legitimately contains larger records.

CSV errors stop record processing

Invalid quote syntax rejects CSV indexing with STORAGE_CSV_INVALID. A source record with the wrong number of fields also rejects indexing under the default onError: 'abort' policy; onError: 'skipRow' skips those complete records and reports their locations. This validation happens before schema projection and before the first record-walker callback. Neither schema fallback nor additionalProperties: 'strip' permits dropping or padding fields in a malformed source record. The error contains errorCount, up to 100 location diagnostics, and diagnosticsTruncated. Each diagnostic identifies the parser recordNumber, physical lineStart and lineEnd, startByte, and an error code. Field-count errors can include expected and actual counts. A parser record number can differ from a physical line number because of headers, skipped blank lines and multiline fields. Diagnostics contain no raw rows or cell values and remain available in execution and Job errors. RecordTooLarge aborts parsing as soon as maxRecordBytes is exceeded, including for an unterminated quoted field. In this case, counts and diagnostics describe only the inspected prefix. Correct the source file and retry the intended dialect; do not silently disable quoting to make a malformed standard CSV pass. For raw CSV input with computeStats: 'sync', validation is part of finalization and failure prevents publication. A failed object write through putObject() attempts to clean up its uncommitted object; cleanup failure is logged. Upload-session finalization, including the session used by SFTP import, can retain temporary bytes until explicit abort or later expiry cleanup. Inspect the session state and abort an abandoned session rather than assuming a validation error removed its bytes or changed its status to FAILED. A raw multipart upload may already have consumed its parts before validation fails, so do not blindly retry finalization; upload the corrected source through a verified new or repaired session. SFTP import leaves the remote source in place. When an already finalized file fails later indexing, its Storage bytes remain available to an authorized operator. With asynchronous indexing, require successful synchronization before writing business data.

Records and physical lines

CSV statistics distinguish data records from physical source layout:

Statistic	Meaning
`rowCount`	Logical data records, excluding the header and skipped blank records
`physicalLineCount`	Actual lines, including the header, blank lines and lines inside multiline fields
`blankLineCount`	Physical lines that are empty or contain only whitespace
`endsWithRecordDelimiter`	Whether the source ends with the configured record delimiter
`lineCount`	Existing line-window count, retaining the terminal empty element for compatibility

For id|name\n1|A\n, rowCount is 1, physicalLineCount is 2 and legacy lineCount is 3. The terminating delimiter does not add a third actual physical line. An empty file has zero physical lines. Synchronous raw CSV finalization validates records and counts physical lines in one scan. Native generated row parts capture counts while encoding, and finalization reuses those counts without rereading the generated file just to count lines. A physical line index can still be built lazily when a line-window read is requested. Use record mode for CSV, TSV, XLSX, NDJSON/JSONL, JSON arrays and record-oriented XML. Explicit mode: 'lines' is a separate text-window contract for custom raw-line workflows: it returns strings without parsing fields and preserves headers, blank lines and the terminal empty element. It is not an automatic recovery mode for malformed CSV. For mixed line endings, line windows follow their selected or detected separator; physical CSV metrics count the actual CR, LF and CRLF boundaries. Once synchronization has built statistics for the same source and dialect, record reads reuse them. A normal walker needs no extra getFileStats() call. Use explicit statistics for discovery or validation before separate preparatory work, and reloadStats: true only when recomputation is required. Preserve established processing sizes; file size alone is not a reason to shrink callback batches or split the file into extra transfers.

Handle CSV records with the wrong column count

CSV and TSV record reads default to onError: 'abort'. A record whose field count differs from the source header makes indexing fail before rows are returned or a walker callback runs. Without a header, the first data record establishes the expected width. Schema fallback does not pad short records or discard surplus cells. Choose onError: 'skipRow' when processing the valid records is acceptable and the skipped records will be reviewed. This skips complete records with the wrong field count. Invalid quote syntax and records exceeding maxRecordBytes still abort; the default record-size limit is 64 MiB. Pass the policy to getFileStats(), getFileData(), or walkFileData(). Upload sessions accept csv: { onError: 'skipRow' } for finalization and background indexing. Each reader still defaults to abort, so use skipRow explicitly on every read, including continuation pages. Keep the policy unchanged while following a cursor; restart pagination to change it.

let acceptedRows = 0;

const result = await storage.walkFileData(
  storageEntryId,
  { onError: 'skipRow' },
  async (rows) => {
    acceptedRows += rows.length;
  },
  { fullScan: true, batchSize: 100_000, readBatchSize: 100_000 },
);

return { acceptedRows, csvReport: result.csvReport };

Statistics, pages, walker callback contexts, and the walker result expose csvReport:

skippedRows is the exact number of skipped records in the complete source.
diagnostics contains up to 100 samples with recordNumber, physical lineStart/lineEnd, startByte, and expected/actual field counts. Locations refer to the original file; samples contain no cell values.
diagnosticsTruncated indicates that further diagnostic samples were omitted.

The report describes the whole indexed file even when a read selects only part of it. Do not sum repeated reports across pages or callbacks. Pagination, rowCount, totalRows, and walker counters count accepted records. If every data record is skipped, the result still includes the report and no walker callback runs. The original file remains unchanged. This CSV parser policy is separate from database import onError, which controls validation and database write failures. It does not skip business-rule failures or apply to XLSX, other structured formats, physical-line mode, or byte reads.

Generate typed rows

For generated CSV/XLSX, parts may contain row objects or positional arrays instead of serialized bytes.

const { session } = await storage.createUploadSession({
  name: 'report.xlsx',
  contentTypeHint: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
  uploadMode: 'incremental',
  writeMode: 'direct',
  computeStats: 'sync',
  sheets: [{
    name: 'Report',
    table: {
      header: true,
      autoFilter: true,
      columns: [
        { key: 'productId', header: 'Product ID', type: 'string' },
        { key: 'price', header: 'Price', type: 'number' },
        { key: 'metadata', header: 'Metadata', type: 'object' },
      ],
    },
  }],
});

const sessionId = session.storageUploadSessionId;

await storage.uploadPart(sessionId, {
  rows: [
    { productId: 'P-1', price: 12.5, metadata: { segment: 'A' } },
    { productId: 'P-2', price: 9.99, metadata: ['sale'] },
  ],
});

const result = await storage.finalizeUploadSession(sessionId);
return { storageEntryId: result.entry.storageEntryId };

Choose the structured writeMode independently from uploadMode:

`writeMode`	Use when
`staged`	Independent requests may be retried, replaced, or reordered; sheets or columns can be inferred; XLSX needs sparse cells, formulas, or dynamic worksheets
`direct`	Every CSV/XLSX column and worksheet is declared before the first row; XLSX parts contain rows only

staged is the safer default for ordinary multi-request writers. Structured direct is for a fixed layout and rejects undeclared worksheets or cell-oriented XLSX parts.

Compute timing

The declared schema is honored in every compute mode:

`computeStats`	Structured index availability
`sync`	Ready when finalization completes
`async`	Queued after the file entry is published
`none`	Built lazily by `getFileStats()` or `getFileData()`

CSV, TSV, and XLSX sessions default to sync. Set the mode explicitly when the next workflow has a strict latency or readiness requirement.

Read the typed result

const page = await storage.getFileData(storageEntryId, {
  sheet: 'Products',
  limit: 2_000,
});

return {
  rows: page.rows,
  columns: page.columns,
  hasMore: page.next,
  nextCursor: page.nextCursor,
};

Omit sheet to use the first visible XLSX worksheet. Omit columns to read every source column. The read-time columns option projects, renames, or casts a result; it does not replace the schema declared on the upload session. Use storage.walkFileData() for a complete structured scan without accumulating every row in memory. Record mode accepts CSV/TSV, XLSX, NDJSON/JSONL, a top-level JSON array, and record-oriented XML. It does not walk arbitrary plain text, a top-level JSON object, or binary files as rows; explicit mode: 'lines' handles text windows separately. For XML, pass recordPath; for XLSX, select a worksheet with sheet or use the first visible one. The callback is awaited for each batch; omitting take scans all records.

let total = 0;

await storage.walkFileData(
  storageEntryId,
  { sheet: 'Products' },
  async (rows) => {
    total += rows.length;
  },
  { fullScan: true, batchSize: 100_000, readBatchSize: 100_000 },
);

return { total };

Other structured formats

getFileData() and walkFileData() also support NDJSON/JSONL, JSON arrays, and record-oriented XML. These formats use their native record structure rather than the CSV/TSV/XLSX sheets[].table.columns[] upload schema. XML reads and walks require recordPath.

Next steps

Upload sessions — direct, incremental, and chunked completion rules.
Storage overview — workspaces, access, retention, and lifecycle.
Storage low-code reference — exact reader and writer signatures.

Endpoints

Data

Files

Automation

Collaboration and realtime

Observability

Identity and security

Structured files in Storage

Shared schema contract

Column types

Choose validation and extra-column handling

Strict validation

CSV and TSV dialect

CSV errors stop record processing

Records and physical lines

Handle CSV records with the wrong column count

Generate typed rows

Compute timing

Read the typed result

Other structured formats

Next steps

​Shared schema contract

​Column types

​Choose validation and extra-column handling

​Strict validation

​CSV and TSV dialect

​CSV errors stop record processing

​Records and physical lines

​Handle CSV records with the wrong column count

​Generate typed rows

​Compute timing

​Read the typed result

​Other structured formats

​Next steps

Shared schema contract

Column types

Choose validation and extra-column handling

Strict validation

CSV and TSV dialect

CSV errors stop record processing

Records and physical lines

Handle CSV records with the wrong column count

Generate typed rows

Compute timing

Read the typed result

Other structured formats

Next steps