CSV and TSV dialects

CSV and TSV dialect

CSV and TSV use the same column schema plus optional dialect settings:

const { session } = await storage.createUploadSession({
  name: 'products.tsv',
  contentTypeHint: 'text/tab-separated-values',
  uploadMode: 'incremental',
  computeStats: 'sync',
  schemaPolicy: { profile: 'fallback', additionalProperties: 'allow' },
  csv: {
    delimiter: '\t',
    recordDelimiter: '\n',
    encoding: 'utf8',
    headerRow: 1,
  },
  sheets: [
    {
      name: 'Data',
      table: {
        columns: [
          { key: 'sku', header: 'SKU', type: 'string' },
          { key: 'price', header: 'Price', type: 'number' },
        ],
      },
    },
  ],
});

Declare the delimiter and encoding when the source is known. Automatic detection remains available for ordinary imports, but an explicit dialect prevents ambiguity and is reused by indexed reads. Quoting defaults to "; escape also defaults to ". Quoted fields can contain delimiters, escaped quotes and line breaks. Use quote: false only when the source contract treats quotes and backslashes as literal characters. In that dialect, escape is ignored; a backslash does not protect an unquoted delimiter. Native row generation rejects values containing a delimiter or line break when quoting is disabled. The same dialect must be used for indexing and reading. Upload sessions accept it under csv; getFileStats() and record readers accept read-time dialect options directly:

const csv = {
  delimiter: '|',
  recordDelimiter: '\r\n',
  quote: '"',
  escape: '"',
};

await storage.walkFileData(
  storageEntryId,
  csv,
  async (rows) => {
    await processRows(rows);
  },
  { fullScan: true, batchSize: 100_000, readBatchSize: 100_000 },
);

Set encoding when known. Otherwise, native detection reads the source encoding and BOM. A read does not rewrite the file’s encoding or line endings. Empty or whitespace-only source lines outside quoted fields are skipped as records; quoted empty fields and blank lines inside a multiline field remain part of the CSV data. Optional maxRecordBytes is a positive integer limiting the source bytes in one logical CSV record, including multiline quoted fields. The default is 64 MiB. It is independent of file size, batchSize and readBatchSize; raise it explicitly only when the source legitimately contains larger records.

CSV errors stop record processing

Invalid quote syntax rejects CSV indexing with STORAGE_CSV_INVALID. A source record with the wrong number of fields also rejects indexing under the default onError: 'abort' policy; onError: 'skipRow' skips those complete records and reports their locations. This validation happens before schema projection and before the first record-walker callback. Neither schema fallback nor additionalProperties: 'strip' permits dropping or padding fields in a malformed source record. The error contains errorCount, up to 100 location diagnostics, and diagnosticsTruncated. Each diagnostic identifies the parser recordNumber, physical lineStart and lineEnd, startByte, and an error code. Field-count errors can include expected and actual counts. A parser record number can differ from a physical line number because of headers, skipped blank lines and multiline fields. Diagnostics contain no raw rows or cell values and remain available in execution and Job errors. RecordTooLarge aborts parsing as soon as maxRecordBytes is exceeded, including for an unterminated quoted field. In this case, counts and diagnostics describe only the inspected prefix. Correct the source file and retry the intended dialect; do not silently disable quoting to make a malformed standard CSV pass. For raw CSV input with computeStats: 'sync', validation is part of finalization and failure prevents publication. A failed object write through putObject() attempts to clean up its uncommitted object; cleanup failure is logged. Upload-session finalization, including the session used by SFTP import, can retain temporary bytes until explicit abort or later expiry cleanup. Inspect the session state and abort an abandoned session rather than assuming a validation error removed its bytes or changed its status to FAILED. A raw multipart upload may already have consumed its parts before validation fails, so do not blindly retry finalization; upload the corrected source through a verified new or repaired session. SFTP import leaves the remote source in place. When an already finalized file fails later indexing, its Storage bytes remain available to an authorized operator. With asynchronous indexing, require successful synchronization before writing business data.

Records and physical lines

CSV statistics distinguish data records from physical source layout:

Statistic	Meaning
`rowCount`	Logical data records, excluding the header and skipped blank records
`physicalLineCount`	Actual lines, including the header, blank lines and lines inside multiline fields
`blankLineCount`	Physical lines that are empty or contain only whitespace
`endsWithRecordDelimiter`	Whether the source ends with the configured record delimiter
`lineCount`	Existing line-window count, retaining the terminal empty element for compatibility

For id|name\n1|A\n, rowCount is 1, physicalLineCount is 2 and legacy lineCount is 3. The terminating delimiter does not add a third actual physical line. An empty file has zero physical lines. Synchronous raw CSV finalization validates records and counts physical lines in one scan. Native generated row parts capture counts while encoding, and finalization reuses those counts without rereading the generated file just to count lines. A physical line index can still be built lazily when a line-window read is requested. Use record mode for CSV, TSV, XLSX, NDJSON/JSONL, JSON arrays and record-oriented XML. Explicit mode: 'lines' is a separate text-window contract for custom raw-line workflows: it returns strings without parsing fields and preserves headers, blank lines and the terminal empty element. It is not an automatic recovery mode for malformed CSV. For mixed line endings, line windows follow their selected or detected separator; physical CSV metrics count the actual CR, LF and CRLF boundaries. Once synchronization has built statistics for the same source and dialect, record reads reuse them. A normal walker needs no extra getFileStats() call. Use explicit statistics for discovery or validation before separate preparatory work, and reloadStats: true only when recomputation is required. Preserve established processing sizes; file size alone is not a reason to shrink callback batches or split the file into extra transfers.

Handle CSV records with the wrong column count

CSV and TSV record reads default to onError: 'abort'. A record whose field count differs from the source header makes indexing fail before rows are returned or a walker callback runs. Without a header, the first data record establishes the expected width. Schema fallback does not pad short records or discard surplus cells. Choose onError: 'skipRow' when processing the valid records is acceptable and the skipped records will be reviewed. This skips complete records with the wrong field count. Invalid quote syntax and records exceeding maxRecordBytes still abort; the default record-size limit is 64 MiB. Pass the policy to getFileStats(), getFileData(), or walkFileData(). Upload sessions accept csv: { onError: 'skipRow' } for finalization and background indexing. Each reader still defaults to abort, so use skipRow explicitly on every read, including continuation pages. Keep the policy unchanged while following a cursor; restart pagination to change it.

let acceptedRows = 0;

const result = await storage.walkFileData(
  storageEntryId,
  { onError: 'skipRow' },
  async (rows) => {
    acceptedRows += rows.length;
  },
  { fullScan: true, batchSize: 100_000, readBatchSize: 100_000 },
);

return { acceptedRows, csvReport: result.csvReport };

Statistics, pages, walker callback contexts, and the walker result expose csvReport:

skippedRows is the exact number of skipped records in the complete source.
diagnostics contains up to 100 samples with recordNumber, physical lineStart/lineEnd, startByte, and expected/actual field counts. Locations refer to the original file; samples contain no cell values.
diagnosticsTruncated indicates that further diagnostic samples were omitted.

The report describes the whole indexed file even when a read selects only part of it. Do not sum repeated reports across pages or callbacks. Pagination, rowCount, totalRows, and walker counters count accepted records. If every data record is skipped, the result still includes the report and no walker callback runs. The original file remains unchanged. This CSV parser policy is separate from database import onError, which controls validation and database write failures. It does not skip business-rule failures or apply to XLSX, other structured formats, physical-line mode, or byte reads. Continue with storage structured files.

Start here

Build and integrate

Operate

AI and Agents

Help

CSV and TSV dialects