> ## Documentation Index
> Fetch the complete documentation index at: https://docs.revoengine.com/llms.txt
> Use this file to discover all available pages before exploring further.

# CSV and TSV dialects

> Configure delimiters and understand record and column errors.

## CSV and TSV dialect

CSV and TSV use the same column schema plus optional dialect settings:

```js theme={null}
const { session } = await storage.createUploadSession({
  name: 'products.tsv',
  contentTypeHint: 'text/tab-separated-values',
  uploadMode: 'incremental',
  computeStats: 'sync',
  schemaPolicy: { profile: 'fallback', additionalProperties: 'allow' },
  csv: {
    delimiter: '\t',
    recordDelimiter: '\n',
    encoding: 'utf8',
    headerRow: 1,
  },
  sheets: [
    {
      name: 'Data',
      table: {
        columns: [
          { key: 'sku', header: 'SKU', type: 'string' },
          { key: 'price', header: 'Price', type: 'number' },
        ],
      },
    },
  ],
});
```

Declare the delimiter and encoding when the source is known. Automatic detection remains available for ordinary imports, but an explicit dialect prevents ambiguity and is reused by indexed reads.

Quoting defaults to `"`; `escape` also defaults to `"`. Quoted fields can contain delimiters, escaped quotes and line breaks. Use `quote: false` only when the source contract treats quotes and backslashes as literal characters. In that dialect, `escape` is ignored; a backslash does not protect an unquoted delimiter. Native row generation rejects values containing a delimiter or line break when quoting is disabled.

The same dialect must be used for indexing and reading. Upload sessions accept it under `csv`; `getFileStats()` and record readers accept read-time dialect options directly:

```js theme={null}
const csv = {
  delimiter: '|',
  recordDelimiter: '\r\n',
  quote: '"',
  escape: '"',
};

await storage.walkFileData(
  storageEntryId,
  csv,
  async (rows) => {
    await processRows(rows);
  },
  { fullScan: true, batchSize: 100_000, readBatchSize: 100_000 },
);
```

Set `encoding` when known. Otherwise, native detection reads the source encoding and BOM. A read does not rewrite the file's encoding or line endings. Empty or whitespace-only source lines outside quoted fields are skipped as records; quoted empty fields and blank lines inside a multiline field remain part of the CSV data.

Optional `maxRecordBytes` is a positive integer limiting the source bytes in one logical CSV record, including multiline quoted fields. The default is 64 MiB. It is independent of file size, `batchSize` and `readBatchSize`; raise it explicitly only when the source legitimately contains larger records.

## CSV errors stop record processing

Invalid quote syntax rejects CSV indexing with `STORAGE_CSV_INVALID`. A source record with the wrong number of fields also rejects indexing under the default `onError: 'abort'` policy; `onError: 'skipRow'` skips those complete records and reports their locations. This validation happens before schema projection and before the first record-walker callback. Neither schema fallback nor `additionalProperties: 'strip'` permits dropping or padding fields in a malformed source record.

The error contains `errorCount`, up to 100 location diagnostics, and `diagnosticsTruncated`. Each diagnostic identifies the parser `recordNumber`, physical `lineStart` and `lineEnd`, `startByte`, and an error code. Field-count errors can include expected and actual counts. A parser record number can differ from a physical line number because of headers, skipped blank lines and multiline fields. Diagnostics contain no raw rows or cell values and remain available in execution and Job errors.

`RecordTooLarge` aborts parsing as soon as `maxRecordBytes` is exceeded, including for an unterminated quoted field. In this case, counts and diagnostics describe only the inspected prefix. Correct the source file and retry the intended dialect; do not silently disable quoting to make a malformed standard CSV pass.

For raw CSV input with `computeStats: 'sync'`, validation is part of finalization and failure prevents publication. A failed object write through `putObject()` attempts to clean up its uncommitted object; cleanup failure is logged. Upload-session finalization, including the session used by SFTP import, can retain temporary bytes until explicit abort or later expiry cleanup. Inspect the session state and abort an abandoned session rather than assuming a validation error removed its bytes or changed its status to `FAILED`. A raw multipart upload may already have consumed its parts before validation fails, so do not blindly retry finalization; upload the corrected source through a verified new or repaired session. SFTP import leaves the remote source in place. When an already finalized file fails later indexing, its Storage bytes remain available to an authorized operator. With asynchronous indexing, require successful synchronization before writing business data.

## Records and physical lines

CSV statistics distinguish data records from physical source layout:

| Statistic | Meaning |
| - | - |
| `rowCount` | Logical data records, excluding the header and skipped blank records |
| `physicalLineCount` | Actual lines, including the header, blank lines and lines inside multiline fields |
| `blankLineCount` | Physical lines that are empty or contain only whitespace |
| `endsWithRecordDelimiter` | Whether the source ends with the configured record delimiter |
| `lineCount` | Existing line-window count, retaining the terminal empty element for compatibility |

For `id|name\n1|A\n`, `rowCount` is 1, `physicalLineCount` is 2 and legacy `lineCount` is 3. The terminating delimiter does not add a third actual physical line. An empty file has zero physical lines.

Synchronous raw CSV finalization validates records and counts physical lines in one scan. Native generated row parts capture counts while encoding, and finalization reuses those counts without rereading the generated file just to count lines. A physical line index can still be built lazily when a line-window read is requested.

Use record mode for CSV, TSV, XLSX, NDJSON/JSONL, JSON arrays and record-oriented XML. Explicit `mode: 'lines'` is a separate text-window contract for custom raw-line workflows: it returns strings without parsing fields and preserves headers, blank lines and the terminal empty element. It is not an automatic recovery mode for malformed CSV. For mixed line endings, line windows follow their selected or detected separator; physical CSV metrics count the actual CR, LF and CRLF boundaries.

Once synchronization has built statistics for the same source and dialect, record reads reuse them. A normal walker needs no extra `getFileStats()` call. Use explicit statistics for discovery or validation before separate preparatory work, and `reloadStats: true` only when recomputation is required. Preserve established processing sizes; file size alone is not a reason to shrink callback batches or split the file into extra transfers.

## Handle CSV records with the wrong column count

CSV and TSV record reads default to `onError: 'abort'`. A record whose field count differs from the source header makes indexing fail before rows are returned or a walker callback runs. Without a header, the first data record establishes the expected width. Schema fallback does not pad short records or discard surplus cells.

Choose `onError: 'skipRow'` when processing the valid records is acceptable and the skipped records will be reviewed. This skips complete records with the wrong field count. Invalid quote syntax and records exceeding `maxRecordBytes` still abort; the default record-size limit is 64 MiB.

Pass the policy to `getFileStats()`, `getFileData()`, or `walkFileData()`. Upload sessions accept `csv: { onError: 'skipRow' }` for finalization and background indexing. Each reader still defaults to `abort`, so use `skipRow` explicitly on every read, including continuation pages. Keep the policy unchanged while following a cursor; restart pagination to change it.

```js theme={null}
let acceptedRows = 0;

const result = await storage.walkFileData(
  storageEntryId,
  { onError: 'skipRow' },
  async (rows) => {
    acceptedRows += rows.length;
  },
  { fullScan: true, batchSize: 100_000, readBatchSize: 100_000 },
);

return { acceptedRows, csvReport: result.csvReport };
```

Statistics, pages, walker callback contexts, and the walker result expose `csvReport`:

* `skippedRows` is the exact number of skipped records in the complete source.
* `diagnostics` contains up to 100 samples with `recordNumber`, physical `lineStart`/`lineEnd`, `startByte`, and expected/actual field counts. Locations refer to the original file; samples contain no cell values.
* `diagnosticsTruncated` indicates that further diagnostic samples were omitted.

The report describes the whole indexed file even when a read selects only part of it. Do not sum repeated reports across pages or callbacks. Pagination, `rowCount`, `totalRows`, and walker counters count accepted records. If every data record is skipped, the result still includes the report and no walker callback runs. The original file remains unchanged.

This CSV parser policy is separate from database import `onError`, which controls validation and database write failures. It does not skip business-rule failures or apply to XLSX, other structured formats, physical-line mode, or byte reads.

Continue with [storage structured files](https://docs.revoengine.com/operate/storage-structured-files.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.