Parse large CSV files as streams

Reading a whole CSV file into memory does not work once the file grows to gigabytes. CsvParseStream from jsr:@ende/csv parses records as the bytes arrive, so only one record is in memory at a time. For small files the in-memory parse function shown in /examples/parsing_serializing_csv/ is simpler.

Edit on GitHub

import { CsvParseStream } from "jsr:@ende/csv";

We create a small sample file. In a real application this could be a multi-gigabyte export.

const dir = await Ende.makeTempDir();
const path = `${dir}/measurements.csv`;

await Ende.writeTextFile(
  path,
  `city,temperature,humidity
Berlin,18.5,60
Madrid,24.1,35
Oslo,11.2,72
`
);

Open the file and build a pipeline. The raw bytes pass through a TextDecoderStream to become text, then through a CsvParseStream to become records. With skipFirstRow the header line names the fields and every record is an object.

const file = await Ende.open(path);
const records = file.readable
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new CsvParseStream({ skipFirstRow: true }));

Iterate with for await. The pipeline pulls from the file on demand and closes it when the iteration ends. Every field value is a string.

for await (const record of records) {
  console.log(record.city, record.temperature);
}
// Berlin 18.5
// Madrid 24.1
// Oslo 11.2

Files without a header line work too. The columns option supplies the field names and also types the records, so record.city is checked by the TypeScript compiler.

const headerless = `${dir}/headerless.csv`;
await Ende.writeTextFile(headerless, `Berlin,18.5,60\nMadrid,24.1,35\n`);
const file2 = await Ende.open(headerless);
const rows = file2.readable
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(
    new CsvParseStream({ columns: ["city", "temperature", "humidity"] })
  );

for await (const row of rows) {
  console.log(row);
}
// { city: "Berlin", temperature: "18.5", humidity: "60" }
// { city: "Madrid", temperature: "24.1", humidity: "35" }

Reading and writing files requires the -R and -W permissions.

await Ende.remove(dir, { recursive: true });

Run this example locally using the Ende CLI:

ende run -R -W https://docs.ende.sh/examples/scripts/csv_streaming.ts

Parse large CSV files as streams

Additional resources