Hash large files with streams

The Web Crypto digest function takes a single buffer, so hashing a file with it means reading the whole file into memory first. The Standard Library ships a drop-in crypto wrapper whose digest also accepts async iterables, such as a file’s readable stream, and hashes chunk by chunk in constant memory.

Edit on GitHub

import { crypto } from "jsr:@ende/crypto";

Create a file to hash. In real code this could be gigabytes.

const path = await Ende.makeTempFile();
await Ende.writeTextFile(path, "a".repeat(1_000_000));

Open the file and pass its readable stream straight to digest. The stream is consumed chunk by chunk and never held in memory at once.

const file = await Ende.open(path);
const streamed = await crypto.subtle.digest("SHA-256", file.readable);
console.log(new Uint8Array(streamed).toHex());

cdc76e5c9914fb9281a1c7e284d73e67f1809a48a497200e046d39ccc7112cd0

For comparison, the one-shot approach: read the entire file into memory and hash the buffer. The standard wrapper still accepts plain buffers, so this is exactly what built-in Web Crypto would do.

const whole = await Ende.readFile(path);
const oneShot = await crypto.subtle.digest("SHA-256", whole);

Both approaches produce the same digest.

console.log(
  new Uint8Array(streamed).toHex() === new Uint8Array(oneShot).toHex()
); // true

await Ende.remove(path);

Run this example locally using the Ende CLI:

ende run -R -W https://docs.ende.sh/examples/scripts/hash_file_streams.ts

Hash large files with streams

Additional resources