Varve · open source

swath

Parallel, resumable S3 listing for very large buckets.

swath divides an unknown S3 keyspace into guessed ranges, corrects the guesses while it lists — splitting overloaded ranges so idle workers can take the other half — and writes JSONL, TSV, or crash-resumable Parquet. It never reads object contents, and --concurrency is a ceiling, not a target.

View on GitHub Quick start
See how it works

The visual field guide

Why parallel listing is a guessing problem, the range algebra that keeps splits safe, and where swath honestly loses.

Interrogate a real run

39.7M objects · 128 workers

One recorded listing of s3://noaa-gestofs-pds, rendered from its 89,230-event trace: replay, seed distribution, mechanism ledgers.

36 seconds of that run
One guess out of 513 secretly held 68% of the bucket. Every frame is rendered from the run's own trace — the one derived panel says so on screen.