VARVE / OPEN SOURCE

S3 object listing

swath

List very large S3 buckets in parallel, even when you don't know how their keys are distributed.

swath divides S3's ordered keyspace across workers and rebalances it as the listing runs. Stream object metadata as a table, TSV, or JSONL, or write a crash-resumable Parquet dataset for direct querying. Object contents are never read. A live listing is not a point-in-time snapshot of a bucket that changes during the run.

See how swath works → Quickstart
Use swath when

You need an on-demand listing of a very large bucket, no sufficiently current inventory is available, and serial pagination is too slow.

Use existing metadata when

A current S3 Inventory or S3 Metadata table already has the fields you need. Querying it is normally cheaper than a live scan.

For a small prefix or a one-off task, the AWS CLI or an SDK paginator is usually simpler.

36-second visualization39.7M objects · 68.6s trace event span
One live listing of a 39.7-million-object public bucket whose retained trace events span 68.6 seconds (68,592.2 ms), from first event to last. That is not a listing wall clock reported by swath. The initial keyspace split created 513 ranges. One contained 68% of the objects, and swath split that range as idle workers asked for more work. The video is a 36-second rendered visualization of run noaa-gestofs-pds-field-guide-trace, played at a pace chosen for viewing, not on the run's clock. View the trace report and provenance →
What the visualization shows, and what is known about the run

Each band is one key range being drained by one worker, colored by which initial range it descends from; a range splits when a worker steals from it or its own worker sheds work ahead of itself. The picture was generated by tools/explainer from the run's --trace event log. The raw event log was not retained, and this is not a terminal recording, so there is no on-screen command to transcribe.

Provenance: the swath version, commit, capture date, and command are unknown in retained evidence. The retained generated report preserves figures derived from the trace (objects, ranges, splits, and the first-to-last event span), but the raw trace and run summary were not retained. The provenance table lists exactly which evidence survives. This is a separate capture from the v0.2.1 recording in the project README (39,585,029 objects); a live bucket changes between runs, so the two are identified separately rather than treated as one number rounded two ways.

How it works · visual field guide

A step-by-step explanation of how swath works

The guide starts with why a normal S3 listing is serial, then shows how swath assigns key ranges without knowing where the objects are. It walks through safe splitting, work stealing, checkpoints, and the cases where swath is not the right tool.

Read the field guide →

No partitioning setup

Use the same command for flat, deeply prefixed, or skewed keys. swath divides the ordered keyspace while it lists, without a preliminary crawl or hand-written prefix rules.

Checkpoint and resume

Managed Parquet output records committed progress. An interrupted listing resumes from its output directory instead of starting over.

Stream or query

Write a table, TSV, JSONL, or Parquet. Query Parquet parts directly with DuckDB; object bodies are not downloaded.

Command-line workflow

List, resume,
and query

This demo interrupts a listing, resumes it from the output directory, and queries the resulting Parquet files directly with DuckDB.

Open the quickstart →
36-second CLI demolist / resume / DuckDB
A recorded Docker workflow: start a listing, interrupt it, resume it from its output directory, and query the Parquet parts with DuckDB. Captured with swath 0.2.1 (commit 505ae26e6019, read from the video's own opening swath --version frame); the capture date and machine are unknown in retained evidence, and the file entered this site's history on 2026-08-20 (commit 34a6c63), a lower bound on publication rather than a capture date.
Read the commands and result shown in the video

Transcribed from the video's own opening frames (its poster image, which is a still from the recording itself):

swath $ swath --version
swath 0.2.1
Built by Varve: https://varve.io
Source: https://github.com/varveio/swath
Commit: 505ae26e6019
Runtime: 25.0.3+9-LTS
swath $ swath list s3://noaa-gestofs-pds/ \
    --no-sign-request --region us-east-1 \
    --concurrency 128 \
    --format parquet -o /out/noaa-gestofs-pds
→ writing parquet dataset to /out/noaa-gestofs-pds
  listing · 4,002 objects · 3,999 keys/s (avg 3,999) · 5 pages

That is the command and the state visible at the video's start; the remainder — interrupting, resuming, and the DuckDB query — plays out in the recording itself and was not separately transcribed here. The command, version, and commit match run noaa-gestofs-pds-2026-08-03-505ae26 — the README's recorded full-scale run (39,585,029 objects) — but this video's own final object count was not independently confirmed from retained evidence, so treat it as a related capture sharing that run's identity, not as separately verified proof of its ending numbers.

Documentation for swath 0.3.2.