A bigger public bucket, s3://noaa-gestofs-pds: a sorted Parquet inventory of 37 million objects in 61 seconds, then a DuckDB query over it in 3 seconds.
$ swath list s3://noaa-gestofs-pds/ -o noaa \
--anonymous --s3-region us-east-1
37,064,981 objects in 1m00s · 609,802 keys/s
39,016 API calls · in flight avg 52.27 · peak 64
…
11 parts · 825 MiB written · peak RSS 1.51 GiB
$ duckdb -column -c "SELECT
regexp_extract(key, '\.(\w+)$', 1) AS ext,
count(*) AS objects,
format_bytes(sum(size)::BIGINT) AS bytes
FROM 'noaa/data/*.parquet'
GROUP BY ext ORDER BY sum(size) DESC
LIMIT 5"
ext objects bytes
----- -------- --------
nc 382202 1.3 PiB
grib2 13178305 39.8 TiB
tgz 1051 5.3 TiB
tar 18140 4.5 TiB
png 22913029 1.3 TiB
The video replays that listing from its trace. swath hands the busiest part of the bucket to workers as they become free, so the listing keeps moving without being told how the keys are laid out: one of its 257 starting ranges held 64.5% of the objects and was split 1,834 times.
swath is an open-source command-line tool that lists object stores and filesystems in parallel. It reads listing metadata only and never changes what it lists.
In one live run with --workers 2048, swath 0.4.0 wrote a sorted Parquet inventory of 1.09 billion S3 objects in 238 seconds (under 4 minutes) on one 64-vCPU VM; a separate count-only run of the same bucket on the same VM, which wrote nothing, took 169 seconds (how many workers to use). How this was measured
The newest published build is swath 0.4.0, on the releases page. Download the archive for your platform and put swath on your PATH, or run the container image ghcr.io/varveio/swath:0.4.0. Install has the steps.
A public NOAA prefix of 5,252 objects, with no account: --anonymous sends no credentials. Start with a prefix like this one: the recorded listing of the whole bucket, the "Save an inventory, query it with SQL" example above, covered 37,064,981 objects in 39,016 requests.
bash
swath du s3://noaa-gestofs-pds/stofs_2d_glo.20230113/ --anonymous --s3-region us-east-1swath list s3://noaa-gestofs-pds/stofs_2d_glo.20230113/ --anonymous --s3-region us-east-1 | head
On a terminal, bare swath du draws the view at the top of this page: the largest prefixes and objects at any depth, each under its parent with its share of the total. In a pipe it prints the total row only. -d N prints the totals of every prefix down to N levels instead, a table or TSV for scripts. Each swath du lists the whole prefix again: to look at a big bucket several ways, list it once with -o DIR and total the saved inventory with swath du --from-dataset DIR.
No account or credentials needed. Replace ~/Downloads with any directory:
bash
swath du ~/Downloads # where the bytes are, at any depthswath list ~/Downloads -o ~/downloads-inventory # a sorted Parquet inventoryswath show ~/downloads-inventory # what it holds and how complete it is
swath du s3://bucket/prefix/ # where the bytes are, at any depthswath list gs://bucket/prefix/ -o inventory # a sorted Parquet inventory
Listing a bucket makes billable API requests and reads metadata, not object contents; credentials come from the provider's usual chain (sources). An interrupted bucket inventory continues where it stopped with swath resume DIR.
Sources: Amazon S3 and S3-compatible stores, Google Cloud Storage, Azure Blob Storage, and local or mounted filesystems. What each supports