Skip to content

swathParallel listing and inventory for object stores and filesystems

See what's taking up space in a bucket or a directory, and save a Parquet inventory you can query without listing again.

Where 697 TiB sits in a public genomics bucket: 477,714 objects totaled in 3.5 seconds on a 4-vCPU cloud VM. How this was measured
swath du s3://1000genomes --anonymous --s3-region us-east-1
    size  objects  share                        newest          path
 697 TiB     478k   100%  ████████████████████  2022-01-25 UTC  s3://1000genomes
 242 TiB     119k  34.7%  ██████▉               2016-04-29        technical/
 156 TiB    20.0k  22.4%  ████▍                 2016-04-29          working/
79.8 TiB    5.15k  11.4%  ██▎                   2014-09-03            20130718_phase…_cortex_graphs/
44.4 TiB    1.56k   6.4%  █▎                    2014-09-03            20121015_cortex_phase2/
32.2 TiB    13.3k   4.6%  ▉                     2016-04-29            (294 more)
37.2 TiB    11.7k   5.3%  █                     2014-09-03          ncbi_varpipe_data/
37.2 TiB    11.7k   5.3%  █                     2014-09-03            alignment/
1.38 MiB        3   0.0%                        2014-09-02            (3 more)
28.0 TiB    4.69k   4.0%  ▊                     2014-09-04          other_exome_alignments/
20.6 TiB    82.2k   3.0%  ▌                     2015-07-18          (14 more)
 236 TiB     280k  33.8%  ██████▊               2016-03-23        phase3/
 235 TiB     279k  33.7%  ██████▋               2016-03-23          data/
 588 GiB    1.18k   0.1%                        2016-03-22          (1 more)
89.1 TiB    21.7k  12.8%  ██▌                   2016-03-22        phase1/
51.6 TiB    13.3k   7.4%  █▍                    2014-09-09          data/
36.5 TiB    7.90k   5.2%  █                     2016-03-22          technical/
17.2 TiB    3.85k   2.5%  ▍                     2014-09-03            ncbi_varpipe_data/
19.3 TiB    4.05k   2.8%  ▌                     2016-03-22            (3 more)
1.00 TiB      538   0.1%                        2014-12-02          (10 more)
63.3 TiB    8.08k   9.1%  █▊                    2015-09-08        data/
49.3 TiB    6.66k   7.1%  █▍                    2022-01-25        1000G_2504_high_coverage/
37.6 TiB    5.01k   5.4%  █                     2019-04-25          data/
11.7 TiB    1.65k   1.7%  ▎                     2022-01-25          (3 more)
17.5 TiB    42.4k   2.5%  ▌                     2016-12-15        (26 more)
  697 TiB in 477,714 objects
  complete: 477,714 objects, 697 TiB in 3.5s · 135,869 keys/s

Save an inventory, query it with SQL

A bigger public bucket, s3://noaa-gestofs-pds: a sorted Parquet inventory of 37 million objects in 61 seconds, then a DuckDB query over it in 3 seconds.
swath list s3://noaa-gestofs-pds/ -o noaa \
    --anonymous --s3-region us-east-1
  37,064,981 objects in 1m00s · 609,802 keys/s
  39,016 API calls · in flight avg 52.27 · peak 64
  …
  11 parts · 825 MiB written · peak RSS 1.51 GiB
duckdb -column -c "SELECT
    regexp_extract(key, '\.(\w+)$', 1) AS ext,
    count(*) AS objects,
    format_bytes(sum(size)::BIGINT) AS bytes
  FROM 'noaa/data/*.parquet'
  GROUP BY ext ORDER BY sum(size) DESC
  LIMIT 5"
ext    objects   bytes
-----  --------  --------
nc     382202    1.3 PiB
grib2  13178305  39.8 TiB
tgz    1051      5.3 TiB
tar    18140     4.5 TiB
png    22913029  1.3 TiB

The video replays that listing from its trace. swath hands the busiest part of the bucket to workers as they become free, so the listing keeps moving without being told how the keys are laid out: one of its 257 starting ranges held 64.5% of the objects and was split 1,834 times.

What swath is ​

swath is an open-source command-line tool that lists object stores and filesystems in parallel. It reads listing metadata only and never changes what it lists.

In one live run with --workers 2048, swath 0.4.0 wrote a sorted Parquet inventory of 1.09 billion S3 objects in 238 seconds (under 4 minutes) on one 64-vCPU VM; a separate count-only run of the same bucket on the same VM, which wrote nothing, took 169 seconds (how many workers to use). How this was measured

Download ​

The newest published build is swath 0.4.0, on the releases page. Download the archive for your platform and put swath on your PATH, or run the container image ghcr.io/varveio/swath:0.4.0. Install has the steps.

Try a public bucket ​

A public NOAA prefix of 5,252 objects, with no account: --anonymous sends no credentials. Start with a prefix like this one: the recorded listing of the whole bucket, the "Save an inventory, query it with SQL" example above, covered 37,064,981 objects in 39,016 requests.

bash
swath du s3://noaa-gestofs-pds/stofs_2d_glo.20230113/ --anonymous --s3-region us-east-1
swath list s3://noaa-gestofs-pds/stofs_2d_glo.20230113/ --anonymous --s3-region us-east-1 | head

On a terminal, bare swath du draws the view at the top of this page: the largest prefixes and objects at any depth, each under its parent with its share of the total. In a pipe it prints the total row only. -d N prints the totals of every prefix down to N levels instead, a table or TSV for scripts. Each swath du lists the whole prefix again: to look at a big bucket several ways, list it once with -o DIR and total the saved inventory with swath du --from-dataset DIR.

Try it on a local directory ​

No account or credentials needed. Replace ~/Downloads with any directory:

bash
swath du ~/Downloads                            # where the bytes are, at any depth
swath list ~/Downloads -o ~/downloads-inventory  # a sorted Parquet inventory
swath show ~/downloads-inventory                # what it holds and how complete it is

Your first inventory walks through the output, and Understand your results says what the numbers cover.

List your own buckets ​

bash
swath du s3://bucket/prefix/                     # where the bytes are, at any depth
swath list gs://bucket/prefix/ -o inventory      # a sorted Parquet inventory

Listing a bucket makes billable API requests and reads metadata, not object contents; credentials come from the provider's usual chain (sources). An interrupted bucket inventory continues where it stopped with swath resume DIR.

Sources: Amazon S3 and S3-compatible stores, Google Cloud Storage, Azure Blob Storage, and local or mounted filesystems. What each supports

Documentation ​

Released under the Apache-2.0 license · by Varve