Skip to content

Run large listings ​

This page lists a bucket of your own into a saved inventory, start to finish: the command, what it prints while it runs and when it ends, how to stop it, and where the result is. It assumes you have followed Your first inventory and can reach your store: its URL, credentials and permissions are on its source page.

The command ​

bash
swath list s3://bucket/prefix/ -o inventory --max-duration 6h
  • s3://bucket/prefix/ is what to list: a bucket, or a prefix in it. gs://bucket/prefix/ and az://account/container/prefix/ work the same way.
  • -o inventory writes a swath dataset, sorted Parquet parts and a manifest.json that commits them, to the directory inventory. It must be new or empty: a non-empty one is refused (exit 2) unless you give --overwrite, which replaces only a previous swath dataset. Put it on a disk with room for the dataset and for its spill, the temporary sorting data a sorted run writes there on its way (disk sizing). When you list a mounted filesystem, keep the directory outside the tree you list.
  • --max-duration 6h stops the run if it is still going after six hours (exit 124); swath resume can continue it. There is no limit by default. Give one to every run you will not watch: it is the one stop for a service that keeps returning rows without the listing ever finishing.

The workers (64 by default), memory and disk need no flag for a first run. For a bucket of hundreds of millions of objects, read Size a run first.

While it runs ​

On a terminal, swath draws one progress line on stderr, redrawn every second:

text
  listing · 1,441,002 objects · 51.6 TiB · 2.0s elapsed · 969,817 keys/s (avg 720,501) · 0 parts · 63 in flight · 1,510 API calls

It shows the objects listed and their bytes, the time and the rate, the Parquet parts written, then the requests in flight and the requests sent, and last an estimate of their cost when swath knows the store's price (left out here). There is no percentage: a listing does not know its total in advance. On a large bucket parts can stay at 0 for a long time: swath lists the whole key space at once, holds the rows that are not yet next in key order (on disk once its memory budget is full), and writes them out in order, much of it near the end of the run.

When stderr is not a terminal (a log, a scheduler), add --progress to get the same line every 30 seconds. What each number means, the file inventory/.swath/progress.json and the signs of trouble are in Watch and tune a run.

When it ends ​

The run ends with two closing lines on stderr. On a public prefix of 5,252 objects:

text
  complete: 5,252 objects, 383 GiB in 0.9s → inventory (1 part, 121 KiB)
  next: swath show inventory · swath du --from-dataset inventory

complete and exit code 0 mean the dataset is committed and covers the whole prefix. The directory now holds data/ (the Parquet parts), manifest.json, run.json (the run report), cuts.txt (for the next listing, below) and .swath/ (the run's workspace, which readers ignore). swath show inventory prints the verdict, what was listed and when, and the coverage; Query an inventory reads the rows.

A run that stops early says why on its first closing line, and on the second what to run next:

Closing lineExitWhat to do
interrupted129, 130, 143swath resume inventory (below)
stopped: --max-duration reached124swath resume inventory, which keeps the recorded deadline; swath resume inventory --max-duration 0 lifts it
partial: N unreadable entries (a mounted filesystem)3the dataset is committed without the entries that could not be read; fix their permissions and list again (coverage)

A run that fails before listing starts, on a bucket that does not exist or access that is denied, prints one swath: line with the cause (exit 1). A run that fails later prints a failed: closing line first, then the swath: line; here the output directory's data/ could not be written:

text
  failed: 5,252 objects, 383 GiB in 6.0s · 869 keys/s
swath: encode: publish part 0: renameat .swath/parts/part-00000.parquet.claimed data/part-00000.parquet: permission denied

Troubleshooting has each message and its fix, and Exit codes every code, the stuck run (75) and the full disk (74) among them.

For a script, the exit code and a file in the directory are the outcome: inventory/run.json beside a committed dataset, else inventory/error.json, each with class, exit_code and resumable. --report FILE keeps a copy of each attempt's full report: give each attempt a new path outside the output directory (Run report).

Stop it and continue ​

Ctrl-C (SIGINT), SIGTERM, and closing the terminal (SIGHUP, unless the run is under nohup) stop the run cleanly: the parts already written stay, and error.json records the stop.

text
  interrupted: 0 saved, 2,233,277 listed in 3.0s · 746,666 keys/s → inventory
  next: swath resume inventory

saved counts the rows in written parts; rows listed but not yet saved may be listed again. To continue:

bash
swath resume inventory

The resume replays the recorded command, keeps the saved parts and continues after them. swath show inventory says where a stopped run is. To start over instead, run the command again with --overwrite, or into a new directory. A run of a mounted filesystem keeps no journal and cannot be resumed: run it again with --overwrite. What a resume keeps and when it is refused: Stop, resume and refresh.

List it again ​

To refresh the inventory later, list into a new directory and pass the previous dataset as --seed-from:

bash
swath list s3://bucket/prefix/ -o inventory-2 --seed-from inventory --max-duration 6h

The previous dataset must be of the same bucket and prefix (S3, GCS and Azure Blob; a mounted filesystem takes none). It tells swath how the keys are spread, so the run divides the work up front instead of finding it as it goes; the rows are the same (warm start). Keep the old directory until readers have moved to the new one. --overwrite replaces a dataset in place instead (overwrite and refresh), and a nightly run under a supervisor is in Run on a schedule.

More ​

  • Size a run: workers, disk, memory and the cost of requests.
  • Watch and tune a run: the progress line and file, the closing lines, throttling, memory and disk, a stuck run.
  • Stop, resume and refresh: signals, swath resume, replacing an inventory and seeding a refresh.
  • Run on a schedule: a nightly run under a supervisor, a dataset written to a bucket, preemptible machines.
  • List every version: S3 versions and delete markers, GCS generations, Azure Blob versions, snapshots and soft-deleted blobs.
  • Query an inventory: reading and verifying the dataset a run writes.
  • Known limits: platforms, power loss, live scans and the scale measured so far.

Released under the Apache-2.0 license · by Varve