Run large listings
This page lists a bucket of your own into a saved inventory, start to finish: the command, what it prints while it runs and when it ends, how to stop it, and where the result is. It assumes you have followed Your first inventory and can reach your store: its URL, credentials and permissions are on its source page.
The command
swath list s3://bucket/prefix/ -o inventory --max-duration 6hs3://bucket/prefix/is what to list: a bucket, or a prefix in it.gs://bucket/prefix/andaz://account/container/prefix/work the same way.-o inventorywrites a swath dataset, sorted Parquet parts and amanifest.jsonthat commits them, to the directoryinventory. It must be new or empty: a non-empty one is refused (exit 2) unless you give--overwrite, which replaces only a previous swath dataset. Put it on a disk with room for the dataset and for its spill, the temporary sorting data a sorted run writes there on its way (disk sizing). When you list a mounted filesystem, keep the directory outside the tree you list.--max-duration 6hstops the run if it is still going after six hours (exit 124);swath resumecan continue it. There is no limit by default. Give one to every run you will not watch: it is the one stop for a service that keeps returning rows without the listing ever finishing.
The workers (64 by default), memory and disk need no flag for a first run. For a bucket of hundreds of millions of objects, read Size a run first.
While it runs
On a terminal, swath draws one progress line on stderr, redrawn every second:
listing · 1,441,002 objects · 51.6 TiB · 2.0s elapsed · 969,817 keys/s (avg 720,501) · 0 parts · 63 in flight · 1,510 API callsIt shows the objects listed and their bytes, the time and the rate, the Parquet parts written, then the requests in flight and the requests sent, and last an estimate of their cost when swath knows the store's price (left out here). There is no percentage: a listing does not know its total in advance. On a large bucket parts can stay at 0 for a long time: swath lists the whole key space at once, holds the rows that are not yet next in key order (on disk once its memory budget is full), and writes them out in order, much of it near the end of the run.
When stderr is not a terminal (a log, a scheduler), add --progress to get the same line every 30 seconds. What each number means, the file inventory/.swath/progress.json and the signs of trouble are in Watch and tune a run.
When it ends
The run ends with two closing lines on stderr. On a public prefix of 5,252 objects:
complete: 5,252 objects, 383 GiB in 0.9s → inventory (1 part, 121 KiB)
next: swath show inventory · swath du --from-dataset inventorycomplete and exit code 0 mean the dataset is committed and covers the whole prefix. The directory now holds data/ (the Parquet parts), manifest.json, run.json (the run report), cuts.txt (for the next listing, below) and .swath/ (the run's workspace, which readers ignore). swath show inventory prints the verdict, what was listed and when, and the coverage; Query an inventory reads the rows.
A run that stops early says why on its first closing line, and on the second what to run next:
| Closing line | Exit | What to do |
|---|---|---|
interrupted | 129, 130, 143 | swath resume inventory (below) |
stopped: --max-duration reached | 124 | swath resume inventory, which keeps the recorded deadline; swath resume inventory --max-duration 0 lifts it |
partial: N unreadable entries (a mounted filesystem) | 3 | the dataset is committed without the entries that could not be read; fix their permissions and list again (coverage) |
A run that fails before listing starts, on a bucket that does not exist or access that is denied, prints one swath: line with the cause (exit 1). A run that fails later prints a failed: closing line first, then the swath: line; here the output directory's data/ could not be written:
failed: 5,252 objects, 383 GiB in 6.0s · 869 keys/s
swath: encode: publish part 0: renameat .swath/parts/part-00000.parquet.claimed data/part-00000.parquet: permission deniedTroubleshooting has each message and its fix, and Exit codes every code, the stuck run (75) and the full disk (74) among them.
For a script, the exit code and a file in the directory are the outcome: inventory/run.json beside a committed dataset, else inventory/error.json, each with class, exit_code and resumable. --report FILE keeps a copy of each attempt's full report: give each attempt a new path outside the output directory (Run report).
Stop it and continue
Ctrl-C (SIGINT), SIGTERM, and closing the terminal (SIGHUP, unless the run is under nohup) stop the run cleanly: the parts already written stay, and error.json records the stop.
interrupted: 0 saved, 2,233,277 listed in 3.0s · 746,666 keys/s → inventory
next: swath resume inventorysaved counts the rows in written parts; rows listed but not yet saved may be listed again. To continue:
swath resume inventoryThe resume replays the recorded command, keeps the saved parts and continues after them. swath show inventory says where a stopped run is. To start over instead, run the command again with --overwrite, or into a new directory. A run of a mounted filesystem keeps no journal and cannot be resumed: run it again with --overwrite. What a resume keeps and when it is refused: Stop, resume and refresh.
List it again
To refresh the inventory later, list into a new directory and pass the previous dataset as --seed-from:
swath list s3://bucket/prefix/ -o inventory-2 --seed-from inventory --max-duration 6hThe previous dataset must be of the same bucket and prefix (S3, GCS and Azure Blob; a mounted filesystem takes none). It tells swath how the keys are spread, so the run divides the work up front instead of finding it as it goes; the rows are the same (warm start). Keep the old directory until readers have moved to the new one. --overwrite replaces a dataset in place instead (overwrite and refresh), and a nightly run under a supervisor is in Run on a schedule.
More
- Size a run: workers, disk, memory and the cost of requests.
- Watch and tune a run: the progress line and file, the closing lines, throttling, memory and disk, a stuck run.
- Stop, resume and refresh: signals,
swath resume, replacing an inventory and seeding a refresh. - Run on a schedule: a nightly run under a supervisor, a dataset written to a bucket, preemptible machines.
- List every version: S3 versions and delete markers, GCS generations, Azure Blob versions, snapshots and soft-deleted blobs.
- Query an inventory: reading and verifying the dataset a run writes.
- Known limits: platforms, power loss, live scans and the scale measured so far.