swath
swath lists object stores and file trees in parallel, into a sorted Parquet dataset, as text rows, or as size totals.
Synopsis
swath COMMAND [OPTIONS] ...
swath --help | -h
swath --version | -VDescription
swath reads listing metadata (names, sizes, times, storage classes) through each object store's listing API or by walking a directory tree. It never reads or changes object contents.
| Command | What it does |
|---|---|
swath list (also swath ls) | list SOURCE as text rows on stdout or into a .tsv or .jsonl file, or into a sorted Parquet dataset with -o DIR |
swath du | total SOURCE's objects and bytes, per prefix down to -d levels |
swath resume | continue an interrupted dataset run |
swath show | print a dataset's summary, or an interrupted run's recorded command and progress |
swath completion | print a shell completion script |
swath help COMMAND | the same as swath COMMAND --help |
swath version | the same as swath --version |
swath diag runs list or a count with options meant for experiments and comparisons, not for operating the tool; it is described with the internals.
SOURCE
| Form | Source |
|---|---|
s3://BUCKET[/PREFIX] | an Amazon S3 bucket, or an S3-compatible store with --endpoint-url |
gs://BUCKET[/PREFIX] | a Google Cloud Storage bucket |
az://ACCOUNT/CONTAINER[/PREFIX] | an Azure Blob Storage container; https://ACCOUNT.blob.core.windows.net/CONTAINER[/PREFIX] is the same source |
| a path | a local or mounted directory, or a single file |
The scheme is matched in any case (S3://bucket). The prefix is a key prefix, not a directory: s3://bucket/logs lists logs/ and logs-old/, s3://bucket/logs/ only logs/. Each source's URL form, credentials and limits are in Sources.
Command line
Options are parsed GNU-style: short options bundle (-vq) and take attached values (-d1, -oPATH); long options take --name value or --name=value; options and operands may come in any order; -- ends the options. A long name with one dash (-depth) is refused. A switch that is on by default is turned off by its --no- spelling (--no-progress, --no-s3-path-style). A list option (--columns key,size, --storage-class STANDARD,GLACIER) refuses an empty element (key,) with exit 2.
The global options (-v, -q, --color, --log-format) may stand anywhere on the line, before or after the command word, and count only before --. A command line refused early, an unknown option among them, still reports in the --log-format it names:
$ swath list /data --bogus --log-format json
{"ts":"...","level":"error","event":"outcome","msg":"unknown flag --bogus","class":"usage","exit_code":2,"resumable":false,"status":"failed","complete":false}An unknown option is a usage error that names the nearest one and the help to read:
$ swath list s3://bucket --region eu-west-1
swath: unknown flag --region (did you mean --s3-region?)
See 'swath list --help'.An option that belongs to one provider carries its prefix (--s3-region, --gcs-credentials-file, --azure-sas-token) and is refused, with exit 2, for another provider's source. Options every object store shares (--endpoint-url, --anonymous) and the local tree's own (--one-file-system, -x) are bare. A familiar spelling (--endpoint, --profile, --region, --force-path-style, --idle-timeout) gets the same suggestion of the current name, and an option swath does not have (--compression, --output-type) says what does its work. A value error names the option and the value as typed, then what is wanted: swath: --min-size "1e3": want a size such as 1K, .... Every usage error ends with the help to read.
A value that can carry a secret is never shown or recorded as typed: --azure-sas-token, --s3-bearer-token-command, and the user information, query or fragment of SOURCE, --endpoint-url or a URL given to -o are [redacted] in errors, logs, the report and the journal. Plain paths are shown as typed.
-h prints a command's short help (the usage, the examples and the most used options), --help all of it. Both go to stdout and exit 0; swath alone prints its own short help. swath list and swath du refuse -h given with anything else (exit 2): du -h users mean human-readable sizes, which the table already prints.
Sizes and durations
A SIZE is a decimal number and an optional unit, in any case: bytes without one (or B); K, M, G, T and KiB, MiB, GiB, TiB are binary; KB, MB, GB, TB are decimal. 64M is 64 MiB, 64MB 64,000,000 bytes. 1.5G is accepted; a size that does not come to whole bytes (1.3B) is refused. What swath prints for people uses binary units only (KiB, MiB, GiB); TSV, JSONL and the reports carry exact bytes.
A DURATION is a number with a unit: 90s, 1h30m, and d (24 hours) and w (7 days) besides (2d, 1d12h). A negative duration is refused. 0 turns off --max-duration, --stall-timeout and --part-interval.
stdout and stderr
stdout carries data only: text rows, totals, or the run report with --report -. A run that writes a dataset (-o DIR) or a text file (-o FILE) prints nothing on stdout unless --report - puts the report there; -o - sends the rows to stdout. Everything else goes to stderr:
Errors and warnings, each one line starting
swath:. A usage error adds the help to read.The progress line, on a terminal that does not also show the rows, redrawn every second:
textlisting · 33,010 objects · 5.30 TiB · 3.0s elapsed · 14,012 keys/s (avg 11,003) · 0 in flight · 84 API callsThe objects listed and their bytes come first, then the elapsed time and the rate, the parts a dataset run has published, and last, for an object store, the requests in flight and sent and the cost estimate; a narrow terminal drops fields from the end. While every request fails, the cause being retried takes the rate's place (
retrying: connection refused). A local tree countsentries(andentries/s),--all-versionscountsrows. When stderr is not a terminal,--progressappends the same record as a line every 30 seconds (--progress-intervalsets it). There is no percentage or ETA: a listing does not know its total in advance. A warning erases the line and it is drawn again below.The closing lines, at the end of a run that took over 1.5 seconds, wrote output or did not complete:
textcomplete: 60,010 objects, 5.55 TiB in 8.4s · 7,183 keys/s → all.tsv (4.83 MiB)The first line is the verdict (
complete, or with its causepartial: 1 unreadable entry,stopped: --max-duration reached,interrupted), then what the run counted and the bytes listed when known (none when a size is unknown, as with--metadata names), the time it took, the rate for a run of 2 seconds or more, and where the output went (nothing for stdout). Rows the filters dropped follow the ones kept (3 objects kept, 1 filtered out);swath ducounts from its total row (7 files, 9 directories), which counts a hard-linked file once, and gives the filters' account in rows after it (1 file (2 entries kept, 1 filtered out)); a dataset run that stopped says what it saved for a resume beside what it listed (7,003 saved, 12,003 listed), and a resume gives the dataset's total with the rows from earlier attempts apart. A dataset run adds anext:line:swath show DIR · swath du --from-dataset DIRonce committed,swath resume DIRwhen it can be resumed. The warnings that explain a verdict come before the lines.--statsprints them always, followed by the run's statistics: the API calls with the mean and peak in flight, the throttles and retries when any, the cost estimate, and the peak RSS.--no-statsleaves the closing lines out.--colorcolors the progress line and the closing lines (the verdict in red when the output is partial or missing).
-v adds a line saying where the rows go (swath: listing s3://bkt to all.tsv.zst (attempt 57ee1c0a)) and shows progress as appended records; -vv adds every failed request and retry with its cause; -vvv every request, its credentials and tokens redacted. -q drops the warnings and the automatic progress and statistics; -qq also drops the final error line, leaving the exit code and the reports.
When requests keep failing with one cause for 10 seconds without an answer, swath says so without -v, and again at most once a minute while it lasts:
swath: warning: S3 127.0.0.1:9: connection refused (retrying; 14 attempts)--log-format json writes stderr as one JSON object per line instead, with no redrawn line and no statistics block: Log events.
On Unix, SIGUSR1 (and SIGINFO, Ctrl-T, on macOS and the BSDs) prints one progress record on stderr, whatever the progress settings.
Options
These options are taken by every command, before or after the command word. -h, --help and -V, --version are described above.
-v, --verbose
a start line, progress records; -vv, -vvv: the request log-vv adds each failed request and retry with its cause, -vvv every request (credentials and tokens redacted).
-q, --quiet
no warnings, no automatic progress or closing lines (-qq: no errors)With -qq the exit code and the reports stand alone.
--color auto|always|never
color progress, closing lines, the help and du's view (defaultauto)auto: on a terminal, without NO_COLOR or TERM=dumb; CLICOLOR_FORCE forces it.
--log-format text|json
stderr as text, or json: one JSON object per line (defaulttext)json: warnings, the start line, progress records, the request log and the outcome as events, no redrawn line or closing lines.
Examples
List a prefix as text rows and keep the first ten:
swath list s3://bucket/logs/ | headWrite a sorted Parquet dataset:
swath list s3://bucket/logs/ -o logs-inventoryTotal a local tree one level down:
swath du -d 1 /dataPrint the short help of swath du:
swath du -hExit status
| Code | Meaning |
|---|---|
| 0 | completed (also a reader that closed the pipe early, as head does) |
| 1 | failed: a person fixes the cause; a run whose outcome says resumable then resumes |
| 2 | a usage error, or a refusal that is the caller's to fix |
| 3 | finished, but the output is incomplete; in a listing, entries could not be read |
| 74 | the output's filesystem is full |
| 75 | stuck: no progress for --stall-timeout, an endpoint that never answered, or a bucket commit whose outcome is unknown |
| 124 | --max-duration reached |
| 129 | hung up (SIGHUP: a closed terminal or SSH session) |
| 130 | interrupted (SIGINT) |
| 143 | terminated (SIGTERM) |
What each one means and what to do: Exit codes.
Environment
| Variable | Effect |
|---|---|
NO_COLOR | set and not empty: no color with --color auto |
CLICOLOR_FORCE | color even when stderr is not a terminal |
TERM | dumb: no color with --color auto |
TZ | the time zone of the tables' local times |
TMPDIR | where text rows on stdout spill (see Size a run) |
AWS_*, GOOGLE_APPLICATION_CREDENTIALS, AZURE_STORAGE_* | credentials, regions and endpoints: Sources |
The development variables (profiles, engine switches) are in Diagnostics.
See also
swath list, swath du, swath resume, swath show, swath completion, Your first inventory, Output, Sources