Skip to content

swath ​

swath lists object stores and file trees in parallel, into a sorted Parquet dataset, as text rows, or as size totals.

Synopsis ​

text
swath COMMAND [OPTIONS] ...
swath --help | -h
swath --version | -V

Description ​

swath reads listing metadata (names, sizes, times, storage classes) through each object store's listing API or by walking a directory tree. It never reads or changes object contents.

CommandWhat it does
swath list (also swath ls)list SOURCE as text rows on stdout or into a .tsv or .jsonl file, or into a sorted Parquet dataset with -o DIR
swath dutotal SOURCE's objects and bytes, per prefix down to -d levels
swath resumecontinue an interrupted dataset run
swath showprint a dataset's summary, or an interrupted run's recorded command and progress
swath completionprint a shell completion script
swath help COMMANDthe same as swath COMMAND --help
swath versionthe same as swath --version

swath diag runs list or a count with options meant for experiments and comparisons, not for operating the tool; it is described with the internals.

SOURCE ​

FormSource
s3://BUCKET[/PREFIX]an Amazon S3 bucket, or an S3-compatible store with --endpoint-url
gs://BUCKET[/PREFIX]a Google Cloud Storage bucket
az://ACCOUNT/CONTAINER[/PREFIX]an Azure Blob Storage container; https://ACCOUNT.blob.core.windows.net/CONTAINER[/PREFIX] is the same source
a patha local or mounted directory, or a single file

The scheme is matched in any case (S3://bucket). The prefix is a key prefix, not a directory: s3://bucket/logs lists logs/ and logs-old/, s3://bucket/logs/ only logs/. Each source's URL form, credentials and limits are in Sources.

Command line ​

Options are parsed GNU-style: short options bundle (-vq) and take attached values (-d1, -oPATH); long options take --name value or --name=value; options and operands may come in any order; -- ends the options. A long name with one dash (-depth) is refused. A switch that is on by default is turned off by its --no- spelling (--no-progress, --no-s3-path-style). A list option (--columns key,size, --storage-class STANDARD,GLACIER) refuses an empty element (key,) with exit 2.

The global options (-v, -q, --color, --log-format) may stand anywhere on the line, before or after the command word, and count only before --. A command line refused early, an unknown option among them, still reports in the --log-format it names:

text
$ swath list /data --bogus --log-format json
{"ts":"...","level":"error","event":"outcome","msg":"unknown flag --bogus","class":"usage","exit_code":2,"resumable":false,"status":"failed","complete":false}

An unknown option is a usage error that names the nearest one and the help to read:

text
$ swath list s3://bucket --region eu-west-1
swath: unknown flag --region (did you mean --s3-region?)
See 'swath list --help'.

An option that belongs to one provider carries its prefix (--s3-region, --gcs-credentials-file, --azure-sas-token) and is refused, with exit 2, for another provider's source. Options every object store shares (--endpoint-url, --anonymous) and the local tree's own (--one-file-system, -x) are bare. A familiar spelling (--endpoint, --profile, --region, --force-path-style, --idle-timeout) gets the same suggestion of the current name, and an option swath does not have (--compression, --output-type) says what does its work. A value error names the option and the value as typed, then what is wanted: swath: --min-size "1e3": want a size such as 1K, .... Every usage error ends with the help to read.

A value that can carry a secret is never shown or recorded as typed: --azure-sas-token, --s3-bearer-token-command, and the user information, query or fragment of SOURCE, --endpoint-url or a URL given to -o are [redacted] in errors, logs, the report and the journal. Plain paths are shown as typed.

-h prints a command's short help (the usage, the examples and the most used options), --help all of it. Both go to stdout and exit 0; swath alone prints its own short help. swath list and swath du refuse -h given with anything else (exit 2): du -h users mean human-readable sizes, which the table already prints.

Sizes and durations ​

A SIZE is a decimal number and an optional unit, in any case: bytes without one (or B); K, M, G, T and KiB, MiB, GiB, TiB are binary; KB, MB, GB, TB are decimal. 64M is 64 MiB, 64MB 64,000,000 bytes. 1.5G is accepted; a size that does not come to whole bytes (1.3B) is refused. What swath prints for people uses binary units only (KiB, MiB, GiB); TSV, JSONL and the reports carry exact bytes.

A DURATION is a number with a unit: 90s, 1h30m, and d (24 hours) and w (7 days) besides (2d, 1d12h). A negative duration is refused. 0 turns off --max-duration, --stall-timeout and --part-interval.

stdout and stderr ​

stdout carries data only: text rows, totals, or the run report with --report -. A run that writes a dataset (-o DIR) or a text file (-o FILE) prints nothing on stdout unless --report - puts the report there; -o - sends the rows to stdout. Everything else goes to stderr:

  • Errors and warnings, each one line starting swath:. A usage error adds the help to read.

  • The progress line, on a terminal that does not also show the rows, redrawn every second:

    text
    listing · 33,010 objects · 5.30 TiB · 3.0s elapsed · 14,012 keys/s (avg 11,003) · 0 in flight · 84 API calls

    The objects listed and their bytes come first, then the elapsed time and the rate, the parts a dataset run has published, and last, for an object store, the requests in flight and sent and the cost estimate; a narrow terminal drops fields from the end. While every request fails, the cause being retried takes the rate's place (retrying: connection refused). A local tree counts entries (and entries/s), --all-versions counts rows. When stderr is not a terminal, --progress appends the same record as a line every 30 seconds (--progress-interval sets it). There is no percentage or ETA: a listing does not know its total in advance. A warning erases the line and it is drawn again below.

  • The closing lines, at the end of a run that took over 1.5 seconds, wrote output or did not complete:

    text
      complete: 60,010 objects, 5.55 TiB in 8.4s · 7,183 keys/s → all.tsv (4.83 MiB)

    The first line is the verdict (complete, or with its cause partial: 1 unreadable entry, stopped: --max-duration reached, interrupted), then what the run counted and the bytes listed when known (none when a size is unknown, as with --metadata names), the time it took, the rate for a run of 2 seconds or more, and where the output went (nothing for stdout). Rows the filters dropped follow the ones kept (3 objects kept, 1 filtered out); swath du counts from its total row (7 files, 9 directories), which counts a hard-linked file once, and gives the filters' account in rows after it (1 file (2 entries kept, 1 filtered out)); a dataset run that stopped says what it saved for a resume beside what it listed (7,003 saved, 12,003 listed), and a resume gives the dataset's total with the rows from earlier attempts apart. A dataset run adds a next: line: swath show DIR · swath du --from-dataset DIR once committed, swath resume DIR when it can be resumed. The warnings that explain a verdict come before the lines. --stats prints them always, followed by the run's statistics: the API calls with the mean and peak in flight, the throttles and retries when any, the cost estimate, and the peak RSS. --no-stats leaves the closing lines out. --color colors the progress line and the closing lines (the verdict in red when the output is partial or missing).

-v adds a line saying where the rows go (swath: listing s3://bkt to all.tsv.zst (attempt 57ee1c0a)) and shows progress as appended records; -vv adds every failed request and retry with its cause; -vvv every request, its credentials and tokens redacted. -q drops the warnings and the automatic progress and statistics; -qq also drops the final error line, leaving the exit code and the reports.

When requests keep failing with one cause for 10 seconds without an answer, swath says so without -v, and again at most once a minute while it lasts:

text
swath: warning: S3 127.0.0.1:9: connection refused (retrying; 14 attempts)

--log-format json writes stderr as one JSON object per line instead, with no redrawn line and no statistics block: Log events.

On Unix, SIGUSR1 (and SIGINFO, Ctrl-T, on macOS and the BSDs) prints one progress record on stderr, whatever the progress settings.

Options ​

These options are taken by every command, before or after the command word. -h, --help and -V, --version are described above.

  • -v, --verbose
    a start line, progress records; -vv, -vvv: the request log

    -vv adds each failed request and retry with its cause, -vvv every request (credentials and tokens redacted).

  • -q, --quiet
    no warnings, no automatic progress or closing lines (-qq: no errors)

    With -qq the exit code and the reports stand alone.

  • --color auto|always|never
    color progress, closing lines, the help and du's view (default auto)

    auto: on a terminal, without NO_COLOR or TERM=dumb; CLICOLOR_FORCE forces it.

  • --log-format text|json
    stderr as text, or json: one JSON object per line (default text)

    json: warnings, the start line, progress records, the request log and the outcome as events, no redrawn line or closing lines.

Examples ​

List a prefix as text rows and keep the first ten:

bash
swath list s3://bucket/logs/ | head

Write a sorted Parquet dataset:

bash
swath list s3://bucket/logs/ -o logs-inventory

Total a local tree one level down:

bash
swath du -d 1 /data

Print the short help of swath du:

bash
swath du -h

Exit status ​

CodeMeaning
0completed (also a reader that closed the pipe early, as head does)
1failed: a person fixes the cause; a run whose outcome says resumable then resumes
2a usage error, or a refusal that is the caller's to fix
3finished, but the output is incomplete; in a listing, entries could not be read
74the output's filesystem is full
75stuck: no progress for --stall-timeout, an endpoint that never answered, or a bucket commit whose outcome is unknown
124--max-duration reached
129hung up (SIGHUP: a closed terminal or SSH session)
130interrupted (SIGINT)
143terminated (SIGTERM)

What each one means and what to do: Exit codes.

Environment ​

VariableEffect
NO_COLORset and not empty: no color with --color auto
CLICOLOR_FORCEcolor even when stderr is not a terminal
TERMdumb: no color with --color auto
TZthe time zone of the tables' local times
TMPDIRwhere text rows on stdout spill (see Size a run)
AWS_*, GOOGLE_APPLICATION_CREDENTIALS, AZURE_STORAGE_*credentials, regions and endpoints: Sources

The development variables (profiles, engine switches) are in Diagnostics.

See also ​

swath list, swath du, swath resume, swath show, swath completion, Your first inventory, Output, Sources

Released under the Apache-2.0 license · by Varve