Skip to content

swath list ​

swath list lists every object (or entry) below SOURCE, as text rows or into a sorted Parquet dataset.

Synopsis ​

text
swath list SOURCE [OPTIONS]
swath ls SOURCE [OPTIONS]

Description ​

Without -o, or with -o -, the rows go to stdout: a table on a terminal, TSV in a pipe (--format chooses table, tsv or jsonl). -o NAME.tsv or -o NAME.jsonl, optionally ending in .gz or .zst, writes that file, compressed as its name says. Any other -o name, and any name ending in /, is a directory that receives a swath dataset: sorted Parquet parts and a manifest.json that commits them. -o gs://BUCKET[/PREFIX], -o s3://BUCKET[/PREFIX] or -o az://ACCOUNT/CONTAINER[/PREFIX], with --work-dir DIR, writes the dataset straight to Google Cloud Storage, S3 or Azure Blob Storage (A dataset in a bucket): its parts are uploaded as they close, its manifest is created only if the prefix holds none, and DIR is a local work directory for the lock, the journal and spill, which swath resume DIR continues. The destination has its own credentials, region and endpoint (Writing to a bucket). A name ending in an extension swath does not write (.csv, .json, .txt, a bare .gz, ...), --overwrite of a bucket prefix and a destination that overlaps a gs://, s3:// or az:// SOURCE are refused.

TSV and JSONL rows carry the source's columns under the dataset's names, by default key first, then size, last_modified and the rest of the source's schema; --columns chooses them and their order, and the table puts key last (Output). Every format and every source lists in ascending byte order of the key. A local tree lists its directories too (kind dir, the key ending in /) unless --no-dirs. A local SOURCE that is a file lists as its one row, read as the same entry would be in a listing of its parent: at the --metadata tier, and a symbolic link as the link, not its target.

A dataset directory must be new or empty; --overwrite replaces a previous swath dataset there, and nothing else, keeping it readable until the new one commits (Overwrite and refresh). A text file is written beside its name and renamed into place when complete, so a failed run leaves the previous file as it was; it always replaces the file, with or without --overwrite. A reader that closes stdout early (| head) ends the run with exit 0.

Only a dataset run of an object store keeps a journal: if it is interrupted, swath resume continues it. Text output and local trees are run again.

The filters keep or drop rows by key, size, time and storage class. A filtered dataset says so in its manifest.

Options ​

Output ​

  • -o, --output PATH
    a dataset DIR, a .tsv or .jsonl file (.gz, .zst), or - (stdout)

    A name ending in .parquet is refused unless it ends in / (an explicit directory). A file's name says its format; a --format that agrees with it is accepted.

  • --format auto|table|tsv|jsonl
    on stdout: auto is the table on a terminal, TSV otherwise

  • --no-header
    TSV: leave out the header line of column names

    Not with the table, JSONL or a dataset; on a terminal the rows are then TSV.

  • --columns NAME[,NAME...]
    TSV and JSONL: these columns, in order (repeatable)

    By the dataset's column names; a name the source's schema lacks is refused, listing its columns. Not with the table or a dataset; on a terminal the rows are then TSV.

  • --overwrite
    replace a previous dataset in -o

  • --work-dir DIR
    with a bucket -o: the local work directory (lock, journal, spill)

    Required when -o is a bucket prefix (GCS, S3 or Azure Blob), and only then. It holds what a resume needs, never a part: swath resume DIR continues the run. Size it for the spill budget, the journal and receipts, and a reserve.

  • --output-gcs-credentials-file FILE
    credentials for a GCS -o (default: Application Default Credentials)

    A service account key or authorized-user JSON, with the read-write scope. The destination's own: the source's credentials and --anonymous never apply to it. It may change on resume.

  • --output-s3-region NAME
    the region of an S3 -o

    Default: AWS_REGION, AWS_DEFAULT_REGION, the --output-s3-profile's, else us-east-1. The region resolved is fixed for the run.

  • --output-s3-profile NAME
    the shared config profile an S3 -o takes credentials (and region) from

    Default: the AWS SDK chain (environment, AWS_PROFILE or default, SSO, web identity, IMDS). The destination's own: the source's credentials and --anonymous never apply to it, and its requests are always signed. It may change on resume.

  • --output-azure-sas-token TOKEN
    a shared access signature for an Azure -o; never recorded

    Default: Entra ID (DefaultAzureCredential). The destination's own: the source's SAS, key and --anonymous never apply to it. Give it again to swath resume; it may change there.

  • --output-endpoint-url URL
    the endpoint of a bucket -o (default: the provider's)

    GCS's JSON API base, an S3-compatible endpoint URL, or an Azure blob service base. GCS: as the SDK takes it (https://storage.googleapis.com/storage/v1/). S3: addressed path-style; AWS_ENDPOINT_URL and AWS_ENDPOINT_URL_S3 are never applied to the destination. Azure: the account is its first path segment, as for an emulator (http://127.0.0.1:10000 serves ENDPOINT/ACCOUNT/CONTAINER). Fixed for the run.

Filters ​

  • --include GLOB
    keep keys this glob matches (repeatable: any of them)

    Matched against the key below SOURCE: with a '/' from SOURCE's root (logs/*.gz), without one the last segment at any depth (*.gz). * and ? stay within a segment, ** is whole segments, [a-z], [!a-z], {a,b}, \c. Includes sharing a literal prefix narrow the listed prefix. Filters select rows; recursive excludes can also prune local traversal.

  • --exclude GLOB
    drop keys this glob matches (repeatable; beats any include)

    Matched as --include is. dir/ drops a directory's own row, dir/** the rows below it. Local trees skip traversal below directories covered by a terminal /**; other excludes only filter rows.

  • --include-regex REGEX
    keep keys this RE2 regex matches anywhere in the key (repeatable)

    Matched against the key below SOURCE, as a glob is, on every source: ^b/ is SOURCE's b/. Any include, glob or regex, keeps a key. ^ with a literal prefix narrows the listed prefix as a glob's does.

  • --exclude-regex REGEX
    drop keys this RE2 regex matches anywhere in the key (repeatable)

  • --min-size SIZE
    keep objects (local: entries) of at least this size (1K, 1.5GiB)

    Every SIZE: bytes, or K, M, G, T and KiB, MiB, GiB, TiB (binary) or KB, MB, GB, TB (decimal), any case.

  • --max-size SIZE
    keep objects or entries of at most this size

  • --modified-since TIME
    keep objects or entries modified at or after TIME (2026-01-01, 7d)

    2026-01-01, 2026-01-01T12:00 or 2026-01-01 12:00:00 is local time (TZ), as the tables print it; with Z or an offset, that zone. A duration (7d, 12h) is that long before the run started. Either is recorded as the UTC time it names.

  • --modified-until TIME
    keep objects or entries modified at or before this time

  • --storage-class NAME[,NAME...]
    keep objects in these storage classes (STANDARD,GLACIER; repeatable)

    In any case; an object without one is in none. Azure Blob: the access tier (Hot, Cool, Cold, Archive).

Object stores ​

  • --anonymous
    send no credentials (a public bucket or container)

    Ignores any configured. Without it a source with no usable credentials fails (auth) before listing.

  • --endpoint-url URL
    the service URL in place of the provider's own (emulators, MinIO)

    S3: path-style addressing unless --no-s3-path-style; default AWS_ENDPOINT_URL_S3, else AWS_ENDPOINT_URL. GCS: the JSON API base URL (fake-gcs-server). Azure Blob: the blob service URL, the account then its first path segment (Azurite). User information, a query and a fragment are refused. Part of the resume identity.

  • --ca-bundle FILE
    trust the certificates in this PEM file besides the system's

    For a service behind a private certificate authority. S3: default AWS_CA_BUNDLE.

  • --no-verify-ssl
    do not verify the service's TLS certificate (warns)

    For a test endpoint only. A resume replays it, and warns again.

  • --all-versions
    list every version of every key, not only the current objects

    S3: versions and delete markers (needs s3:ListBucketVersions). GCS: generations, live and noncurrent. Azure Blob: versions, snapshots and soft-deleted blobs. A key's rows are consecutive and never split across parts.

S3 ​

  • --s3-region NAME
    the bucket's region

    Default: AWS_REGION, AWS_DEFAULT_REGION, the profile's, else us-east-1.

  • --s3-profile NAME
    the shared config profile to take credentials and region from

    Default: the AWS SDK chain (environment, AWS_PROFILE or default, SSO, web identity, IMDS).

  • --s3-requester-pays
    accept the request charges of a requester-pays bucket

  • --s3-path-style
    address the bucket in the path (default with --endpoint-url)

  • --no-s3-path-style
    address the bucket in the host name, also with --endpoint-url

  • --s3-bearer-token-command COMMAND
    a shell command printing a bearer token, sent instead of SigV4

    For GCS's XML API: 'gcloud auth print-access-token'. Never recorded: swath resume needs it again.

  • --s3-bearer-token-refresh-interval DURATION
    run --s3-bearer-token-command again after this long (default 45m)

  • --s3-fips
    use the FIPS endpoint (s3-fips.REGION.amazonaws.com)

    Default: AWS_USE_FIPS_ENDPOINT. Refused with --endpoint-url, where it would select nothing.

  • --s3-dualstack
    use the dual-stack (IPv6) endpoint

    Default: AWS_USE_DUALSTACK_ENDPOINT. Refused with --endpoint-url, where it would select nothing.

  • --s3-fetch-owner
    fill owner_id and owner_display_name

GCS ​

Azure Blob ​

  • --azure-sas-token TOKEN
    a shared access signature (a query string); never recorded

    Else a SAS in the source's query, else AZURE_STORAGE_SAS_TOKEN; with none, an account key in AZURE_STORAGE_KEY (Shared Key), else Entra ID (DefaultAzureCredential). Give it again to swath resume.

  • --azure-fetch-metadata
    fill the metadata column with each blob's user metadata

    A JSON object, keys sorted; {} for a blob without any, null for one encrypted with a customer-provided key. Up to 8 KiB a blob: it grows the responses, the memory a row holds and the dataset.

  • --azure-fetch-tags
    fill the tags column with each blob's index tags

    A JSON object, keys sorted; {} for a blob without any. Needs the right to read tags (a SAS with the t permission, or an Entra ID role such as Storage Blob Data Owner); without it every request fails, anonymous ones included.

  • --azure-fetch-immutability
    fill the immutability policy and legal hold columns

  • --azure-assume-flat
    assume a flat namespace where the account will not tell

    For when Get Account Information and the Data Lake host's Filesystem Get Properties are both refused or silent (a Data Lake host that refuses the account for soft delete or blob events already establishes a flat one); a hierarchical namespace they report still refuses. Frozen for resume and shown in the report.

Local ​

Performance ​

  • -j, --workers N
    listing workers (default 64; a local tree: 4 per CPU, at most 64)

    Raise it (128, 256) for a large listing the service does not throttle: listing waits on round trips, not the CPU; lower it when it throttles at the default.

  • --encoders N
    Parquet encoders, or a .zst file's compression jobs (default: CPUs)

    Lower it to leave cores to other work; the default follows GOMAXPROCS, the descriptor limit and an encoder allowance of 1/16 of memory (as the default sort buffer, whatever --sort-buffer says; at most 1GiB). An explicit count is kept, with a warning when they cannot carry it.

  • --sort-buffer SIZE
    bytes buffered for sorted output; not a total-memory limit

    Rows listed ahead of the output, the output queues and the scheduler's metadata. Past it rows spill to disk; with no spill space listing waits. Request pages, directory reads, writers, compression and the Go runtime are extra, and the tail after listing holds about one more; encoders are admitted against an allowance of their own, so this sizes rows only. Default: 1/16 of the least of detected physical memory, the cgroup's limit and GOMEMLIMIT, at most 1GiB (128MiB when none is detected; the report's diagnostics.sort_buffer says which). Raise it when spill is heavy and memory is free; to use less memory lower it with --workers and --encoders.

  • --spill-budget SIZE
    fixed cap on spill (default: follows the spill disk's free space)

    By default spill may use the spill filesystem's free space after a reserve (half of it unless the dataset is in a bucket, as the output shares it), re-read while it holds listing back, so a disk grown under a live run is used. Set it when the filesystem is shared with other work. Past it, or at the reserve, listing waits instead of spilling.

  • --part-size SIZE
    target size of a Parquet part (default 64MiB)

    Smaller parts for readers that load whole parts, larger for fewer files on a very large listing.

  • --part-interval DURATION
    close a part with rows after this long, full or not (default 30s)

    0: by size only. Lower it to close parts sooner on a slow listing (it bounds how long rows wait in an open part, not how much a resume repeats: a closed part still publishes in order); 0 for part boundaries independent of timing.

  • --part-max-rows N
    also close a part at N rows (0: by size and --part-interval only)

    Set it when readers want parts of a bounded row count; it is fixed for the run.

  • --seed-from PATH
    object stores: start from a previous listing's key distribution

    A committed dataset of the same source and scope (its cuts.txt, else its parts' key page index), or a cut file (one key per line, ascending bytewise, optionally TAB and the keys expected since the previous cut; a bucket dataset's cuts.txt is one); the engine still splits adaptively.

Limits ​

  • --max-duration DURATION
    stop the run after this long (exit 124); 0: no limit

    Give every unattended run one. Only a dataset run of an object store (one with a journal) resumes where it stopped; text output and swath du run again.

  • --stall-timeout DURATION
    stop as stuck after this long without progress (exit 75) (default 10m)

    0: never. Raise it for a service that is slow to answer for minutes at a time. Only a dataset run of an object store (one with a journal) resumes where it stopped.

  • --spill-wait-timeout DURATION
    stop after waiting this long on a full spill cap (exit 74); 0: wait

    Listing waits on the output while spill is refused, which on a large bucket can be close to a stall; after 2 minutes swath says so. Set it on an unattended run to stop instead (cause spill_budget), then resume with more disk or a larger --spill-budget. Only a dataset run of an object store (one with a journal) resumes where it stopped.

  • --request-timeout DURATION
    per-request timeout; a request over it is retried (default 30s)

    Raise it for a slow endpoint or emulator whose pages take longer to answer.

  • --request-rate N
    cap every HTTP attempt at N per second (0: no cap)

    Set it to stay under a request quota shared with other clients. Refused when the spacing exceeds half --stall-timeout.

  • --no-adaptive-limit
    do not lower concurrency when the service throttles requests

    Turn it off (with --request-rate) when a run crawls under sustained throttling with the limit near 1. Without it each of the --workers listing workers waits on one page at a time, and seed listings and structure probes run beside them.

Reporting ​

  • --report FILE
    write the run report (JSON) to FILE, or - for stdout, at the end

    Written on failure too, with the outcome and the attempt id; not for a command line that does not parse (an unknown flag, a bad value). A directory, or a name whose directory does not exist, is refused before listing; a report that cannot be written at the end fails the command (exit 1). - is refused where stdout carries rows.

  • -v, --verbose
    a start line, progress records; -vv, -vvv: the request log

    -vv adds each failed request and retry with its cause, -vvv every request (credentials and tokens redacted).

  • -q, --quiet
    no warnings, no automatic progress or closing lines (-qq: no errors)

    With -qq the exit code and the reports stand alone.

  • --progress
    show progress on stderr (default: a terminal that rows do not share)

    Not by default with -q or -v (which appends progress records), nor while rows go to the same terminal, where swath list refuses it; swath du's table comes after the line is erased.

  • --no-progress
    no live progress on stderr

  • --progress-interval DURATION
    how often progress is printed (at least 1s); implies --progress

    Default 1s while the line redraws on a terminal, 30s for appended records.

  • --stats
    print the closing lines on stderr, with the run's statistics

    The closing lines (the verdict, the count, the time, where the output went, what to run next) print by default for runs over 1.5 s, runs that write output, and runs that do not complete; not with -q, nor after a closed downstream pipe. --stats prints them always, followed by the requests, in flight, throttles, retries, the estimate and the peak RSS.

  • --no-stats
    no closing lines on stderr

  • --color auto|always|never
    color progress, closing lines, the help and du's view (default auto)

    auto: on a terminal, without NO_COLOR or TERM=dumb; CLICOLOR_FORCE forces it.

  • --log-format text|json
    stderr as text, or json: one JSON object per line (default text)

    json: warnings, the start line, progress records, the request log and the outcome as events, no redrawn line or closing lines.

Examples ​

Show the first rows of a prefix:

bash
swath list s3://bucket/logs/ | head

Write a dataset with a run report beside it, stopping after six hours:

bash
swath list s3://bucket/logs/ -o /data/logs-2026-11-10 --report /var/log/swath/attempt-1.json --max-duration 6h

Write every object's key and size as compressed JSONL:

bash
swath list gs://bucket -o rows.jsonl.zst --columns key,size

List a public bucket without credentials:

bash
swath list s3://noaa-gestofs-pds/stofs_2d_glo.20230113/ --anonymous

Keep the .parquet files modified in the last week of a local tree:

bash
swath list /data --include '*.parquet' --modified-since 7d

List every version and delete marker:

bash
swath list s3://bucket/logs/ --all-versions -o logs-versions

Exit status ​

0 when the listing completed, or the reader closed stdout. 1 when it failed (access denied, no such bucket), 2 for a usage error or a refusal (a non-empty -o directory without --overwrite), 3 when the output is written but incomplete (entries could not be read). A dataset run of an object store that stops early exits 74, 75, 124, 129, 130 or 143 and can be resumed. See Exit codes.

Environment ​

The provider's credential and endpoint variables (Sources), and those of swath(1).

Files ​

A dataset directory holds:

text
DIR/data/part-NNNNN.parquet   the parts, in key order
DIR/manifest.json             the commit: written last; a reader starts only once it exists
DIR/run.json                  the run report, after the commit
DIR/cuts.txt                  the cut file, for the next listing's --seed-from, after run.json (not for a local tree)
DIR/error.json                the outcome of an attempt that did not complete
DIR/.swath/                   swath's workspace: the lock, the journal, spill, progress.json

A dataset in a bucket (-o gs://BUCKET/PREFIX, -o s3://BUCKET/PREFIX or -o az://ACCOUNT/CONTAINER/PREFIX, with --work-dir DIR) holds PREFIX/manifest.json, PREFIX/data/RUN/ATTEMPT/part-NNNNN.parquet and PREFIX/run.json; its work directory holds DIR/.swath/ (the lock, the journal, spill, and remote/: the allocated names, the claim receipts, the candidate manifest), DIR/error.json and, once committed, DIR/commit.json.

The swath dataset format describes the parts and the manifest; Stop, resume and refresh the workspace, and Size a run its disk.

See also ​

swath du, swath resume, swath show, Output, Filters, Sources, Run report and progress file

Released under the Apache-2.0 license · by Varve