swath list
swath list lists every object (or entry) below SOURCE, as text rows or into a sorted Parquet dataset.
Synopsis
swath list SOURCE [OPTIONS]
swath ls SOURCE [OPTIONS]Description
Without -o, or with -o -, the rows go to stdout: a table on a terminal, TSV in a pipe (--format chooses table, tsv or jsonl). -o NAME.tsv or -o NAME.jsonl, optionally ending in .gz or .zst, writes that file, compressed as its name says. Any other -o name, and any name ending in /, is a directory that receives a swath dataset: sorted Parquet parts and a manifest.json that commits them. -o gs://BUCKET[/PREFIX], -o s3://BUCKET[/PREFIX] or -o az://ACCOUNT/CONTAINER[/PREFIX], with --work-dir DIR, writes the dataset straight to Google Cloud Storage, S3 or Azure Blob Storage (A dataset in a bucket): its parts are uploaded as they close, its manifest is created only if the prefix holds none, and DIR is a local work directory for the lock, the journal and spill, which swath resume DIR continues. The destination has its own credentials, region and endpoint (Writing to a bucket). A name ending in an extension swath does not write (.csv, .json, .txt, a bare .gz, ...), --overwrite of a bucket prefix and a destination that overlaps a gs://, s3:// or az:// SOURCE are refused.
TSV and JSONL rows carry the source's columns under the dataset's names, by default key first, then size, last_modified and the rest of the source's schema; --columns chooses them and their order, and the table puts key last (Output). Every format and every source lists in ascending byte order of the key. A local tree lists its directories too (kind dir, the key ending in /) unless --no-dirs. A local SOURCE that is a file lists as its one row, read as the same entry would be in a listing of its parent: at the --metadata tier, and a symbolic link as the link, not its target.
A dataset directory must be new or empty; --overwrite replaces a previous swath dataset there, and nothing else, keeping it readable until the new one commits (Overwrite and refresh). A text file is written beside its name and renamed into place when complete, so a failed run leaves the previous file as it was; it always replaces the file, with or without --overwrite. A reader that closes stdout early (| head) ends the run with exit 0.
Only a dataset run of an object store keeps a journal: if it is interrupted, swath resume continues it. Text output and local trees are run again.
The filters keep or drop rows by key, size, time and storage class. A filtered dataset says so in its manifest.
Options
Output
-o, --output PATH
a dataset DIR, a .tsv or .jsonl file (.gz, .zst), or - (stdout)A name ending in .parquet is refused unless it ends in / (an explicit directory). A file's name says its format; a --format that agrees with it is accepted.
--format auto|table|tsv|jsonl
on stdout: auto is the table on a terminal, TSV otherwise--no-header
TSV: leave out the header line of column namesNot with the table, JSONL or a dataset; on a terminal the rows are then TSV.
--columns NAME[,NAME...]
TSV and JSONL: these columns, in order (repeatable)By the dataset's column names; a name the source's schema lacks is refused, listing its columns. Not with the table or a dataset; on a terminal the rows are then TSV.
--overwrite
replace a previous dataset in -o--work-dir DIR
with a bucket -o: the local work directory (lock, journal, spill)Required when -o is a bucket prefix (GCS, S3 or Azure Blob), and only then. It holds what a resume needs, never a part: swath resume DIR continues the run. Size it for the spill budget, the journal and receipts, and a reserve.
--output-gcs-credentials-file FILE
credentials for a GCS -o (default: Application Default Credentials)A service account key or authorized-user JSON, with the read-write scope. The destination's own: the source's credentials and --anonymous never apply to it. It may change on resume.
--output-s3-region NAME
the region of an S3 -oDefault: AWS_REGION, AWS_DEFAULT_REGION, the --output-s3-profile's, else us-east-1. The region resolved is fixed for the run.
--output-s3-profile NAME
the shared config profile an S3 -o takes credentials (and region) fromDefault: the AWS SDK chain (environment, AWS_PROFILE or default, SSO, web identity, IMDS). The destination's own: the source's credentials and --anonymous never apply to it, and its requests are always signed. It may change on resume.
--output-azure-sas-token TOKEN
a shared access signature for an Azure -o; never recordedDefault: Entra ID (DefaultAzureCredential). The destination's own: the source's SAS, key and --anonymous never apply to it. Give it again to swath resume; it may change there.
--output-endpoint-url URL
the endpoint of a bucket -o (default: the provider's)GCS's JSON API base, an S3-compatible endpoint URL, or an Azure blob service base. GCS: as the SDK takes it (https://storage.googleapis.com/storage/v1/). S3: addressed path-style; AWS_ENDPOINT_URL and AWS_ENDPOINT_URL_S3 are never applied to the destination. Azure: the account is its first path segment, as for an emulator (http://127.0.0.1:10000 serves ENDPOINT/ACCOUNT/CONTAINER). Fixed for the run.
Filters
--include GLOB
keep keys this glob matches (repeatable: any of them)Matched against the key below SOURCE: with a '/' from SOURCE's root (logs/*.gz), without one the last segment at any depth (*.gz). * and ? stay within a segment, ** is whole segments, [a-z], [!a-z], {a,b}, \c. Includes sharing a literal prefix narrow the listed prefix. Filters select rows; recursive excludes can also prune local traversal.
--exclude GLOB
drop keys this glob matches (repeatable; beats any include)Matched as --include is. dir/ drops a directory's own row, dir/** the rows below it. Local trees skip traversal below directories covered by a terminal /**; other excludes only filter rows.
--include-regex REGEX
keep keys this RE2 regex matches anywhere in the key (repeatable)Matched against the key below SOURCE, as a glob is, on every source: ^b/ is SOURCE's b/. Any include, glob or regex, keeps a key. ^ with a literal prefix narrows the listed prefix as a glob's does.
--exclude-regex REGEX
drop keys this RE2 regex matches anywhere in the key (repeatable)--min-size SIZE
keep objects (local: entries) of at least this size (1K, 1.5GiB)Every SIZE: bytes, or K, M, G, T and KiB, MiB, GiB, TiB (binary) or KB, MB, GB, TB (decimal), any case.
--max-size SIZE
keep objects or entries of at most this size--modified-since TIME
keep objects or entries modified at or after TIME (2026-01-01, 7d)2026-01-01, 2026-01-01T12:00 or 2026-01-01 12:00:00 is local time (TZ), as the tables print it; with Z or an offset, that zone. A duration (7d, 12h) is that long before the run started. Either is recorded as the UTC time it names.
--modified-until TIME
keep objects or entries modified at or before this time--storage-class NAME[,NAME...]
keep objects in these storage classes (STANDARD,GLACIER; repeatable)In any case; an object without one is in none. Azure Blob: the access tier (Hot, Cool, Cold, Archive).
Object stores
--anonymous
send no credentials (a public bucket or container)Ignores any configured. Without it a source with no usable credentials fails (auth) before listing.
--endpoint-url URL
the service URL in place of the provider's own (emulators, MinIO)S3: path-style addressing unless --no-s3-path-style; default AWS_ENDPOINT_URL_S3, else AWS_ENDPOINT_URL. GCS: the JSON API base URL (fake-gcs-server). Azure Blob: the blob service URL, the account then its first path segment (Azurite). User information, a query and a fragment are refused. Part of the resume identity.
--ca-bundle FILE
trust the certificates in this PEM file besides the system'sFor a service behind a private certificate authority. S3: default AWS_CA_BUNDLE.
--no-verify-ssl
do not verify the service's TLS certificate (warns)For a test endpoint only. A resume replays it, and warns again.
--all-versions
list every version of every key, not only the current objectsS3: versions and delete markers (needs s3:ListBucketVersions). GCS: generations, live and noncurrent. Azure Blob: versions, snapshots and soft-deleted blobs. A key's rows are consecutive and never split across parts.
S3
--s3-region NAME
the bucket's regionDefault: AWS_REGION, AWS_DEFAULT_REGION, the profile's, else us-east-1.
--s3-profile NAME
the shared config profile to take credentials and region fromDefault: the AWS SDK chain (environment, AWS_PROFILE or default, SSO, web identity, IMDS).
--s3-requester-pays
accept the request charges of a requester-pays bucket--s3-path-style
address the bucket in the path (default with --endpoint-url)--no-s3-path-style
address the bucket in the host name, also with --endpoint-url--s3-bearer-token-command COMMAND
a shell command printing a bearer token, sent instead of SigV4For GCS's XML API: 'gcloud auth print-access-token'. Never recorded: swath resume needs it again.
--s3-bearer-token-refresh-interval DURATION
run --s3-bearer-token-command again after this long (default45m)--s3-fips
use the FIPS endpoint (s3-fips.REGION.amazonaws.com)Default: AWS_USE_FIPS_ENDPOINT. Refused with --endpoint-url, where it would select nothing.
--s3-dualstack
use the dual-stack (IPv6) endpointDefault: AWS_USE_DUALSTACK_ENDPOINT. Refused with --endpoint-url, where it would select nothing.
--s3-fetch-owner
fill owner_id and owner_display_name
GCS
--gcs-credentials-file FILE
a service-account or authorized-user JSON file (default: ADC)Without it, Application Default Credentials.
--gcs-billing-project NAME
bill a requester-pays bucket's requests to this project
Azure Blob
--azure-sas-token TOKEN
a shared access signature (a query string); never recordedElse a SAS in the source's query, else AZURE_STORAGE_SAS_TOKEN; with none, an account key in AZURE_STORAGE_KEY (Shared Key), else Entra ID (DefaultAzureCredential). Give it again to swath resume.
--azure-fetch-metadata
fill the metadata column with each blob's user metadataA JSON object, keys sorted; {} for a blob without any, null for one encrypted with a customer-provided key. Up to 8 KiB a blob: it grows the responses, the memory a row holds and the dataset.
--azure-fetch-tags
fill the tags column with each blob's index tagsA JSON object, keys sorted; {} for a blob without any. Needs the right to read tags (a SAS with the t permission, or an Entra ID role such as Storage Blob Data Owner); without it every request fails, anonymous ones included.
--azure-fetch-immutability
fill the immutability policy and legal hold columns--azure-assume-flat
assume a flat namespace where the account will not tellFor when Get Account Information and the Data Lake host's Filesystem Get Properties are both refused or silent (a Data Lake host that refuses the account for soft delete or blob events already establishes a flat one); a hierarchical namespace they report still refuses. Frozen for resume and shown in the report.
Local
--metadata names|stat|statx
what is read per entry (statx adds created_at; Linux) (defaultstat)--no-dirs
omit the directories' rows (kind dir)--no-link-target
do not read symbolic links' targets-x, --one-file-system
do not walk into a mount point below the root, as du -x
Performance
-j, --workers N
listing workers (default 64; a local tree: 4 per CPU, at most 64)Raise it (128, 256) for a large listing the service does not throttle: listing waits on round trips, not the CPU; lower it when it throttles at the default.
--encoders N
Parquet encoders, or a .zst file's compression jobs (default: CPUs)Lower it to leave cores to other work; the default follows GOMAXPROCS, the descriptor limit and an encoder allowance of 1/16 of memory (as the default sort buffer, whatever --sort-buffer says; at most 1GiB). An explicit count is kept, with a warning when they cannot carry it.
--sort-buffer SIZE
bytes buffered for sorted output; not a total-memory limitRows listed ahead of the output, the output queues and the scheduler's metadata. Past it rows spill to disk; with no spill space listing waits. Request pages, directory reads, writers, compression and the Go runtime are extra, and the tail after listing holds about one more; encoders are admitted against an allowance of their own, so this sizes rows only. Default: 1/16 of the least of detected physical memory, the cgroup's limit and GOMEMLIMIT, at most 1GiB (128MiB when none is detected; the report's diagnostics.sort_buffer says which). Raise it when spill is heavy and memory is free; to use less memory lower it with --workers and --encoders.
--spill-budget SIZE
fixed cap on spill (default: follows the spill disk's free space)By default spill may use the spill filesystem's free space after a reserve (half of it unless the dataset is in a bucket, as the output shares it), re-read while it holds listing back, so a disk grown under a live run is used. Set it when the filesystem is shared with other work. Past it, or at the reserve, listing waits instead of spilling.
--part-size SIZE
target size of a Parquet part (default64MiB)Smaller parts for readers that load whole parts, larger for fewer files on a very large listing.
--part-interval DURATION
close a part with rows after this long, full or not (default30s)0: by size only. Lower it to close parts sooner on a slow listing (it bounds how long rows wait in an open part, not how much a resume repeats: a closed part still publishes in order); 0 for part boundaries independent of timing.
--part-max-rows N
also close a part at N rows (0: by size and --part-interval only)Set it when readers want parts of a bounded row count; it is fixed for the run.
--seed-from PATH
object stores: start from a previous listing's key distributionA committed dataset of the same source and scope (its cuts.txt, else its parts' key page index), or a cut file (one key per line, ascending bytewise, optionally TAB and the keys expected since the previous cut; a bucket dataset's cuts.txt is one); the engine still splits adaptively.
Limits
--max-duration DURATION
stop the run after this long (exit 124); 0: no limitGive every unattended run one. Only a dataset run of an object store (one with a journal) resumes where it stopped; text output and swath du run again.
--stall-timeout DURATION
stop as stuck after this long without progress (exit 75) (default10m)0: never. Raise it for a service that is slow to answer for minutes at a time. Only a dataset run of an object store (one with a journal) resumes where it stopped.
--spill-wait-timeout DURATION
stop after waiting this long on a full spill cap (exit 74); 0: waitListing waits on the output while spill is refused, which on a large bucket can be close to a stall; after 2 minutes swath says so. Set it on an unattended run to stop instead (cause spill_budget), then resume with more disk or a larger --spill-budget. Only a dataset run of an object store (one with a journal) resumes where it stopped.
--request-timeout DURATION
per-request timeout; a request over it is retried (default30s)Raise it for a slow endpoint or emulator whose pages take longer to answer.
--request-rate N
cap every HTTP attempt at N per second (0: no cap)Set it to stay under a request quota shared with other clients. Refused when the spacing exceeds half --stall-timeout.
--no-adaptive-limit
do not lower concurrency when the service throttles requestsTurn it off (with --request-rate) when a run crawls under sustained throttling with the limit near 1. Without it each of the --workers listing workers waits on one page at a time, and seed listings and structure probes run beside them.
Reporting
--report FILE
write the run report (JSON) to FILE, or - for stdout, at the endWritten on failure too, with the outcome and the attempt id; not for a command line that does not parse (an unknown flag, a bad value). A directory, or a name whose directory does not exist, is refused before listing; a report that cannot be written at the end fails the command (exit 1). - is refused where stdout carries rows.
-v, --verbose
a start line, progress records; -vv, -vvv: the request log-vv adds each failed request and retry with its cause, -vvv every request (credentials and tokens redacted).
-q, --quiet
no warnings, no automatic progress or closing lines (-qq: no errors)With -qq the exit code and the reports stand alone.
--progress
show progress on stderr (default: a terminal that rows do not share)Not by default with -q or -v (which appends progress records), nor while rows go to the same terminal, where swath list refuses it; swath du's table comes after the line is erased.
--no-progress
no live progress on stderr--progress-interval DURATION
how often progress is printed (at least 1s); implies --progressDefault 1s while the line redraws on a terminal, 30s for appended records.
--stats
print the closing lines on stderr, with the run's statisticsThe closing lines (the verdict, the count, the time, where the output went, what to run next) print by default for runs over 1.5 s, runs that write output, and runs that do not complete; not with -q, nor after a closed downstream pipe. --stats prints them always, followed by the requests, in flight, throttles, retries, the estimate and the peak RSS.
--no-stats
no closing lines on stderr--color auto|always|never
color progress, closing lines, the help and du's view (defaultauto)auto: on a terminal, without NO_COLOR or TERM=dumb; CLICOLOR_FORCE forces it.
--log-format text|json
stderr as text, or json: one JSON object per line (defaulttext)json: warnings, the start line, progress records, the request log and the outcome as events, no redrawn line or closing lines.
Examples
Show the first rows of a prefix:
swath list s3://bucket/logs/ | headWrite a dataset with a run report beside it, stopping after six hours:
swath list s3://bucket/logs/ -o /data/logs-2026-11-10 --report /var/log/swath/attempt-1.json --max-duration 6hWrite every object's key and size as compressed JSONL:
swath list gs://bucket -o rows.jsonl.zst --columns key,sizeList a public bucket without credentials:
swath list s3://noaa-gestofs-pds/stofs_2d_glo.20230113/ --anonymousKeep the .parquet files modified in the last week of a local tree:
swath list /data --include '*.parquet' --modified-since 7dList every version and delete marker:
swath list s3://bucket/logs/ --all-versions -o logs-versionsExit status
0 when the listing completed, or the reader closed stdout. 1 when it failed (access denied, no such bucket), 2 for a usage error or a refusal (a non-empty -o directory without --overwrite), 3 when the output is written but incomplete (entries could not be read). A dataset run of an object store that stops early exits 74, 75, 124, 129, 130 or 143 and can be resumed. See Exit codes.
Environment
The provider's credential and endpoint variables (Sources), and those of swath(1).
Files
A dataset directory holds:
DIR/data/part-NNNNN.parquet the parts, in key order
DIR/manifest.json the commit: written last; a reader starts only once it exists
DIR/run.json the run report, after the commit
DIR/cuts.txt the cut file, for the next listing's --seed-from, after run.json (not for a local tree)
DIR/error.json the outcome of an attempt that did not complete
DIR/.swath/ swath's workspace: the lock, the journal, spill, progress.jsonA dataset in a bucket (-o gs://BUCKET/PREFIX, -o s3://BUCKET/PREFIX or -o az://ACCOUNT/CONTAINER/PREFIX, with --work-dir DIR) holds PREFIX/manifest.json, PREFIX/data/RUN/ATTEMPT/part-NNNNN.parquet and PREFIX/run.json; its work directory holds DIR/.swath/ (the lock, the journal, spill, and remote/: the allocated names, the claim receipts, the candidate manifest), DIR/error.json and, once committed, DIR/commit.json.
The swath dataset format describes the parts and the manifest; Stop, resume and refresh the workspace, and Size a run its disk.
See also
swath du, swath resume, swath show, Output, Filters, Sources, Run report and progress file