Amazon S3 and S3-compatible stores
s3://BUCKET[/PREFIX] lists the current objects of a general-purpose bucket, or those whose keys start with the prefix; --all-versions lists every version and delete marker. With --endpoint-url the same source lists an S3-compatible store.
The smallest command
A public NOAA prefix, with no AWS account:
swath du s3://noaa-gestofs-pds/stofs_2d_glo.20230113/ -d 1 --anonymous --s3-region us-east-1Without --anonymous swath signs its requests with your AWS credentials.
Authentication and permissions
Permissions. s3:ListBucket on the bucket; s3:ListBucketVersions with --all-versions (swath never falls back to a current-object listing when it is refused).
Credentials. The AWS SDK's default chain: environment keys (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN), the shared config and credentials files and their profiles (--s3-profile, else AWS_PROFILE), IAM Identity Center (SSO), web identity, credential_process, AssumeRole profiles, container credentials (ECS, EKS Pod Identity) and EC2 instance roles (IMDSv2). The first lookup is waited for up to 30 seconds (a web-identity token file mounted late included). Refused before any request as configured but broken: an access key without its secret or the reverse, a session token without both keys, AWS_ROLE_ARN without AWS_WEB_IDENTITY_TOKEN_FILE, a profile that does not exist. Rules shared by every object store are in Choose a source.
Long runs. Credentials that expire (an assumed role, web identity, SSO, credential_process, container credentials, an instance role) are refreshed before they expire: 5 minutes before, or when a quarter of their lifetime is left if that is shorter. If S3 still answers that the token has expired, swath refreshes them and sends the request once more. Keys that do not refresh (AWS_ACCESS_KEY_ID with AWS_SESSION_TOKEN, or a profile's keys) are used as given: once they expire the run stops with cause auth and says so. The run report counts the refreshes (credentials.refreshes) and those an expired-token answer forced (credentials.expired_refreshes). The refresh is tested against fakes only, and an upload to an S3 dataset destination answered as expired fails instead of being sent again (known limits).
Region. --s3-region, else AWS_REGION, AWS_DEFAULT_REGION, the profile's region, else us-east-1, never the EC2 instance metadata's region. swath does not discover a bucket's region: a request to the wrong one fails with an error that names the bucket's region and the option to give (--s3-region eu-west-1). STS calls use the environment's or the profile's region, not --s3-region.
Options for this source
--s3-requester-paysaccepts the request charges of a requester-pays bucket.--s3-fetch-ownerfills theowner_idandowner_display_namecolumns.--s3-fipsand--s3-dualstack(elseAWS_USE_FIPS_ENDPOINT,AWS_USE_DUALSTACK_ENDPOINT) select AWS's FIPS and dual-stack endpoints; they are refused with a custom endpoint.--s3-bearer-token-command COMMANDsends an OAuth token instead of SigV4, for an S3-compatible endpoint that takes one, such as GCS's XML API. The command runs under/bin/sh -c(on Windowscmd.exe /d /s /c); its one line of output is sent asAuthorization: Bearer TOKEN, and no AWS credential is looked up. It runs again after--s3-bearer-token-refresh-interval(45 minutes by default). A command that fails, runs past 30 seconds, or prints nothing or more than one line fails the request with causeauth. The published image has no shell, so the command cannot run in it: use the release binary, or an image of your own that adds a shell and the command's program.
S3-compatible stores
Give the store's URL and, usually, a region it accepts:
swath list s3://bucket --endpoint-url http://127.0.0.1:9000 --s3-region us-east-1Without --endpoint-url, AWS_ENDPOINT_URL_S3, else AWS_ENDPOINT_URL, is the endpoint (unless AWS_IGNORE_CONFIGURED_ENDPOINT_URLS is true). A custom endpoint is addressed path-style (https://host/BUCKET); --no-s3-path-style addresses the bucket in the host name instead, and --s3-path-style uses the path on AWS's own endpoint. Requests are signed with SigV4 for the region given. AWS_CA_BUNDLE is the CA bundle when --ca-bundle is not given. An endpoint_url in the shared config file (a profile's, or its services section's) is not read: give the endpoint as --endpoint-url or AWS_ENDPOINT_URL_S3.
Proxies. Requests to an https:// endpoint, AWS's included, go through HTTPS_PROXY (or https_proxy); HTTP_PROXY is used only for an http:// endpoint. NO_PROXY lists hosts to reach directly, and a loopback endpoint is never proxied. With only HTTP_PROXY set where the network allows no direct connection, requests to AWS do not get through.
The store must answer ListObjectsV2 as S3 does: in ascending byte order of the key, with continuation tokens. swath checks every page's order and refuses a service that breaks it (exit 1, cause protocol). It trusts the store's listing answers: a page with more keys than asked for (a response body up to 16 MiB) is listed as it is.
Limit: an empty page ends a range. swath lists ranges of keys in parallel, each from a start key (start-after), and takes a page with no keys and no continuation token as the end of its range. Some S3-compatible stores answer such a page while later keys remain, and swath then misses those keys without an error. In testing, MinIO RELEASE.2026-10-23T20-06-51Z answered every start key inside one prefix this way, and a listing of 250,000 keys returned 107,651. Before you rely on listings of a store other than AWS, compare the object count of one prefix, not being written to, with a listing that follows continuation tokens from the first key, such as the AWS CLI's:
swath du s3://bucket/prefix/ -d 0 --endpoint-url https://store.example --s3-region us-east-1
aws s3 ls s3://bucket/prefix/ --recursive --summarize --endpoint-url https://store.example | tail -2The objects of swath's total row and the AWS CLI's Total Objects must be equal. Run it on the prefix you intend to list: a small or unaffected prefix can pass on a store that fails elsewhere. Equal counts are a check, not a proof: they do not show that every key was listed in order. Tracked in #265.
Writing a dataset to S3
-o s3://BUCKET/PREFIX --work-dir DIR writes the dataset to S3; the command, what appears in the bucket and how to resume and clean up are in Writing to a bucket. What is S3's own:
- Credentials:
--output-s3-profile NAME(a shared config profile: keys, SSO, an assumed role, a credential process) or the AWS SDK's default chain; requests are always signed, whatever the source's--anonymousor--s3-profile. The default chain reads the same environment as the source's: give the destination a profile when the two must differ. The profile may change on resume. - Permissions: on the bucket (
arn:aws:s3:::BUCKET),s3:ListBucket: without it S3 answers a missingmanifest.jsonwith 403, not 404, and the fresh-prefix check fails. On the dataset's objects (arn:aws:s3:::BUCKET/PREFIX/*),s3:PutObject,s3:GetObject,s3:DeleteObject(orphans),s3:AbortMultipartUploadands3:ListMultipartUploadParts(an abort's confirmation); in a versioned bucket alsos3:GetObjectVersionands3:DeleteObjectVersion. - Region and endpoint:
--output-s3-region, elseAWS_REGION,AWS_DEFAULT_REGIONor the profile's, elseus-east-1; the one resolved, and--output-endpoint-url(an S3-compatible store, addressed path-style), are fixed for the run. The environment's endpoint variables (AWS_ENDPOINT_URL,AWS_ENDPOINT_URL_S3, FIPS, dual-stack) are never applied to the destination; a CA bundle is (AWS_CA_BUNDLE, or the profile'sca_bundle). - A lifecycle rule is required: a part larger than 8 MiB is a multipart upload. A failed or cancelled upload is aborted and the abort confirmed, but one that a killed process leaves is not, nor one whose abort the bucket refused or did not confirm (the report's
output.remote.aborts_unconfirmed), and S3 keeps an incomplete upload's parts, billed and invisible to listings, until it is aborted. Set a lifecycle rule that aborts incomplete multipart uploads (AbortIncompleteMultipartUpload,DaysAfterInitiation1 or more) on the prefix or the bucket before writing to it.
Limits and known differences
- Directory buckets (Express One Zone, names ending in
--x-s3) are refused with exit 2; access points, Multi-Region Access Points and Outposts are not supported. - A prefix is a string prefix:
s3://bucket/logscoverslogs/andlogs-old/. - The run report estimates request cost at AWS's reference rate only without an endpoint override.
- An S3-compatible store that answers an empty page before the end of a range makes swath miss keys without an error: compare one prefix's object count first (S3-compatible stores).