Skip to content

Choose a source ​

A source is what swath lists. Each has a page that follows one outline: what it lists and its URL, the smallest working command, authentication and permissions, its own options, writing a dataset to it, and its limits.

SourceURLListsEvery version (--all-versions)Resume of a dataset runA dataset written to it (-o)
Local and mounted filesystemsa pathfiles and directories–no: an interrupted run is run againa local directory
Amazon S3 and S3-compatible storess3://BUCKET[/PREFIX]current objects of a general-purpose bucketversions and delete markersyes-o s3://BUCKET/PREFIX
Google Cloud Storagegs://BUCKET[/PREFIX]current objectslive and noncurrent generationsyes-o gs://BUCKET/PREFIX
Azure Blob Storageaz://ACCOUNT/CONTAINER[/PREFIX]current blobs of a flat namespaceversions, snapshots and soft-deleted blobsyes-o az://ACCOUNT/CONTAINER/PREFIX

A dataset of any source can be written to a local directory (-o DIR) or to any of the three object stores (-o a bucket prefix with --work-dir DIR).

swath reads listings only: it needs permission to list, never to read or write objects. Each object-store listing sends requests the provider may charge for; the run report estimates their cost on AWS S3 and GCS (Cost).

What has run against the real service ​

Every source is tested against fakes, emulators and replayed recordings (testing). What has also run against the service itself, as recorded in measurements:

SourceListingEvery versionA dataset written to itCredentials
Local and mounted filesystemsreal filesystems; benchmarks on synthetic trees–yes–
Amazon S3one bucket of 1.09 billion objects; 35 public buckets of 50.6 billion in allseven small public bucketsfakes onlyanonymous only
S3-compatible storesMinIO only, where swath can miss keys (S3-compatible stores); R2, Ceph and others not runnot runfakes onlyMinIO access keys
Google Cloud Storagea bucket of 5.1 million objects; public bucketsone small public bucketyes: the datasets of the 35 S3 bucketsdefault credentials, anonymous
Azure Blob Storagea public container of 1.84 million blobsfakes onlyfakes onlyanonymous only; SAS, Shared Key and Entra ID not run

Runs killed and resumed against a real store are the small versioned runs above and MinIO; every other kill and resume is tested against fakes and replays. Credential renewal during a run has not been seen against a real service. Each source's other limits are under "Limits and known differences" on its page. Before relying on a store other than AWS, compare one prefix's object count with an independent listing (S3-compatible stores).

Credentials on every object store ​

Without --anonymous, swath uses the provider's credentials and stops before listing (exit 1, cause auth) when there are none or they do not load. It never falls back to unsigned requests: a public bucket or container needs --anonymous, which sends no credentials whatever the environment holds. The error is one line naming the credential source and the cause's first line; -v shows the whole cause (troubleshooting).

--anonymous beside an explicit credential (--s3-profile, --s3-bearer-token-command, --gcs-credentials-file, --azure-sas-token, a SAS in the source URL) is refused with exit 2. Credentials only the environment holds are ignored, with one warning naming the variables:

text
swath: warning: --anonymous: ignoring the GCS credentials in the environment (GOOGLE_APPLICATION_CREDENTIALS)

Secrets are not taken from flags, with two exceptions that are never recorded: --azure-sas-token and --s3-bearer-token-command. Neither reaches the journal, the report or the terminal (each shows [redacted]), so swath resume must be given them again.

The report's diagnostics.source.stats says where the credentials came from (credentials), whether the run was anonymous, and for S3 the region.

Endpoints and TLS ​

--endpoint-url URL sends requests to another service than the provider's own: an S3-compatible store, an emulator (MinIO, fake-gcs-server, Azurite). It must be http or https with a host; user information (user:password@), a query and a fragment are refused. The endpoint is part of the run's identity: a resume uses the same one. There is no cost estimate for a custom endpoint. An endpoint that has never answered and refuses every connection stops the run after about 30 seconds (exit 75, cause unreachable).

--ca-bundle FILE trusts the certificates in a PEM file besides the system's (a private certificate authority). --no-verify-ssl checks no certificate, for a test endpoint only: it warns on every run that has it, a resume included, and the report's diagnostics.source.stats.no_verify_ssl is true. Both apply to the credential lookups swath makes over HTTPS (S3's STS and SSO, GCS's token exchange) as to the listing. A resume may give either again; --verify-ssl undoes a recorded --no-verify-ssl.

Object-store requests ​

S3, GCS and Azure Blob share the request options:

OptionDefaultEffect
--request-timeout DURATION30sper-request timeout; a request over it is retried
--request-rate Nno capat most N HTTP attempts per second, retries and probes included
--no-adaptive-limitthe limit is ondo not lower concurrency when the service throttles

A failed request that can be retried (a throttle, a timeout, a transient 5xx, a cut connection) is tried up to nine times with exponential backoff (100 ms, at most 5 s apart); a listing page that still fails goes back to the queue and is tried again later. --stall-timeout and --max-duration bound a run that cannot make progress. Throttling and how to tune for it: Watch and tune a run.

More on each provider, for contributors: S3, GCS, Azure Blob, local filesystems.

Released under the Apache-2.0 license · by Varve