Azure Blob Storage
az://ACCOUNT/CONTAINER[/PREFIX], or the same as https://ACCOUNT.blob.core.windows.net/CONTAINER[/PREFIX], lists a container's current blobs, or those whose names start with the prefix, in an account with a flat namespace. --all-versions lists every entry: blob versions, snapshots and soft-deleted blobs beside the current blobs.
The smallest command
A public container of Azure Open Datasets, with no Azure account:
swath du az://azureopendatastorage/citydatacontainer/Crime/ -d 1 --anonymousWithout --anonymous swath uses your Azure credentials.
Authentication and permissions
Permissions. Permission to list the container's blobs: a SAS with the list permission, or an Entra ID role that includes it. --all-versions needs no more: Azure lists versions, snapshots and soft-deleted blobs to anyone who may list, anonymous callers of a public container included.
Credentials, in this order: --anonymous; a SAS from --azure-sas-token, else from the source URL's query (az://account/container?sv=...&sig=...), else AZURE_STORAGE_SAS_TOKEN; an account key from AZURE_STORAGE_KEY (Shared Key); else Entra ID through the Azure SDK's default chain (environment, workload identity, managed identity, the Azure CLI and Developer CLI logins). A SAS is never recorded or shown: the source is shown as ?sig=[redacted], and a resume needs the SAS again, by --azure-sas-token or AZURE_STORAGE_SAS_TOKEN. A SAS that also names a request parameter swath sets (comp, restype, prefix, marker or another List Blobs parameter, in any case) is refused at startup with exit 2, naming the parameter. Rules shared by every object store are in Choose a source.
Flat namespace only. Before listing, swath checks that the account's namespace is flat. An account with a hierarchical namespace (Data Lake Storage Gen2) is refused with exit 2 and cause unsupported. When swath cannot tell (some anonymous callers are refused every answer), the run is refused the same way unless --azure-assume-flat asserts a flat namespace; an account that reports a hierarchical namespace is refused whatever is asserted. A credential refused during the check ends the run as an authentication failure (exit 1, cause auth). A busy or failing service is asked again until it answers or the run stops as stuck (exit 75).
Options for this source
Each row carries what List Blobs returns for every blob at no extra request (size, times, ETag, MD5, content type, blob type, access tier and its change time, archive status, last access time where the account tracks it, encryption, lease state, tag count). Three switches ask for more, at the cost of larger responses; they are part of the run's identity:
| Switch | Fills | Needs |
|---|---|---|
--azure-fetch-metadata | metadata: the user metadata as a JSON object | listing permission |
--azure-fetch-tags | tags: the blob index tags as a JSON object | a SAS with the t permission, or an Entra ID role that reads tags; never anonymous |
--azure-fetch-immutability | immutability_policy_until, immutability_policy_mode, legal_hold | listing permission |
Azure refuses a whole listing that asks for tags without the right to read them; swath then stops with that error and names the permission. --storage-class filters on the access tier (Hot, Cool, Cold, Archive), which the dataset holds as storage_class. --endpoint-url URL replaces https://ACCOUNT.blob.core.windows.net, and the account is then its first path segment (Azurite: --endpoint-url http://127.0.0.1:10000 --azure-assume-flat).
Every entry (--all-versions, schema swath/azure-versions/v1). Each version, snapshot and soft-deleted blob is a row, with version_id, snapshot, is_deleted, deleted_at and remaining_retention_days; is_latest marks the row a current listing has. A name's rows are consecutive, snapshots and versions oldest first, the current blob last. Soft-deleted blobs exist only while the account's retention keeps them, so two listings days apart differ.
Writing a dataset to Azure Blob Storage
-o az://ACCOUNT/CONTAINER/PREFIX --work-dir DIR writes the dataset to a container; the command, what appears in it and how to resume and clean up are in Writing to a bucket. What is Azure's own:
- Credentials:
--output-azure-sas-token TOKEN, else Entra ID throughDefaultAzureCredential, whatever the source's SAS, account key or--anonymous; account keys and connection strings are not taken for a destination (make a SAS from the key). The SAS is never recorded: give it to everyswath resume DIR. - Permissions: read, write and delete on the container's blobs (
sp=rwd, a container SASsr=c, or an account SAS for the blob service withsrt=o); a container listing is not needed. An Entra ID identity needs a data role on the container or its account,Storage Blob Data Contributor, or a custom role with the data actionsMicrosoft.Storage/storageAccounts/blobServices/containers/blobs/read,.../blobs/write(Put Blob, Put Block and Put Block List) and.../blobs/delete(orphans); a control-plane role such asContributoris not enough. - Endpoint:
--output-endpoint-url(the blob service base with the account as its first path segment) ishttps://ACCOUNT.blob.core.windows.netby default, is never taken from the environment, and is fixed for the run. Entra ID needs anhttps://endpoint: an emulator is reached with a SAS. - Staged blocks need no rule: blocks staged by an aborted, failed or killed upload are never a blob, and the service discards them a week after the last was staged. The orphans swath deletes are not always gone: with blob soft delete on, each is kept until the account's retention period expires; with blob versioning on, each becomes a previous version that stays billed until deleted, so add a lifecycle rule that deletes previous versions under the dataset prefixes (
daysAfterCreationGreaterThanonversion).
Limits and known differences
- Hierarchical namespaces (Data Lake Storage Gen2) are not supported.
- Azure lists names in UTF-16 order; swath reorders them into byte order, as every other source's.
- An
--includewith a literal prefix narrows the listing to it, as on S3 and GCS (what narrows the listing). When the source URL cannot carry that prefix beside its SAS, swath lists the URL as given and filters the rows, and the report says why (invocation.narrowingprovider_declined_prefix). - A run with such an include that was interrupted under 0.4.0 must be finished with 0.4.0: 0.4.0 listed the whole source URL, this release lists the narrowed prefix, so
swath resumerefuses the run (exit 2). - The report has no cost estimate for an Azure listing.
--seed-fromfails the run on a seed whose key pages end at a blob name longer than 1,024 bytes; list without it (known limits).