swath documentation
swath lists object stores and filesystems in parallel, prints size totals and text rows, and writes sorted Parquet inventories. Releases are published on the releases page (Install).
Start
- Start here: pick a job, and when swath fits.
- Install: release binaries and their verification, the container image, a source build.
- Your first inventory: total a local directory, list it, save and inspect an inventory.
- Understand your results: scope, sizes, coverage, observation time, live and saved results.
Use
- Sources: Amazon S3 and S3-compatible stores, Google Cloud Storage, Azure Blob Storage, and local and mounted filesystems: URLs, credentials, permissions, endpoints, requests.
- Output: text rows (table, TSV, JSONL), the fields of
swath du's rows, reading them in DuckDB, when to write a dataset. - Total a source:
swath du, the view of where the bytes are, totals per prefix, totals from a saved dataset, and how sizes, hard links and unreadable entries are counted. - Filter rows: globs, regular expressions, sizes, times and storage classes.
- Run large listings: list a bucket of your own start to finish: the command, what it prints, stopping and resuming it, where the result is.
- Size a run: the disk, memory and workers a bucket needs, and what it costs.
- Watch and tune a run: progress, the closing lines, throttling, memory and disk, a stuck run.
- Stop, resume and refresh: signals,
swath resume, overwrite and seeding a refresh. - List every version: versions, generations, snapshots and delete markers.
- Run on a schedule: a scheduled run under a supervisor, a dataset written to a bucket, preemptible machines.
- Troubleshooting: symptom, cause, fix; the known limits.
Read the output
- Query an inventory: DuckDB, pandas and Spark, checking that an inventory is complete, verifying it.
- Dataset format: its files, schemas, manifest and coverage. From Go, the
readermodule.
Reference
- Commands: swath, swath list, swath du, swath resume, swath show, swath completion.
- Exit codes, Run report and progress file, Log events.
- The Go library (experimental): page, look up and walk a bucket, container or directory from a Go program.