Run on a schedule
A short guide for whoever schedules and supervises swath: the nightly command, what to do with each exit code, and writing the dataset to a bucket from a machine that may not last. Listing a bucket once, start to finish, is in Run large listings; every option is in the command reference.
The nightly command
swath list s3://bucket/prefix/ -o /data/inv/2026-11-08 \
--seed-from /data/inv/2026-11-07 \
--max-duration 6h-o DIRwrites a sorted Parquet dataset.DIRmust be new or empty. Write each night to a new directory and point readers at it once itsmanifest.jsonexists (Overwrite and refresh).--seed-fromnames the previous night's dataset (warm start). Leave it out on the first night.- The exit code and the outcome in
DIRare the outcome of record:DIR/run.jsonbeside a committed dataset (itsoutcome), elseDIR/error.json, each withclass,exit_codeandresumable. Anerror.jsonwhoseattempt.started_atis before the attempt began is an earlier attempt's: one that was killed, or could not write it, left it (the outcome of record). After exit 0 or 3 the dataset is committed even if neither file is there: the commit removeserror.json, andrun.jsonis written after the commit, so a failed write leaves it out with the warningrun_report_unwrittenon stderr, without changing the exit code;manifest.jsonthen records the coverage. To keep each attempt's full report, a resume's included, add--report FILEat a fresh path for each attempt, outside the output directory. --max-durationstops a run that takes too long. Give every unattended run one. Set the supervisor's own kill deadline above--max-durationplus 20 s.
What the supervisor does with the exit code:
| Exit | Do |
|---|---|
| 0 | done |
| 3 (partial) | accept the partial output, or fix the permissions and run again |
| 129, 130, 143 (signals) | swath resume DIR |
| 75 (stuck) | resume once; a second 75 in a row on the same output needs a person |
| 74 (disk full) | free space or move the directory to a larger disk, then swath resume DIR |
| 124 (deadline) | resume only within a separate budget of attempts or total time; past it, a person |
| 1 (failed) | a person fixes the cause; then swath resume DIR when the outcome is resumable, else a new run |
| 2 (usage, refused) | a person fixes the command; with no outcome or report, a crash: resume as after a kill |
The reference table, what a supervisor does with each code, adds the cases this short form leaves out: 75 with cause unreachable or class unresolved, and 74 with cause spill_budget.
The rest of a scheduled run has its own pages:
- Watch and tune a run: three flags on a small runner, a healthy run report and three bad patterns.
- Stop, resume and refresh: resume after a kill, where the workspace and spill are, and the warm start for repeat listings.
- Size a run: disk sizing and memory.
Writing to a bucket
To keep the dataset in a bucket instead of on the runner, give -o a bucket prefix and --work-dir a local directory:
swath list s3://bucket/prefix/ -o gs://inventory/2026-11-08 --work-dir /data/work/2026-11-08 \
--max-duration 6h-o s3://BUCKET/PREFIX writes to S3 and -o az://ACCOUNT/CONTAINER/PREFIX to Azure Blob Storage the same way. The parts are uploaded as they close, and no Parquet part is stored locally. The closing line names the prefix, and swath show DIR, given the work directory, prints the dataset's summary.
What it needs
A work directory (
--work-dir DIR), local, new or empty, one per run. It holds the run's lock, journal and spill, never a part, so it needs room for spill and the run's metadata, not for the dataset (disk sizing). Without--work-dirthe command is refused (exit 2). Every resume of the run uses the same directory; if the machine may not last, put it on a disk that outlives it (below).Credentials and permissions on the destination, its own, never the source's (a source's
--anonymousdoes not reach it):Store Credentials (default) Permissions Unfinished uploads Amazon S3 --output-s3-profile NAME(the AWS SDK's default chain)list the bucket; put, get and delete objects (and their versions in a versioned bucket); abort a multipart upload and list its parts kept until a lifecycle rule aborts them: the rule is required Google Cloud Storage --output-gcs-credentials-file FILE(Application Default Credentials)create, get and delete objects expire on their own Azure Blob Storage --output-azure-sas-token TOKEN, never recorded (Entra ID)read, write and delete blobs expire on their own The exact permission and role names, and each store's region and endpoint rules, are on the store's page.
--output-s3-regionand--output-endpoint-url(an emulator, or an S3-compatible store) are fixed for the run; the credentials are resolved again on every resume.Logging: swath turns the storage SDKs' own logging off, so no SDK setting logs a token, a signed assertion or a session URL; with
GOOGLE_SDK_GO_LOGGING_LEVEL=debug, a GCS destination may still log request lines: the method, the object name and a URL without credentials.
What appears in the bucket, and when
- During the run, parts appear under
PREFIX/data/RUN/ATTEMPT/as they close. There is noPREFIX/manifest.jsonyet, so the prefix holds no dataset: a reader that finds no manifest must not read the parts. - At the end, the run creates
PREFIX/manifest.json, only if it is absent: that creation is the commit, and the dataset exists from then on.run.jsonand, for an object-store source,cuts.txtfollow beside it. The uploads happen inside the run, so the bucket's bandwidth is part of its wall time. - Readers take the parts from the manifest, never from a listing of
data/(a dataset in a bucket). - One prefix per run. A prefix that already holds a
manifest.jsonis refused before the run starts (exit 2), as are--overwriteand a prefix that overlaps the listed source: a dataset in a bucket is never replaced. Write each run to a dated prefix and let readers find the newest by its manifest. - Two runs, one prefix. Two runners sharing one work directory are refused by its lock. Two work directories writing one prefix race at the commit and exactly one wins; the other exits 1 (
lost_race) and deletes its parts.
Interrupted, then resumed
As for a local dataset, swath resume DIR (the work directory) continues the run until exit 0; the exit codes mean what the table above says. Before it lists, the resume downloads every part the journal records, which the resumed dataset keeps, and checks its version, size and SHA-256, so everything published so far is read once, at the bucket's read rate and price. Uploads the journal does not record are not read; they are orphans, cleaned up after the commit (what is left behind).
- An unresolved commit (exit 75, class
unresolved: the manifest's creation failed in a way that does not say whether it happened) keeps every part: resume the work directory, which creates the very same manifest again. A local tree written to a bucket keeps no journal and cannot resume: check the prefix for a manifest before deleting anything under it. - An Azure destination's SAS is never recorded: give
--output-azure-sas-tokento every resume (no variable stands in for it), or the resume is refused. The source's secrets are given again too (resume it). - A lost work directory cannot be resumed, and the parts the run uploaded are not reused: start again at a new prefix, and clean up the old one (what is left behind).
On a preemptible or Spot machine
To survive a preempted VM, put --work-dir on a disk that outlives the VM (one not deleted with the instance), attach it to the replacement machine and run swath resume DIR at its mount point there. The resume replays the recorded command with DIR as the work directory and the recorded bucket and prefix. Nothing binds the work directory to a machine or a path: the lock is a flock, and the run's identity (source, schema, part size, filters, destination) names neither the machine nor a local path. On the new machine:
- Paths the command recorded are replayed as written, a relative one against the new working directory.
--gcs-credentials-file,--output-gcs-credentials-fileand a--ca-bundle(unless the command also has--no-verify-ssl, which skips it) must exist there, or be given again toswath resume; a--seed-fromthat is missing only warns, and the resume lists without it. - The spill cap is a positive
--spill-budget, the run's or one given to the resume. Absent or 0, it is derived from the new disk: its free space after the reserve, the spill files the resume keeps counted as free, and it follows the space available from there. Spill files that do not fit under it are listed again (how a resume reuses them).
What is left behind
- A committed run cleans up after itself: after the commit it deletes the parts its earlier attempts uploaded that the manifest does not name. A part whose upload it could not confirm is left, and counted in the report's
output.remote.orphans_unproven. A part it could not delete is counted inorphans_left, and the run warns that its cleanup is unfinished (aftermath_unfinished): the dataset is committed all the same, andswath resume DIRretries the cleanup and the bucket'srun.json. - An abandoned run (never resumed to its commit) leaves its uploaded parts under
PREFIX/data/RUN/with no manifest;DIR/.swath/remote/allocatednames them. Delete them, or set a lifecycle rule on that prefix. A run whose work directory was lost is abandoned the same way, without that list. A new run at the same prefix does not clean them: it is not refused (only amanifest.jsonis), and its manifest names only its own parts, so start it at a new prefix. - Lifecycle rules: never let a lifecycle rule or a cleanup job delete or replace objects under a committed dataset's prefix: the manifest would name parts that are gone, and swath does not notice. Expire objects only under the prefixes of runs you have abandoned. A rule that aborts incomplete uploads is safe, and on S3 required (the rule).
Limits
swath show DIRtakes the work directory, not the bucket URL.swath du --from-datasetrefuses a dataset in a bucket and its work directory (exit 2).--seed-fromreads a local path: to seed the next run, copy the dataset'scuts.txt(the cut file).- Datasets written to GCS have run against the real service (measurements); writing to S3 and to Azure Blob Storage is tested against emulators and in-process fakes only.
- Each store's own limits are on its page.
The report's output.remote holds the commit's outcome, the uploads and their retries, the parts a resume verified and the orphans deleted or left (run report).