Watch and tune a run
While a listing runs, swath shows its progress on the terminal and in a file beside the output, and when it ends it says what it did. This page says which values to look at, and what to do when a run is throttled, short of memory or disk, or stuck. How much disk and memory to give a run before it starts is in Size a run.
Is the run making progress
On a terminal swath draws one progress line on stderr and redraws it every second. This listing of a large public bucket stops itself after five seconds:
swath list s3://noaa-gestofs-pds/ --anonymous --s3-region us-east-1 -o noaa-all --max-duration 5sAt three seconds, on a terminal 150 columns wide, the line read:
listing · 2,026,002 objects · 69.2 TiB · 3.0s elapsed · 832,641 keys/s (avg 675,334) · 0 parts · 64 in flight · 2,095 API callsFrom the left: what the run is doing (listing; verifying while a resume checks the parts it keeps), the objects listed so far (a local directory's are entries; a resume adds (+N recovered)), their total size, the time elapsed, keys a second now and on average, the parts published, requests in flight, API calls made, and an estimate of their cost where the provider's list price is known (not for Azure Blob or a custom endpoint), which this terminal was too narrow to show: a narrow terminal drops whole fields from the right. There is no percentage: swath cannot know how many objects are left.
A run is making progress while the objects and API calls rise. The parts can stay at 0 for a long time, as here: swath list writes its rows in key order, so the output waits until the start of the key space has been listed, and the rows listed ahead of it wait in memory and then on disk meanwhile (short of disk). swath du counts without ordering and never waits so.
Where the line appears:
- Off a terminal (a log file, a pipe) there is no line unless you ask:
--progressappends the same record as a plain line every 30 s, and--progress-interval 10ssets the interval (at least 1 s) and implies--progress. A log never receives control sequences. -qturns the automatic line and the warnings off;--no-progressturns progress off everywhere.- While
swath listwrites rows to the same terminal there is no line, and--progressis refused: send the rows to a file or a dataset.swath dudraws the line while it lists and erases it before its table. --color auto|always|nevercolors the line and the closing lines;autohonorsNO_COLOR.
The progress file
A dataset run (-o DIR) also rewrites DIR/.swath/progress.json every 2 seconds, for a script or a supervisor. Read in another terminal while the run above goes (with jq, if you have it; any JSON reader will do):
jq '{state, rows_fetched, object_bytes, parts_written, part_age_ms, advance, throttles, inflight, limit}' \
noaa-all/.swath/progress.json{
"state": "listing",
"rows_fetched": 2026002,
"object_bytes": 76044354827022,
"parts_written": 0,
"part_age_ms": 3000,
"advance": 2078006,
"throttles": 0,
"inflight": 64,
"limit": 64
}| Field | What it says |
|---|---|
state | listing; draining once listing is done and the rows it holds are being written; publishing; then done or failed |
rows_fetched, object_bytes | rows listed so far, and the objects' total size |
parts_written, part_age_ms | parts published, and the time since the last one was published (or since the start) |
advance | rises whenever rows are listed or written, parts published or a resume's parts verified: flat means nothing moves (stuck) |
throttles, limit, inflight | throttle answers so far, the request limit (it starts at --workers and falls when the service throttles), and requests in flight: throttling |
order.spill_bytes, order.spill_max_bytes, order.spill_pressure_since | rows waiting on disk for the output, their cap, and since when listing has waited on a full cap (present only while it does): short of disk |
Every field is in Run report and progress file. The directory above holds a resumable run; delete it when you are done (rm -r noaa-all).
When the run ends
swath prints its closing lines on stderr:
swath list s3://noaa-gestofs-pds/stofs_2d_glo.20230113/ --anonymous --s3-region us-east-1 -o stofs complete: 5,252 objects, 383 GiB in 1.0s → stofs (1 part, 121 KiB)
next: swath show stofs · swath du --from-dataset stofsThe first line is the verdict, what was listed and its size, the time (and from 2 s the rate), and the output written. next: is what to run next: swath show and swath du --from-dataset for a committed dataset, swath resume DIR for a run that can resume. The closing lines print for runs over 1.5 s, runs that write output and runs that do not complete; --no-stats and -q drop them. --stats prints them always and adds the run's statistics: the requests with the mean and peak in flight, throttled N · retried N when there were any, the cost estimate where the list price is known, and the process's peak memory (a sample).
The five-second run above ended this way:
stopped: --max-duration reached; 0 saved, 3,603,746 listed in 5.0s · 720,605 keys/s → noaa-all
next: swath resume noaa-all
swath: deadline: --max-duration reached (swath resume noaa-all --max-duration 0 continues it without a deadline; a plain resume keeps the recorded one)saved is the rows in published parts, which a resume keeps; listed is what this run listed. The verdicts and what to do:
| Verdict | Exit | What to do |
|---|---|---|
complete | 0 | nothing |
partial: N unreadable entries | 3 | Entries could not be read |
stopped: --max-duration reached | 124 | swath resume DIR, or run it again |
stopped: no progress within --stall-timeout | 75 | Is the run stuck |
stopped: the endpoint cannot be reached | 75 | The endpoint cannot be reached |
stopped: --spill-wait-timeout reached, stopped: the disk is full | 74 | Short of disk |
interrupted (Ctrl-C), interrupted: SIGTERM, interrupted: SIGHUP | 130, 143, 129 | swath resume DIR |
failed | 1 | the error line says why: Troubleshooting |
A command that stops before it lists can print its error line alone, without a verdict; the exit status says which kind it is: 2 for a command line or an output refused before anything ran (an unknown flag, a non-empty -o directory), 1 for a failure such as a bucket that does not exist. Only a dataset run of an object store keeps the journal a resume needs; a local directory, a text file and swath du are run again. Every code is in Exit codes.
The run report. A committed dataset has its report in DIR/run.json; --report FILE writes it for any run, on failure too. A few fields answer whether the run was healthy:
jq '{class: .outcome.class, complete: .outcome.complete, rows: .output.rows_committed,
requests: .requests.http_attempts, throttles: .requests.throttles,
peak_rss: .process.peak_rss_bytes, wall_ms: .timings.wall_ms}' stofs/run.json{
"class": "ok",
"complete": true,
"rows": 5252,
"requests": 32,
"throttles": 0,
"peak_rss": 63320064,
"wall_ms": 968
}A healthy run is ok and complete, with few or no throttles and a peak memory well under the machine's. The fields are in Run report and progress file; its diagnostics, which explain a slow run in depth, are read in Reading a run report.
Throttling
What you see. The service answers some requests with "slow down" (S3's 503 SlowDown, a 429): in progress.json throttles rises and limit falls below --workers, the rate drops, and --stats ends with throttled N. This run went through a local test server that answered 30 % of requests with 503 SlowDown; progress.json every 10 seconds:
{"state":"listing","rows_fetched":785000,"throttles":382,"inflight":2,"limit":2,"part_age_ms":4000}
{"state":"listing","rows_fetched":899000,"throttles":436,"inflight":1,"limit":1,"part_age_ms":8000}
{"state":"listing","rows_fetched":1009000,"throttles":486,"inflight":1,"limit":1,"part_age_ms":14000}and its closing lines with --stats:
complete: 2,000,000 objects, 8.18 TiB in 1m18s · 25,542 keys/s → thr (3 parts, 8.36 MiB)
next: swath show thr · swath du --from-dataset thr
3,479 API calls · in flight avg 1.55 · peak 63
throttled 1,062 · retried 1,062
peak RSS 276 MiBWhat it means. Requests to S3, GCS and Azure Blob pass an adaptive limit that starts at --workers, falls when the service keeps throttling and rises again as answers come back. Occasional throttles do not lower it. Throttled requests are retried: they cost time, not rows.
What to do.
- A few throttles, with
limitnear--workers: nothing. - The store's request quota is shared with other clients (other jobs, several swath runs): set
--request-rate N, requests a second for this run, below your share of it. - The service throttles at the default 64 workers: lower
--workers, which is also the limit's ceiling. - The run crawls with
limitat 1 or 2 for minutes: turn the adaptive limit off and set a fixed rate,--no-adaptive-limit --request-rate N. Without the limit each worker waits on one page at a time, and swath's own probes of the key space run beside them, so more than--workersrequests can be in flight.
Requests that keep failing with the same cause (not only throttles) print a warning once they have done so for 10 seconds without any request answered successfully: A warning repeats about failing requests.
Short of memory or disk
Memory
What you see. peak RSS in the --stats lines and process.peak_rss_bytes in the report. swath sets no memory limit of its own: a run that outgrows the machine or the container is killed by the system.
What it means. --sort-buffer (by default 1/16 of memory, at most 1 GiB; 128 MiB when no memory limit is found) bounds the rows a sorted run holds between listing and writing. It is not a cap on the process: pages in flight, the Parquet writers (admitted against their own 1/16 of memory) and the Go runtime come on top. Memory is the least of the physical memory, the container's limit and GOMEMLIMIT.
What to do.
Peak RSS near the machine's memory: lower
--sort-buffertogether with--workersand--encoders; a smaller sort buffer alone spills more rows to disk without always lowering the peak.Many rows spilled (the report's
diagnostics.resources.spill.rows, the rows written to spill, near the rows listed) and memory free: raise--sort-bufferso more rows wait in memory.On a machine of one vCPU and 1 to 2 GB, start from the defaults: on one core the sorted output is the limit, so keep the default
--workers(workers). Peak memory has been measured only on larger machines (measurements): read the peak on a representative run first.A warning that spill is in memory:
textswath: warning: rows are spilling to inv/.swath/spill, which is on tmpfs: they are held in memory (outside the peak RSS); write the output to a diskThe output directory (for rows on stdout,
TMPDIR) is on a memory filesystem, a common/tmp: put it on a disk.
How much memory to plan for: Memory.
Disk
What you see. swath list keeps the rows it listed ahead of its output in spill files on the output's filesystem (DIR/.swath/spill/ for a dataset; beside a text file; in TMPDIR for rows on stdout); a large bucket spills almost every row once. By default spill may use half the filesystem's free space (all of it for a dataset in a bucket), after a reserve of at least 1 GiB. When it is full, listing slows to the pace of the output, progress.json shows order.spill_pressure_since, and after 2 minutes swath says so, again every 30 minutes while it lasts. This run went through a local test server with a deliberately small --sort-buffer 8MiB and a fixed --spill-budget 2MiB, which its remedy names:
swath: spill cap too small: listing has waited on the output for 2m1s: the spill cap cannot hold the rows listed ahead of it (spill 2.00 MiB, cap 2.00 MiB, free-space reserve 1.00 GiB); the output writes 6,403 rows/s; stop the run and resume it with more disk or a larger --spill-budget (swath resume sp2 --spill-budget SIZE), or let it run: it finishes, slowlyWith the default cap the remedy reads the cap follows the spill disk: grow the disk, or free space on it, and listing continues.
What it means. The run does not fail: left alone it finishes at the output's pace, which on a very large bucket can be hours or days.
What to do.
With the default cap, grow the filesystem under the running process (a cloud disk resized and its filesystem grown online) or free space on it: within seconds the cap rises and listing continues.
With
--spill-budget, which is fixed, stop the run (Ctrl-C) and resume it with a larger one on a disk that has the room:swath resume DIR --spill-budget 200GiB. A plainswath resume DIRkeeps the recorded cap.On an unattended run,
--spill-wait-timeout 30mstops the run once listing has waited that long, exit 74, resumable. The same test with--spill-wait-timeout 10s:textstopped: --spill-wait-timeout reached; 0 saved, 1,022,591 listed in 12.0s · 85,195 keys/s → sp next: swath resume sp swath: spill cap too small: listing has waited on the output for 12s: the spill cap cannot hold the rows listed ahead of it (spill 1.99 MiB, cap 2.00 MiB, free-space reserve 1.00 GiB); the output writes 43,825 rows/s; stopped by --spill-wait-timeout 10s: resume it with more disk or a larger --spill-budget (swath resume sp --spill-budget SIZE)A repeat listing of the same bucket spills far less when it starts from the previous dataset (warm start).
A part, the journal or the manifest that cannot be written because the filesystem or the quota is full stops the run with
stopped: the disk is full, exit 74: The disk filled. Another write failure, such as a read-only filesystem or a permission refused, isfailed, exit 1.
How much disk to plan for: Disk sizing.
Is the run stuck
What you see. The objects stop rising and the rate reads 0. While every request fails, the rate gives way to retrying: and the cause. At half of --stall-timeout swath warns. This run went through a local test server that stopped answering after a second and a half, with --stall-timeout 1m and --progress --progress-interval 15s:
listing · 743,000 objects · 3.04 TiB · 2.0s elapsed · 371,500 keys/s (avg 371,500) · 0 parts · 64 in flight · 887 API calls
listing · 743,000 objects · 3.04 TiB · 18.0s elapsed · 0 keys/s (avg 41,278) · 0 parts · 64 in flight · 887 API calls
listing · 743,000 objects · 3.04 TiB · 34.0s elapsed · retrying: context deadline exceeded · 1 part · 64 in flight · 951 API calls
listing · 743,000 objects · 3.04 TiB · 50.0s elapsed · retrying: context deadline exceeded · 1 part · 64 in flight · 951 API calls
swath: warning: S3 127.0.0.1:19424: context deadline exceeded (retrying; 65 attempts; the run stops as stuck after 1m0s without progress)
swath: warning: no progress for 30s (the run is stopped as stuck at 1m0s)
listing · 743,000 objects · 3.04 TiB · 1m06s elapsed · retrying: context deadline exceeded · 1 part · 64 in flight · 1,015 API calls
listing · 743,000 objects · 3.04 TiB · 1m22s elapsed · retrying: context deadline exceeded · 1 part · 64 in flight · 1,015 API calls
swath: warning: no progress for 1m2s (stuck_timeouts): stopping the run as stuck
swath: warning: goroutine dump written to stall/.swath/stacks-stuck.txt
stopped: no progress within --stall-timeout; 503,000 saved, 743,000 listed in 1m36s · 7,738 keys/s → stall (1 part, 2.10 MiB)
next: swath resume stall
swath: stuck: no progress within the stall timeout (stuck_timeouts); every request was failing: S3 127.0.0.1:19424: context deadline exceeded (192 attempts)What swath does. Rows listed or written, parts published and pages of entries answered count as progress; failed, throttled and hung requests do not, nor a service that answers only empty pages. After --stall-timeout (10 minutes by default) without progress, swath stops the run with exit 75 and writes every goroutine's stack to DIR/.swath/stacks-stuck.txt. The cause in the last line, in DIR/error.json and in the report's outcome.cause says what kept rising: stuck_throttle, stuck_timeouts, stuck_empty_chain (the service answered only empty pages) or stuck_no_rows (exit 75). A part published while requests hang counts too: above, the part at 30 seconds restarted the clock.
What to do.
- Resume it once:
swath resume DIR. - Stuck again on the same output: keep
stacks-stuck.txtand the report, and report the problem. - A service that is slow to answer for minutes at a time: raise
--stall-timeout. - An endpoint that nothing in the run has ever answered, with every connection refused, its host not found or no route to it, stops sooner: after 30 seconds by default, at most half
--stall-timeout(exit 75, causeunreachable). Timeouts and reset connections do not count: they wait for--stall-timeout. The endpoint cannot be reached.
The objects can also keep rising while the parts stay flat and part_age_ms grows: one slow range at the start of the key space holds the output back. The run goes on listing ahead, spilling to disk, and writes its parts once that range answers; watch the disk (short of disk).