Batch: Jobs & Sweeps

Two tools for non-interactive work: a job runs one command to completion on an ephemeral session; a sweep (fcloud map) fans one command out over many argument bindings as a durable server-side batch.

Jobs (fcloud job)

A job is a session without a saved filesystem: image + --include files + volumes in; exit code, logs, and volume writes out. The workspace is discarded at close — a job can never be resumed, ls'd, mounted, or downloaded after it ends. In exchange, nothing syncs to storage while it runs, making it cheaper than a session for batch/CI-style runs.

job run flags: --sku SKU · --volume NAME[:MOUNT] (repeatable — the durable output surface) · --include PATH (repeatable; uploaded before the command starts) · --env K=V · --secret KEY[=VALUE] · --retries N · --min-disk-gb N · --detach (print the job id and return).

Write outputs to a volume — that's the only filesystem that survives:

fcloud volume create results
fcloud job run --sku gpu_1x_l4 --volume results:/results -- \
  python train.py --out /results/model.pt
fcloud volume files results               # outputs survive the job
fcloud volume download results model.pt   # pull them back locally

job run returns once the command has exited and its volumes are committed, so fcloud volume files right after it shows the job's output.

Jobs are never checkpoint/migrated: a lost host (spot preemption) kills the run, and the system automatically requeues it (--retries N covers real failures) — so make job commands re-runnable, and don't resubmit a preempted job yourself; follow the same job id with fcloud job wait/logs.

Rule of thumb: a session when you'll iterate interactively or want the filesystem later; a job for batch runs with declared outputs.

Sweeps (fcloud map)

A sweep runs your command N times with different arguments as a durable server-side fan-out: submit, disconnect, and the service drives every task through its own session — queueing for capacity, retrying failures, and recording exit codes and logs. No changes to your code; your script's CLI is the interface.

# Hyperparameter sweep (cartesian product: 3 × 2 = 6 tasks)
fcloud map --name lr-sweep --sku gpu_1x_l4 \
  -- python train.py --lr {lr} --bs {bs} ::: lr=1e-4,3e-4,1e-3 ::: bs=32,64

# Datagen over indices (shell brace expansion feeds the group)
fcloud map --name gen -- python gen.py --shard {} --of 500 ::: {0..499}

# One task per line from a pipe (no ::: → stdin binds {})
jq -r '.items[]' work.json | fcloud map --name batch -- python proc.py {}

Supplying arguments (GNU-parallel style)

Everything after the command is parsed for ::: markers: each one starts a group of values (space- or comma-separated), and each task runs the command with one combination substituted in.

Placeholders: {} = the single group's value · {1} {2} = group by position · {lr} = named group (write it ::: lr=1e-4,3e-4) · {i} = task index (0-based) · {n} = task count. A command with no placeholders gets each task's values appended at the end. Every task also gets env vars FCLOUD_TASK_ID, FCLOUD_WORLD_SIZE, and FCLOUD_JOB, so a script can self-shard and skip templating entirely. fcloud help map prints this reference.

Code binding

The sweep pins an immutable snapshot at submit, so task 400 runs the same bytes as task 0:

• Default: the current dir (or --code DIR) is hashed and uploaded once; each task session starts from it.
• --from <session-id>: fork a session you just verified — each task starts from that session's synced workspace (deps installed, weights downloaded) and inherits its image. Stop the session first (fcloud stop <sid> --wait) so the manifest is finalized.

Canary

Task 0 runs first and gates the rest — a broken sweep costs one task, not N. fcloud map blocks until the canary passes, then detaches (Ctrl-C detaches the watch; it does not cancel). Use --canary-smoke 'python -c "import train"' to gate on a cheap check when one task takes minutes; --no-canary/--no-wait opt out entirely.

Fan-out width

--max-parallel N caps how many tasks run at once (default 8; more tasks run in waves). Changeable on a live sweep: fcloud sweep set <name> --max-parallel N.

Outputs

Write task-keyed files (shard-{i}.jsonl) into each task's workspace and pull them all with harvest — no session-id bookkeeping, no filename collisions:

fcloud sweep harvest lr-sweep '*.json' ./out   # → out/task-<index>/<path>

Or write into a shared volume: commits are per-path merges (last writer wins per file), so key every task's output by {i}.

Tracking & recovery

fcloud sweeps                             # all sweeps: state + done/run/fail counts
fcloud sweep status <name>                # counts + failures clustered by error
fcloud sweep status <name> --watch        # repainted until terminal
fcloud sweep logs <name> [--task N]       # durable output (defaults to the exemplar failure)
fcloud sweep retry <name> [--all]         # re-run only failed tasks
fcloud sweep cancel <name> [--remaining]  # stop; --remaining keeps running tasks
fcloud sweep wait <name>                  # block until terminal; exit 0 on success

A task with no host yet keeps its queued session and is reported as blocked on capacity — that's the fleet scaling up, not your sweep breaking. Spot-preempted tasks requeue without consuming the retry budget.

--webhook URL POSTs the status document on canary pass/fail and completion — agents should submit with --no-wait --json and wake on the webhook instead of polling.

SDK mirrors

# Job
job = project.job("python train.py --out /results/model.pt",
                  sku="gpu_1x_l4",
                  volume_mounts=[{"name": "results", "mount_path": "/results"}],
                  include=["./src"], retries=1)
log = job.wait()                   # ProcessLog: exit_code, output

# Sweep — the *type* of args picks the combinator (no ::: syntax in the SDK)
sweep = client.map("python train.py --lr {lr} --bs {bs}",
                   args={"lr": [1e-4, 3e-4], "bs": [32, 64]},   # 4 tasks
                   name="lr-sweep", sku="gpu_1x_l4", code_dir=".")
sweep.wait_canary()
sweep.wait()