Logs & Streaming

Every process's output is captured to a durable log that outlives the host — you can follow it live, fetch it after the fact, or read it long after the session is gone.

Follow a process live (CLI)

fcloud spawn --on s-abc123 python train.py     # prints proc-456
fcloud logs s-abc123 proc-456 --follow

The default stream is combined (stdout + stderr) — prefer it. Most ML tooling (tqdm progress bars, HuggingFace/TRL logs) writes to stderr, so --stream stdout on a training job often looks empty even though it's running fine.

Fetch logs after the fact

fcloud logs s-abc123 proc-456 --output all                    # the whole log
fcloud logs s-abc123 proc-456 --output head --output-bytes 128k
fcloud logs s-abc123 proc-456 --stream stderr --output all

If an exec --json result has stdout_truncated: true, don't rerun the command — fetch the durable log with --output all instead.

After a session stops, the same logs are browsable as files under a virtual _logs/ directory: fcloud ls s-abc123 _logs, then fcloud download s-abc123 _logs/exec-proc-456.log.

Block until done

fcloud wait reads the durable log, so it survives host migration and works even after the host is gone. It exits with the process's own exit code — the reliable way to detect a failed background job:

fcloud wait s-abc123 proc-456 --timeout 0   # 0 = wait indefinitely

Stream from the SDK

Session.watch(proc) streams output in real time. Each event is a dict with type ("stream_event" or "stream_end"), data (the output text), and stream ("stdout" or "stderr"):

with project.session(sku="gpu_1x_a10g") as s:
    proc = s.spawn(["python", "train.py", "--epochs=100"])
    for event in s.watch(proc):
        print(event.get("data", ""), end="")

Fetch durable logs programmatically with s.logs(...):

log = s.logs("proc-456", output_range="all")
print(log.output)

stderr = s.logs("proc-456", stream="stderr", output_range="all")