Skip to content

Observe and control runs

Run state persists when an observation client leaves, and reconnecting does not change the run lifetime.

Browser

Run zeroshot ui, then select a run in Runs. The read-only graph uses that run's admitted definition. Select a node for its inputs, outputs, decisions, and transcript; with the details panel open, select the empty canvas or the first link above a node's name for the run's own details. Use the timeline to replay its history. Active runs update live. Seeking pauses following; Go live returns to the latest event. Scrolling a transcript keeps the graph live and follows new output only when you reach the bottom.

For a direct target, open /ui/ on its configured public origin. Closing either browser view leaves the run active.

Foreground and detached submission

zeroshot run follows the durable event stream by default and writes NDJSON. Add --detach to return after submission:

zeroshot run \
  --title "Update parser" \
  --template software-change --delivery none \
  --input input.json \
  --uniform-runtime-config runtime.json \
  --detach

Ctrl-C closes the terminal stream and leaves the controller running.

Inventory and current status

These commands are read-only and return JSON:

zeroshot list
zeroshot status RUN_ID

A run ID belongs to the controller or target that created it. Pass --target NAME when reading one from a named target. Finished status separates graph-authored terminalResult.output from optional runtime-confirmed metadata.deliveryReceipt; graphs without delivery omit the receipt.

When the graph ends, status and list expose its recorded outcome as status.graphResult, using the same shape as terminalResult, while phase remains running during the Active Memory postmortem and finalization. For example:

{
  "phase": "running",
  "activeExecutions": [],
  "graphResult": { "status": "failed", "reason": "worker_failed" }
}

graphResult is absent before the graph ends and after the run finishes, when terminalResult is the only result. It remains visible if finalization enters stopping. watch emits the result as soon as it is recorded, and foreground run prints that event and keeps waiting for finalization. The log records the graph result once and says when the postmortem is running. Exit codes still follow the final run result.

The recorded graph result does not guarantee the final run result. A force stop during the postmortem cancels and drains that work and preserves the recorded graph result; ordinary postmortem failures and controller restart after confirmed cleanup preserve it too. A runtime or cleanup failure during finalization can instead prevent completion or end the run with runtime_failed or runtime_lost. Read terminalResult once the run is finished.

Local inspection (status, list, checkpoints, vectors, watch, logs and attach) reads retained history without taking recovery ownership or writing to the run directory. Finished and suspended runs need no controller and retain their usual exit codes.

Each status.activeExecutions entry can include startedAt, the ledger recording time of that execution's NodeStarted. Watch notifications also include optional recordedAt for the event that produced the notification. Match the opaque execution selector across active sets to identify starts and settlements, including retries and simultaneous map items. Status remains a summary; completed execution timing is available through watch replay and the history API.

These times are Unix milliseconds recorded by the ledger, not precise provider start/stop measurements. Events in one append transaction share a time, and cursor alone defines ordering and reconnection boundaries. Missing times in older ledgers stay omitted. Imported history carries its import time, not the time the original work ran. Log timestamp retains its producer meaning.

For an unfinished local run whose controller does not answer, status and each list entry keep the recorded status and add an optional entry beside it:

{ "controller": { "state": "lost", "observedAt": 1794470707000 } }

lost means the original controller process is confirmed absent and its original lease is released. unknown means the evidence is insufficient, including an unreachable endpoint, a stalled process, a reused PID, a replaced lease or a permission failure. observedAt is the time of this check in Unix epoch milliseconds, not the time the controller died. The entry is omitted when the controller answers and for finished or suspended runs. checkpoints also includes it when applicable. These three commands exit 0 after a successful read, including when controller availability is lost or unknown.

Local controller requests wait at most 10 seconds for an answer. This includes the status checks before resume and checkpoints, and the initial exchanges for watch, logs and attach. list checks controllers with bounded concurrency under one shared deadline. Runs that could not be checked within that budget retain their recorded status with an unknown controller. An established stream may stay quiet indefinitely. Controller handoffs keep their separate retry budget.

For an unfinished run, watch, logs and attach print the recorded part, then exit 1 when the local controller is unavailable. Their error names zeroshot resume RUN_ID and zeroshot force-stop RUN_ID. Inspection does not record a terminal result: an unclaimed lost run keeps its recorded status on every read. Only these explicit controls confirm loss under the controller lease before recording runtime_lost; they refuse to take over a controller that still owns its lease or whose loss cannot be confirmed. steer and suspend never take over: on a lost run they exit non-zero, name the two commands and leave the recorded state unchanged.

Token usage

metadata.tokenUsage contains run totals summed across agent invocations, including retries, correction turns and auxiliary agents such as postmortem specialists. inputTokens means uncached input only for every harness. Cache reads and cache writes appear only in their respective cache counters, so the four token counters do not overlap and can be added.

Field Meaning
inputTokens Uncached input tokens, excluding cache reads/writes.
outputTokens Output tokens reported by the harness.
cacheReadInputTokens Input tokens reported as read from a prompt cache.
cacheCreationInputTokens Input tokens reported as written to a prompt cache.
complete Whether every invocation reported usable token usage.

When complete is false, the totals retain known usage but at least one invocation's usage is missing or unusable. An omitted cache counter means at least one invocation did not report that counter; its run total is unknown, not zero. Cache counters can be omitted even when complete is true. Script steps record zero for all four counters and preserve known cache totals. Previously recorded runs retain their original numbers.

Failure evidence and mapped executions

Failed terminalResult.context separates the known runtime cause (primary) from recent recorded execution errors (recorded) and verifier results (verifiers). Each verifier result is the latest recorded result for that graph node and map path. Its bounded, credential-redacted message contains the graph's actual signals and diagnostic; the runtime assigns no general meaning to their names or labels. A later result replaces an earlier result in this summary, while logs retain both. Recorded evidence can include recovered attempts or parallel work and does not prove that a result caused the terminal failure. An independently known runtime cause remains in primary.

exhaustedLoops identifies completed exhausted loop visits with node, completedIterations, and an optional mapIndices path. The count is the actual number of completed loop rounds, including all work within each round; execution retries and verifier completion counts do not add rounds. Completed exhausted loops in settled branches of an unreachable parallel join are retained. Discarded race and map-item paths do not contribute loop entries or consume the summary bound. A converged loop does not appear as exhausted. The summary keeps at most five entries in each collection. incomplete: true means evidence was omitted or could not be retained. This warning persists even if later loop visits converge and remove all retained loop entries. Full details may still be available in logs. Failure context freezes with the main outcome, before auxiliary work.

Each diagnostic can carry an execution object with the node name and stable opaque execution selector, and a cursor pointing to its retained log. Missing cursors cannot be used for log navigation. Status/watch activeExecutions and execution-associated log notifications also expose mapIndices: zero-based indices, outermost map first. For example, [1] is the second item and [1, 2] is its third nested item. Outside maps the field is omitted (older data also reads as an empty path). Indices supplement execution identity without changing its opaque selector. Successful terminal results keep their existing representation.

Resume event and log streams

Codex, Claude, Copilot and Pi record native tool calls and completions in both logs and node transcripts by default, including tool identity, available call IDs, arguments, reported status and result excerpts. Missing status remains unknown. Known credentials are redacted before each entry is shortened. Each complete call entry and each completion entry is limited to 8 KiB, including its header and any visible truncation marker; large payloads retain their head and tail. Intermediate cumulative updates are omitted. These limits apply only to logs, so storage grows with tool count; they do not limit provider inputs, agent-visible results or total run storage.

watch follows status events; logs follows safe provider and controller log records. Both accept an opaque cursor and resume strictly after it:

zeroshot watch RUN_ID --after STATUS_CURSOR
zeroshot logs RUN_ID --after LOG_CURSOR

Cursors are opaque, so store the last fully consumed value and pass it back unchanged. logs --execution EXECUTION_REF restricts output to one execution selector from status.

watch exits 0 when the run succeeded or suspended. When the run finished failed, it prints the same failure summary on stderr as a failed foreground run and exits 1, so a script can start a run with --detach and wait for its result with zeroshot watch RUN_ID. logs exits 0 when its stream ends, whatever the run's result. The CLI reference lists the exit status of run, watch and logs.

The producer records each log timestamp as a positive JavaScript-safe Unix epoch millisecond, and durable replay preserves it.

JSON error codes

Set ZEROSHOT_ERROR_FORMAT=json to receive a zeroshot.error/v1 object on stderr when a CLI command fails. Successful stdout and NDJSON events keep their existing format. The object contains kind, code, message, optional path and node context, and an object-valued details. Use code to distinguish a failed run or a request that needs correction from a transport outage:

Condition Kind Code Details
A followed run failed (run or watch) run_failed run.failed runId
Unknown registered target name invalid_request request.invalid {}
Local registry has no workspace-recovery authorization invalid_request request.invalid {}
Workspace-recovery authorization does not match the original run invalid_request request.invalid {}
Local workspace-recovery authorization is malformed invalid_request request.invalid {}
Refusal relayed from the run's controller or a target target invalid_request Target's details, else {}
Target transport failed target target.unavailable target and origin when known

run.failed reports a terminal outcome; retrying observation does not restart that run. Inspect its status and logs before selecting a recovery action. Correct the request or local registry state for the listed request.invalid refusals. A relayed invalid_request refusal says what to correct; do not retry it unchanged. Transport failures remain target.unavailable.

The Python SDK maps request.invalid to InvalidRequestError and transport failures to TargetError; relayed invalid_request refusals also arrive as TargetError, carrying the controller's message. A failed run ends Run.watch() normally; Run.wait() returns its RunResult. Call RunResult.raise_for_failure() to opt into RunFailedError.

Attach to an active execution

Status can expose an active execution reference. Attach to its interactive event stream with:

zeroshot attach RUN_ID EXECUTION_REF

Closing the attachment stream does not cancel the execution.

Download a retained workspace

Download the latest retained files without starting another run:

zeroshot export-workspace RUN_ID --target platform --output workspace.tar
zeroshot export-workspace RUN_ID --target platform --from-checkpoint CHECKPOINT_ID --output checkpoint.tar

Use zeroshot checkpoints RUN_ID --target platform to find an entry checkpoint ID. Omit --from-checkpoint to export the latest retained workspace. The target authorizes access and checks whether the retained state is still available. Other named hosted targets work the same way, using their existing device login or ZEROSHOT_API_KEY and organization selection.

The target must advertise workspace export. An older target without this capability returns a clear unsupported error. Local and direct-target exports are unavailable.

--output is required and must name a UTF-8 path that does not exist, including directories and dangling symlinks. The CLI streams the raw tar into a private temporary file beside that path and publishes it atomically without overwriting a competing file. It waits for the entire HTTP transfer, including its framing, even after a complete tar has arrived. Transfer failures and Ctrl-C remove temporary output and fail the command. The archive is never extracted and its contents are never executed.

Restart or resume a failed run

Failed local runs and direct targets retain their workspace. Restart a failed run where it started with:

zeroshot resume RUN_ID

A run you submitted restarts at the root of its graph. A successor restarts at the point it began at: after the answer for the successor of an answered request, after the held node for the successor of a suspended run, and at the selected node for a checkpoint successor. An answered request therefore stays answered, also when several successors fail in a row. To run the whole graph from its root again, submit a new run.

Confirmed Claude account/session quota exhaustion fails promptly as refusal, with the affected node, available reset information and manual-resume guidance in status and terminal failure context. Resume after provider usage becomes available; Zeroshot does not wait or resume automatically. A clock hint without a date or timezone stays as supplied. Codex's message-only usage-limit errors retain their existing crash retries; see the native failure signals.

A local restart uses your checkout as it is now. You can fix files after a failure and resume: commits, staged and unstaged edits, untracked and ignored files, the current branch, and the index are kept. Agents then change the checkout as in any run. Named targets restart on the latest snapshot of their own retained workspace.

To resume at an earlier node, list its available input checkpoints and select one:

zeroshot checkpoints RUN_ID
zeroshot resume RUN_ID --from-checkpoint CHECKPOINT_ID --discard-workspace-changes

Restoring a local checkpoint rolls back your checkout, so it requires --discard-workspace-changes. The checkpoint replaces tracked, staged, untracked, and ignored files and the index. Without the flag the resume fails before it changes anything and shows the command to opt in. HEAD stays attached to its branch at the current tip while files and the index return to the checkpoint. Branches, tags, commits, Git configuration, and other worktrees are kept.

For a named target, add --target NAME to both commands and omit --discard-workspace-changes: the target restores its own retained copy, never your checkout. Checkpoint listings are paginated; use --after CHECKPOINT_ID with the returned nextAfter value to fetch the next page.

A selected checkpoint restores the files from immediately before its node and the completed prerequisite outputs. The selected node runs again. Parallel and mapped groups have one checkpoint for the whole outer group, including nested groups and all mapped batches; individual concurrent writers do not have separate snapshots. Read-only boundaries can share the same workspace bytes.

Both modes create a new run ID, keep the original graph, input, source, and delivery lineage, and resolve credentials again. A selected checkpoint or the continuation of a suspended run restores each supported node_instance agent's provider conversation as it was at the checkpoint, even on another machine or without restoring workspace files. Restarting such a successor restores the same conversations again, without the turns it added before it failed. A restarted run you submitted and every execution-scoped agent start new conversations. Successor token usage starts at zero.

Checkpoints hold raw provider transcripts and conversation files, up to 1 GiB per conversation. They are kept and deleted with their checkpoint, deduplicated across its lineage, and never included in workspace exports. Restoring replaces only that conversation's provider files, including later turns still present locally. If files are missing, oversized, unsupported or cannot be restored, the node starts a new conversation and the run log explains why. A failed continued turn follows the node's attempt policy; its retry starts a new conversation. A sudden target loss can precede the next snapshot boundary, so the last writer may run again. Local workspaces must be idle while they are restored.

A local checkpoint resume restores the snapshot before any agent starts. That preparation may take up to 10 minutes. For larger workspaces or slower disks, set ZEROSHOT_PREPARATION_TIMEOUT to a whole number of seconds from 1 to 86400 before zeroshot resume. When a restore fails, or its extraction reaches that deadline, the error names the step that stopped and its elapsed time. Replacing the workspace files, once started, is allowed to finish. If it is still running 30 seconds after the deadline, the resume is stopped with an error that names the deadline. Either way the run stays recoverable.

A restore never changes the checkout before the complete snapshot is extracted. If a crash interrupts it while it replaces files, the checkout can stay partially restored. A plain zeroshot resume is then refused until a restore completes, so no agent runs on a partial checkout. Repeat the interrupted command, or select another checkpoint with --discard-workspace-changes. That resume removes the interrupted staging copy beside the workspace and finishes the restore before any agent runs. Earlier releases also restored the latest snapshot on a plain resume; complete such an interrupted restore with zeroshot resume RUN_ID --discard-workspace-changes, which installs the same snapshot again.

Local and Docker targets deduplicate checkpoints in a private Restic repository beside their run storage and advertise openengine.workspace-checkpoints/v1. They retain that repository only for a failed recovery lineage and delete it after a successful run. Force-stopping a local run also deletes the repository, together with the checkpoints listed by every run in its recovery lineage, so neither the original run nor its successors list checkpoints that can no longer be restored. Local checkpoints never expire automatically. To free the space kept by a finished failed local run, use zeroshot discard-workspace RUN_ID. It deletes checkpoint data and lists for the whole recovery lineage while keeping the checkout, run directory, ledger, logs and delivery recovery state. A second call reports discarded: false. Plain resume still uses the current checkout, with workspace checkpoints disabled; checkpoint selection and --discard-workspace-changes are refused. A running or resuming lineage must be force-stopped first. An uncontinued suspension must also be force-stopped, and an interrupted restore must be completed by repeating its resume. Other lineages keep theirs. Named-target behavior is unchanged. Platform exposes recovery through authenticated hosted run routes, so failed-run checkpoints remain available for 30 days after the original capsule is removed. Platform resolves declared connections from their original references. Direct-target resume reuses the original source connection authorization and resolves fresh values; export GH_TOKEN again and pass --forward-gh-token when the run relied on the shell's token instead of the target's stored github connection.

Stop only with explicit intent

force-stop is the destructive run-lifetime command:

zeroshot force-stop RUN_ID

The command asks the controller to stop active work and writes the resulting status as JSON. Read status first if another operator may already have stopped the run, or if it may have completed.

If a local controller does not answer within 10 seconds, force-stop exits non-zero and names the run and the controller PID recorded at readiness. The request stays queued and takes effect if the controller answers again. The command does not kill the controller process.

Python follows the same rule. A timeout or task cancellation detaches observation; await run.force_stop() changes the run lifetime.

Suspend and continue

zeroshot suspend RUN_ID records a soft stop and returns the current status immediately. Running nodes finish, including the whole outer parallel or mapped group. At the next checkpoint, status becomes suspended, with reason, node and createdAt. The controller exits and releases the workspace lock. A graph that finishes first keeps its ordinary result. Running status includes suspendRequested after the request. Suspension requires workspace checkpoints.

zeroshot resume RUN_ID creates a successor (runId, resumedFrom) beginning at the held-back node. It preserves workspace changes made after suspension and restores captured node-instance provider conversations from the checkpoint. Unsupported or missing conversations start new with a system log. A suspended wait can continue only from its reported wait.dueAt, and continues after the wait. No timer starts successors locally or on a direct target. Selected checkpoint resume and --discard-workspace-changes are refused for suspended runs. The predecessor stays suspended and workspaceRecovery.successorRunId links to its sole successor. force-stop can finish an unclaimed suspended attempt as force_stopped and removes its retained checkpoints. A direct target refuses discard-workspace for a suspended run that has no successor; force-stop it first.

Foreground run and watch end with exit 0 at suspension. logs and read-only status commands continue to work after the controller exits. Add --target NAME for the same controls on a direct target. Hosts advertise suspension with openengine.run-suspension/v1.

Vector record

Every run has an ordered, durable vector record. All resume successors inherit their predecessor's entries unchanged. Suspension writes suspend_request, suspension, then the successor's continuation; wait nodes also write wait with their due time.

zeroshot vectors RUN_ID --limit 50
zeroshot vectors RUN_ID --after ENTRY_ID --limit 100
zeroshot vectors show RUN_ID ENTRY_ID

A page contains runId, vectors and an optional nextAfter cursor. Limits are 1 through 100. show reads the complete entry; unknown IDs fail as not found. Both commands are read-only and accept --target NAME where openengine.run-vectors/v1 is advertised.

Human requests

A request suspends with reason request and carries its text, answer shape and expiresAt in status. Read zeroshot inbox list (paged with --after and --limit) or zeroshot inbox show RUN_ID, then use zeroshot answer RUN_ID --input answer.json. The file contains signals and output. The answer starts one successor on the retained workspace. Ordinary resume refuses requests. Deadlines default to 7 days and accept at most 30 days. A late answer starts the author's timeout route and exits 1 naming its successor. Reads never apply expiry. Add --target NAME for a target that advertises openengine.request-ask/v1. See Ask a person.

Steer a run

zeroshot steer RUN_ID "Prioritize the parser regression."
zeroshot steer RUN_ID --text-file direction.txt --target NAME
printf '%s' 'Check the Windows path too.' | zeroshot steer RUN_ID --text-stdin

Supply exactly one text source. An admitted, running or suspended run accepts direction. Steering keeps a suspended run suspended. A run with a successor refuses and names that successor; steer the successor instead. Stopping runs and runs past the end of their graph refuse steering.

With the default standing mode, every agent conversation, including reviewers, receives each steer at its next tool call or in its next prompt, whichever comes first. It cannot revise a tool call already issued. At a tool call the steer arrives as a user message of its own on Claude Code, Copilot, Pi and OpenCode: the harness hands it to the model after the tool call in progress and keeps it in the conversation. It is never placed inside a tool result, which a model may refuse to follow. Codex has no such channel and receives the steer with the tool call's hook context, which it keeps. A steer of any permitted size arrives whole either way. Subagents do not receive steers: on Claude Code a message sent while a subagent works reaches the node's own conversation when the subagent returns. A message the harness never shows in the conversation is not counted as received, and the steer is delivered with that conversation's next prompt instead. A steer that reached a conversation is not delivered to it a second time, in either mode. A small model can take a large message that arrives while it works for tool output and ignore it; steer again if a direction was not followed. Once messages are consumed per node instance. Direction takes precedence over authored instructions as guidance; permissions, output contracts, routing, runtime bindings, graph hashes and limits remain unchanged. Delivery works with Active Memory off. Standing steers last for the rest of the run and its successors. Continued conversations keep receipts; new conversations receive the standing direction. A lost acknowledgement can cause repeat delivery.

Text must be nonblank UTF-8, at most 262,144 bytes, with no NUL or control characters except tab, newline and carriage return. Text is stored and shown as given: do not put secrets in a steer.

Reading steering status requires a client from this release or newer. The command prints one JSON receipt. zeroshot status RUN_ID lists steers with recording times, byte lengths, character-safe excerpts and receiving execution references. A standing prompt receipt is recorded when the turn carrying the steer starts and stays provisional until the turn succeeds; a failed turn offers the steer in the next prompt, including in a successor. A once prompt receipt is recorded after successful provider completion. A tool-call receipt appears when the harness shows the message in the conversation, or on Codex when its hook has returned. A receipt records that the conversation holds the steer, not that the model followed it. The bounded listing reports omitted directions or older execution references. Read the complete record and full text with:

zeroshot vectors RUN_ID
zeroshot vectors show RUN_ID STEER_ID

The record includes steer and steer_receipt entries in event order. Add --target NAME to these commands for a direct target advertising steering and run vectors. Tool-call receipts record a message the harness showed (on Codex, acknowledged hook context), while once prompt receipts record successful provider completion. Standing prompt receipts may be provisional, as described above.

Send a peer message

A graph can declare networks for selected agents. An authorized publisher sends a message from its own running local execution:

zeroshot steer "$ZEROSHOT_RUN_ID" --network review "The parser fix is ready for review."
zeroshot steer "$ZEROSHOT_RUN_ID" --network review --network implementers --mode standing \
  "Keep the Windows path in scope."

Repeat --network to address several networks. The command accepts the same text, file and stdin sources as human steering. Each recipient receives the message once even when it belongs to several selected networks. All mapped instances of a member node participate, including other instances of the publishing node. The originating instance never receives its own message, even after a retry, a fresh provider conversation or a resume.

Network messages default to --mode once: each recipient node instance receives one confirmed delivery, and that receipt survives fresh provider conversations and successors. Use --mode standing to keep guidance for the rest of the run and its successors. Fresh conversations receive it again. Human steering without --network keeps its standing default; --mode once selects a one-off human direction.

Publishers receive their authorized network names and publication instructions in private context. The host supplies an execution-scoped capability and records the origin itself; environment labels do not grant publication access. Members outside the publisher set can receive messages but cannot send them. The CLI and local RPC reject agent calls to the human broadcast form, including calls with cleared execution labels or another local run as their destination. A caller outside the authorized execution cannot publish a network message. Native execution permissions, including OS account access, remain in effect.

Pending messages enter a recipient's next prompt or arrive at its next tool call, on the same carriers as human steering and never inside a tool result. Sequential recipients receive them once they start. A failed record read leaves the message pending for the next delivery point. Publication never starts an agent, restarts a recipient or wakes a suspended run. Peer messages carry their source as context; they do not override authored instructions or grant permissions. Network destinations, mode and origin remain in the vector record alongside the text and delivery receipts. Lost acknowledgements can cause repeat delivery before the host confirms receipt. If publication reports an unknown outcome, inspect the run vectors before sending the message again.