Observe and control runs¶
Run state persists when an observation client leaves, and reconnecting does not change the run lifetime.
Browser¶
Run zeroshot ui, then select a run in Runs.
The read-only graph uses that run's admitted definition. Select a node for its inputs, outputs,
decisions, and transcript; with the details panel open, select the empty canvas or the first link
above a node's name for the run's own details. Use the timeline to replay its history. Active runs
update live.
Seeking pauses following; Go live returns to the latest event. Scrolling a transcript keeps
the graph live and follows new output only when you reach the bottom.
For a direct target, open /ui/ on its configured public origin. Closing either browser view
leaves the run active.
Foreground and detached submission¶
zeroshot run follows the durable event stream by default and writes NDJSON. Add --detach to
return after submission:
zeroshot run \
--title "Update parser" \
--template software-change --delivery none \
--input input.json \
--uniform-runtime-config runtime.json \
--detach
Ctrl-C closes the terminal stream and leaves the controller running.
Inventory and current status¶
These commands are read-only and return JSON:
A run ID belongs to the controller or target that created it. Pass --target NAME when reading one
from a named target. Finished status separates graph-authored terminalResult.output from optional
runtime-confirmed metadata.deliveryReceipt; graphs without delivery omit the receipt.
When the graph ends, status and list expose its recorded outcome as status.graphResult,
using the same shape as terminalResult, while phase remains running during the Active Memory
postmortem and finalization. For example:
{
"phase": "running",
"activeExecutions": [],
"graphResult": { "status": "failed", "reason": "worker_failed" }
}
graphResult is absent before the graph ends and after the run finishes, when terminalResult
is the only result. It remains visible if finalization enters stopping. watch emits the result
as soon as it is recorded, and foreground run prints that event and keeps waiting for finalization.
The log records the graph result once and says when the postmortem is running. Exit codes still
follow the final run result.
The recorded graph result does not guarantee the final run result. A force stop during the
postmortem cancels and drains that work and preserves the recorded graph result; ordinary
postmortem failures and controller restart after confirmed cleanup preserve it too. A runtime or
cleanup failure during finalization can instead prevent completion or end the run with
runtime_failed or runtime_lost. Read terminalResult once the run is finished.
Local inspection (status, list, checkpoints, vectors, watch, logs and attach) reads
retained history without taking recovery ownership or writing to the run directory. Finished and
suspended runs need no controller and retain their usual exit codes.
Each status.activeExecutions entry can include startedAt, the ledger recording time of that
execution's NodeStarted. Watch notifications also include optional recordedAt for the event
that produced the notification. Match the opaque execution selector across active sets to identify
starts and settlements, including retries and simultaneous map items. Status remains a summary;
completed execution timing is available through watch replay and the history API.
These times are Unix milliseconds recorded by the ledger, not precise provider start/stop
measurements. Events in one append transaction share a time, and cursor alone defines ordering
and reconnection boundaries. Missing times in older ledgers stay omitted. Imported history carries
its import time, not the time the original work ran. Log timestamp retains its producer meaning.
For an unfinished local run whose controller does not answer, status and each list entry keep
the recorded status and add an optional entry beside it:
lost means the original controller process is confirmed absent and its original lease is released.
unknown means the evidence is insufficient, including an unreachable endpoint, a stalled process,
a reused PID, a replaced lease or a permission failure. observedAt is the time of this check in
Unix epoch milliseconds, not the time the controller died. The entry is omitted when the controller
answers and for finished or suspended runs. checkpoints also includes it when applicable. These three commands
exit 0 after a successful read, including when controller availability is lost or unknown.
Local controller requests wait at most 10 seconds for an answer. This includes the status checks
before resume and checkpoints, and the initial exchanges for watch, logs and attach.
list checks controllers with bounded concurrency under one shared deadline. Runs that could not
be checked within that budget retain their recorded status with an unknown controller. An established
stream may stay quiet indefinitely. Controller handoffs keep their separate retry budget.
For an unfinished run, watch, logs and attach print the recorded part, then exit 1 when the
local controller is unavailable. Their error names zeroshot resume RUN_ID and zeroshot force-stop RUN_ID. Inspection
does not record a terminal result: an unclaimed lost run keeps its recorded status on every read.
Only these explicit controls confirm loss under the controller lease before recording runtime_lost;
they refuse to take over a controller that still owns its lease or whose loss cannot be confirmed.
steer and suspend never take over: on a lost run they exit non-zero, name the two commands and
leave the recorded state unchanged.
Token usage¶
metadata.tokenUsage contains run totals summed across agent invocations, including retries,
correction turns and auxiliary agents such as postmortem specialists. inputTokens means uncached
input only for every harness. Cache reads and cache writes appear only in their respective cache
counters, so the four token counters do not overlap and can be added.
| Field | Meaning |
|---|---|
inputTokens |
Uncached input tokens, excluding cache reads/writes. |
outputTokens |
Output tokens reported by the harness. |
cacheReadInputTokens |
Input tokens reported as read from a prompt cache. |
cacheCreationInputTokens |
Input tokens reported as written to a prompt cache. |
complete |
Whether every invocation reported usable token usage. |
When complete is false, the totals retain known usage but at least one invocation's usage is
missing or unusable. An omitted cache counter means at least one invocation did not report that
counter; its run total is unknown, not zero. Cache counters can be omitted even when complete is
true. Script steps record zero for all four counters and preserve known cache totals.
Previously recorded runs retain their original numbers.
Failure evidence and mapped executions¶
Failed terminalResult.context separates the known runtime cause (primary) from recent recorded
execution errors (recorded) and verifier results (verifiers). Each verifier result is the latest
recorded result for that graph node and map path. Its bounded, credential-redacted message contains
the graph's actual signals and diagnostic; the runtime assigns no general meaning to their names
or labels. A later result replaces an earlier result in this summary, while logs retain both.
Recorded evidence can include recovered attempts or parallel work and does not prove that a result
caused the terminal failure. An independently known runtime cause remains in primary.
exhaustedLoops identifies completed exhausted loop visits with node, completedIterations, and
an optional mapIndices path. The count is the actual number of completed loop rounds, including
all work within each round; execution retries and verifier completion counts do not add rounds.
Completed exhausted loops in settled branches of an unreachable parallel join are retained.
Discarded race and map-item paths do not contribute loop entries or consume the summary bound.
A converged loop does not appear as exhausted. The summary keeps at most five entries in each
collection. incomplete: true means evidence was omitted or could not be retained. This warning
persists even if later loop visits converge and remove all retained loop entries. Full details may
still be available in logs. Failure context freezes with the main outcome, before auxiliary work.
Each diagnostic can carry an execution object with the node name and stable opaque execution
selector, and a cursor pointing to its retained log. Missing cursors cannot be used for log
navigation. Status/watch activeExecutions and execution-associated log notifications also expose
mapIndices: zero-based indices, outermost map first. For example, [1] is the second item and
[1, 2] is its third nested item. Outside maps the field is omitted (older data also reads as an
empty path). Indices supplement execution identity without changing its opaque selector.
Successful terminal results keep their existing representation.
Resume event and log streams¶
Codex, Claude, Copilot and Pi record native tool calls and completions in both logs and node transcripts by default, including tool identity, available call IDs, arguments, reported status and result excerpts. Missing status remains unknown. Known credentials are redacted before each entry is shortened. Each complete call entry and each completion entry is limited to 8 KiB, including its header and any visible truncation marker; large payloads retain their head and tail. Intermediate cumulative updates are omitted. These limits apply only to logs, so storage grows with tool count; they do not limit provider inputs, agent-visible results or total run storage.
watch follows status events; logs follows safe provider and controller log records. Both accept
an opaque cursor and resume strictly after it:
Cursors are opaque, so store the last fully consumed value and pass it back unchanged.
logs --execution EXECUTION_REF restricts output to one execution selector from status.
watch exits 0 when the run succeeded or suspended. When the run finished failed, it prints the same failure
summary on stderr as a failed foreground run and exits 1, so a script can start a run with
--detach and wait for its result with zeroshot watch RUN_ID. logs exits 0 when its stream
ends, whatever the run's result. The CLI reference lists the exit status of
run, watch and logs.
The producer records each log timestamp as a positive JavaScript-safe Unix epoch millisecond, and durable replay preserves it.
JSON error codes¶
Set ZEROSHOT_ERROR_FORMAT=json to receive a zeroshot.error/v1 object on stderr when a CLI
command fails. Successful stdout and NDJSON events keep their existing format. The object contains
kind, code, message, optional path and node context, and an object-valued details.
Use code to distinguish a failed run or a request that needs correction from a transport outage:
| Condition | Kind | Code | Details |
|---|---|---|---|
A followed run failed (run or watch) |
run_failed |
run.failed |
runId |
| Unknown registered target name | invalid_request |
request.invalid |
{} |
| Local registry has no workspace-recovery authorization | invalid_request |
request.invalid |
{} |
| Workspace-recovery authorization does not match the original run | invalid_request |
request.invalid |
{} |
| Local workspace-recovery authorization is malformed | invalid_request |
request.invalid |
{} |
| Refusal relayed from the run's controller or a target | target |
invalid_request |
Target's details, else {} |
| Target transport failed | target |
target.unavailable |
target and origin when known |
run.failed reports a terminal outcome; retrying observation does not restart that run. Inspect its
status and logs before selecting a recovery action. Correct the request or local registry state
for the listed request.invalid refusals. A relayed invalid_request refusal says what to correct;
do not retry it unchanged. Transport failures remain target.unavailable.
The Python SDK maps request.invalid to InvalidRequestError and transport failures to
TargetError; relayed invalid_request refusals also arrive as TargetError, carrying the
controller's message. A failed run ends Run.watch() normally; Run.wait() returns its
RunResult. Call RunResult.raise_for_failure() to opt into RunFailedError.
Attach to an active execution¶
Status can expose an active execution reference. Attach to its interactive event stream with:
Closing the attachment stream does not cancel the execution.
Download a retained workspace¶
Download the latest retained files without starting another run:
zeroshot export-workspace RUN_ID --target platform --output workspace.tar
zeroshot export-workspace RUN_ID --target platform --from-checkpoint CHECKPOINT_ID --output checkpoint.tar
Use zeroshot checkpoints RUN_ID --target platform to find an entry checkpoint ID. Omit
--from-checkpoint to export the latest retained workspace. The target authorizes access and
checks whether the retained state is still available. Other named hosted targets work the same way,
using their existing device login or ZEROSHOT_API_KEY and organization selection.
The target must advertise workspace export. An older target without this capability returns a clear unsupported error. Local and direct-target exports are unavailable.
--output is required and must name a UTF-8 path that does not exist, including directories and dangling
symlinks. The CLI streams the raw tar into a private temporary file beside that path and publishes it
atomically without overwriting a competing file. It waits for the entire HTTP transfer, including
its framing, even after a complete tar has arrived. Transfer failures and Ctrl-C remove temporary
output and fail the command. The archive is never extracted and its contents are never executed.
Restart or resume a failed run¶
Failed local runs and direct targets retain their workspace. Restart a failed run where it started with:
A run you submitted restarts at the root of its graph. A successor restarts at the point it began at: after the answer for the successor of an answered request, after the held node for the successor of a suspended run, and at the selected node for a checkpoint successor. An answered request therefore stays answered, also when several successors fail in a row. To run the whole graph from its root again, submit a new run.
Confirmed Claude account/session quota exhaustion fails promptly as refusal, with the affected
node, available reset information and manual-resume guidance in status and terminal failure
context. Resume after provider usage becomes available; Zeroshot does not wait or resume
automatically. A clock hint without a date or timezone stays as supplied. Codex's message-only
usage-limit errors retain their existing crash retries; see the
native failure signals.
A local restart uses your checkout as it is now. You can fix files after a failure and resume: commits, staged and unstaged edits, untracked and ignored files, the current branch, and the index are kept. Agents then change the checkout as in any run. Named targets restart on the latest snapshot of their own retained workspace.
To resume at an earlier node, list its available input checkpoints and select one:
zeroshot checkpoints RUN_ID
zeroshot resume RUN_ID --from-checkpoint CHECKPOINT_ID --discard-workspace-changes
Restoring a local checkpoint rolls back your checkout, so it requires --discard-workspace-changes.
The checkpoint replaces tracked, staged, untracked, and ignored files and the index. Without the
flag the resume fails before it changes anything and shows the command to opt in. HEAD stays
attached to its branch at the current tip while files and the index return to the checkpoint.
Branches, tags, commits, Git configuration, and other worktrees are kept.
For a named target, add --target NAME to both commands and omit --discard-workspace-changes:
the target restores its own retained copy, never your checkout. Checkpoint listings are paginated;
use --after CHECKPOINT_ID with the returned nextAfter value to fetch the next page.
A selected checkpoint restores the files from immediately before its node and the completed prerequisite outputs. The selected node runs again. Parallel and mapped groups have one checkpoint for the whole outer group, including nested groups and all mapped batches; individual concurrent writers do not have separate snapshots. Read-only boundaries can share the same workspace bytes.
Both modes create a new run ID, keep the original graph, input, source, and delivery lineage, and
resolve credentials again. A selected checkpoint or the continuation of a suspended run restores each supported
node_instance agent's provider conversation as it was at the checkpoint, even on another machine
or without restoring workspace files. Restarting such a successor restores the same conversations
again, without the turns it added before it failed. A restarted run you submitted and every
execution-scoped agent start new conversations. Successor token usage starts at zero.
Checkpoints hold raw provider transcripts and conversation files, up to 1 GiB per conversation. They are kept and deleted with their checkpoint, deduplicated across its lineage, and never included in workspace exports. Restoring replaces only that conversation's provider files, including later turns still present locally. If files are missing, oversized, unsupported or cannot be restored, the node starts a new conversation and the run log explains why. A failed continued turn follows the node's attempt policy; its retry starts a new conversation. A sudden target loss can precede the next snapshot boundary, so the last writer may run again. Local workspaces must be idle while they are restored.
A local checkpoint resume restores the snapshot before any agent starts. That preparation may take
up to 10 minutes. For larger workspaces or slower disks, set ZEROSHOT_PREPARATION_TIMEOUT to a
whole number of seconds from 1 to 86400 before zeroshot resume. When a restore fails, or its
extraction reaches that deadline, the error names the step that stopped and its elapsed time.
Replacing the workspace files, once started, is allowed to finish. If it is still running 30 seconds
after the deadline, the resume is stopped with an error that names the deadline. Either way the run
stays recoverable.
A restore never changes the checkout before the complete snapshot is extracted. If a crash
interrupts it while it replaces files, the checkout can stay partially restored. A plain
zeroshot resume is then refused until a restore completes, so no agent runs on a partial
checkout. Repeat the interrupted command, or select another checkpoint with
--discard-workspace-changes. That resume removes the interrupted staging copy beside the
workspace and finishes the restore before any agent runs. Earlier releases also restored the latest
snapshot on a plain resume; complete such an interrupted restore with
zeroshot resume RUN_ID --discard-workspace-changes, which installs the same snapshot again.
Local and Docker targets deduplicate checkpoints in a private Restic repository beside their run
storage and advertise openengine.workspace-checkpoints/v1. They retain that repository only for a
failed recovery lineage and delete it after a successful run. Force-stopping a local run also
deletes the repository, together with the checkpoints listed by every run in its recovery lineage,
so neither the original run nor its successors list checkpoints that can no longer be restored.
Local checkpoints never expire automatically. To free the space kept by a finished failed local
run, use zeroshot discard-workspace RUN_ID. It deletes checkpoint data and lists for the whole
recovery lineage while keeping the checkout, run directory, ledger, logs and delivery recovery
state. A second call reports discarded: false. Plain resume still uses the current checkout,
with workspace checkpoints disabled; checkpoint selection and --discard-workspace-changes
are refused. A running or resuming lineage must be force-stopped first. An uncontinued suspension
must also be force-stopped, and an interrupted restore must be completed by repeating its resume.
Other lineages keep theirs. Named-target behavior is unchanged. Platform exposes recovery through
authenticated hosted run routes, so failed-run checkpoints remain available for 30 days after the
original capsule is removed. Platform resolves declared connections from their original references.
Direct-target resume reuses the original source connection authorization and resolves fresh values;
export GH_TOKEN again and pass --forward-gh-token when the run relied on the shell's token
instead of the target's stored github connection.
Stop only with explicit intent¶
force-stop is the destructive run-lifetime command:
The command asks the controller to stop active work and writes the resulting status as JSON. Read status first if another operator may already have stopped the run, or if it may have completed.
If a local controller does not answer within 10 seconds, force-stop exits non-zero and names the
run and the controller PID recorded at readiness. The request stays queued and takes effect if the
controller answers again. The command does not kill the controller process.
Python follows the same rule. A timeout or task cancellation detaches observation;
await run.force_stop() changes the run lifetime.
Suspend and continue¶
zeroshot suspend RUN_ID records a soft stop and returns the current status immediately.
Running nodes finish, including the whole outer parallel or mapped group. At the next checkpoint,
status becomes suspended, with reason, node and createdAt. The controller exits and releases
the workspace lock. A graph that finishes first keeps its ordinary result. Running status includes
suspendRequested after the request. Suspension requires workspace checkpoints.
zeroshot resume RUN_ID creates a successor (runId, resumedFrom) beginning at the held-back
node. It preserves workspace changes made after suspension and restores captured node-instance provider
conversations from the checkpoint. Unsupported or missing conversations start new with a system log.
A suspended wait can continue only from its reported wait.dueAt, and continues after the wait.
No timer starts successors locally or on a direct target. Selected checkpoint resume and
--discard-workspace-changes are refused for suspended runs. The predecessor stays suspended and
workspaceRecovery.successorRunId links to its sole successor. force-stop can finish an unclaimed
suspended attempt as force_stopped and removes its retained checkpoints. A direct target refuses
discard-workspace for a suspended run that has no successor; force-stop it first.
Foreground run and watch end with exit 0 at suspension. logs and read-only status commands
continue to work after the controller exits. Add --target NAME for the same controls on a direct
target. Hosts advertise suspension with openengine.run-suspension/v1.
Vector record¶
Every run has an ordered, durable vector record. All resume successors inherit their predecessor's
entries unchanged. Suspension writes suspend_request, suspension, then the successor's
continuation; wait nodes also write wait with their due time.
zeroshot vectors RUN_ID --limit 50
zeroshot vectors RUN_ID --after ENTRY_ID --limit 100
zeroshot vectors show RUN_ID ENTRY_ID
A page contains runId, vectors and an optional nextAfter cursor. Limits are 1 through 100.
show reads the complete entry; unknown IDs fail as not found. Both commands are read-only and
accept --target NAME where openengine.run-vectors/v1 is advertised.
Human requests¶
A request suspends with reason request and carries its text, answer shape and expiresAt in
status. Read zeroshot inbox list (paged with --after and --limit) or zeroshot inbox show RUN_ID,
then use zeroshot answer RUN_ID --input answer.json. The file contains signals and output.
The answer starts one successor on the retained workspace. Ordinary resume refuses requests.
Deadlines default to 7 days and accept at most 30 days. A late answer starts the author's timeout
route and exits 1 naming its successor. Reads never apply expiry. Add --target NAME for a target
that advertises openengine.request-ask/v1. See Ask a person.
Steer a run¶
zeroshot steer RUN_ID "Prioritize the parser regression."
zeroshot steer RUN_ID --text-file direction.txt --target NAME
printf '%s' 'Check the Windows path too.' | zeroshot steer RUN_ID --text-stdin
Supply exactly one text source. An admitted, running or suspended run accepts direction. Steering keeps a suspended run suspended. A run with a successor refuses and names that successor; steer the successor instead. Stopping runs and runs past the end of their graph refuse steering.
With the default standing mode, every agent conversation, including reviewers, receives each steer at its next tool call or in its next prompt, whichever comes first. It cannot revise a tool call already issued. At a tool call the steer arrives as a user message of its own on Claude Code, Copilot, Pi and OpenCode: the harness hands it to the model after the tool call in progress and keeps it in the conversation. It is never placed inside a tool result, which a model may refuse to follow. Codex has no such channel and receives the steer with the tool call's hook context, which it keeps. A steer of any permitted size arrives whole either way. Subagents do not receive steers: on Claude Code a message sent while a subagent works reaches the node's own conversation when the subagent returns. A message the harness never shows in the conversation is not counted as received, and the steer is delivered with that conversation's next prompt instead. A steer that reached a conversation is not delivered to it a second time, in either mode. A small model can take a large message that arrives while it works for tool output and ignore it; steer again if a direction was not followed. Once messages are consumed per node instance. Direction takes precedence over authored instructions as guidance; permissions, output contracts, routing, runtime bindings, graph hashes and limits remain unchanged. Delivery works with Active Memory off. Standing steers last for the rest of the run and its successors. Continued conversations keep receipts; new conversations receive the standing direction. A lost acknowledgement can cause repeat delivery.
Text must be nonblank UTF-8, at most 262,144 bytes, with no NUL or control characters except tab, newline and carriage return. Text is stored and shown as given: do not put secrets in a steer.
Reading steering status requires a client from this release or newer.
The command prints one JSON receipt. zeroshot status RUN_ID lists steers with recording times,
byte lengths, character-safe excerpts and receiving execution references. A standing prompt receipt
is recorded when the turn carrying the steer starts and stays provisional until the turn succeeds; a
failed turn offers the steer in the next prompt, including in a successor. A once prompt receipt is
recorded after successful provider completion. A tool-call receipt appears when the harness shows
the message in the conversation, or on Codex when its hook has returned. A receipt records that
the conversation holds the steer, not that the model followed it.
The bounded listing reports omitted directions or older execution references. Read the complete
record and full text with:
The record includes steer and steer_receipt entries in event order. Add --target NAME to these
commands for a direct target advertising steering and run vectors. Tool-call receipts record a
message the harness showed (on Codex, acknowledged hook context), while once prompt receipts record
successful provider completion. Standing prompt receipts may be provisional, as described above.
Send a peer message¶
A graph can declare networks for selected agents. An authorized publisher sends a message from its own running local execution:
zeroshot steer "$ZEROSHOT_RUN_ID" --network review "The parser fix is ready for review."
zeroshot steer "$ZEROSHOT_RUN_ID" --network review --network implementers --mode standing \
"Keep the Windows path in scope."
Repeat --network to address several networks. The command accepts the same text, file and stdin
sources as human steering. Each recipient receives the message once even when it belongs to
several selected networks. All mapped instances of a member node participate, including other
instances of the publishing node. The originating instance never receives its own message,
even after a retry, a fresh provider conversation or a resume.
Network messages default to --mode once: each recipient node instance receives one confirmed
delivery, and that receipt survives fresh provider conversations and successors. Use --mode standing
to keep guidance for the rest of the run and its successors. Fresh conversations receive it again.
Human steering without --network keeps its standing default; --mode once selects a one-off
human direction.
Publishers receive their authorized network names and publication instructions in private context. The host supplies an execution-scoped capability and records the origin itself; environment labels do not grant publication access. Members outside the publisher set can receive messages but cannot send them. The CLI and local RPC reject agent calls to the human broadcast form, including calls with cleared execution labels or another local run as their destination. A caller outside the authorized execution cannot publish a network message. Native execution permissions, including OS account access, remain in effect.
Pending messages enter a recipient's next prompt or arrive at its next tool call, on the same carriers as human steering and never inside a tool result. Sequential recipients receive them once they start. A failed record read leaves the message pending for the next delivery point. Publication never starts an agent, restarts a recipient or wakes a suspended run. Peer messages carry their source as context; they do not override authored instructions or grant permissions. Network destinations, mode and origin remain in the vector record alongside the text and delivery receipts. Lost acknowledgements can cause repeat delivery before the host confirms receipt. If publication reports an unknown outcome, inspect the run vectors before sending the message again.