Handle failures and recovery

Three kinds of recovery, not interchangeable

The platform separates three responses because they address problems of different natures.

Retry handles execution faults: a model gateway timeout, a runtime crash, a command timeout, a missing node protocol. Retry prefers to recover the original session, and creates a new session when the original cannot be recovered reliably. Retry does not create a new attempt.

Rework handles semantic defects: a downstream node has established that an upstream artefact is wrong in substance. Rework always creates a new session, a new attempt and a new artefact version.

Manual rerun handles a single node inside a task that has already failed terminally. Rerun rewrites the failure record by appending the current attempt to history, not by erasing it.

The reason for keeping them apart is that they are counted and bounded differently. Only a downstream node confirming a semantic defect in an upstream artefact increments the rework count; a gateway hiccup never consumes rework budget.

How failures are classified

The coding driver normalises raw errors into platform types: transient fault, driver crash, timeout, contract violation, authentication failure, permission denied, budget exceeded, cancelled and semantic failure.

The supervisor does not parse the runtime's natural-language error text to decide a state transition. That restriction matters: if state transition depended on a model's reading of an error string, the same error worded two ways would produce two different behaviours.

Choosing the rework target

A downstream node that cannot repair a semantic problem itself returns a structured failure. The supervisor validates the target and the quota, then produces a normalised failure report whose metadata carries at least the failing node, the rework target, the artefact identity and version, the failure code and whether it is retryable.

The rework target must be a legitimate producing node of the relevant artefact, not mechanically the previous node in execution order. That constraint is what stops rework being pointed at a node with nothing to do with the artefact in question.

Once upstream has produced a new artefact version, the supervisor invalidates every derived node that consumed the old version along the dependency relation and reruns only the affected subgraph. Unrelated branches are left alone.

Limits

Rework is bounded by three limits at once: the maximum attempts per node, the maximum reworks per target, and the total rework count. Along with the gate-repair limit per attempt, all of them are frozen explicitly by the flow and cannot be overridden by a node.

When any limit trips, the task fails. That is intentional: a rework loop that has slipped out of control costs far more than one failure.

The nodes downstream of a failed node do not run; independent branches with no dependency on it keep executing.

Rerunning a single node manually

A terminally failed task lets the user re-execute one failed node on its own.

Rerun does not rewrite the failure record. It appends the current attempt to history and then creates a new attempt. Artefacts and code revisions from completed upstream nodes are reused as frozen evidence rather than recomputed.

Once the target node succeeds, the runner goes on to execute the downstream nodes that have not yet run and whose dependencies are now satisfied.

A rerun re-resolves the latest revisions of the current global Git connector and of personal credentials, against the task creator and repository permissions, and then refreshes the published configuration held in the sandbox supervisor's memory through an authenticated internal interface. Tokens do not enter the agent, the prompt, the execution specification, a snapshot or routing data. The task top level keeps the snapshot taken at creation, and each attempt separately records the non-secret revisions resolved for it.

Empty responses

When a model repeatedly returns no text, no tool activity and zero input and output tokens, the runner classifies it as an empty model response and starts a bounded number of recovery sessions under a backoff policy.

Once those are exhausted, an explicit node failure is retained, so the user can rerun it individually after updating the model or the connector. An empty response is never disguised as completion. The reason is that a silent empty response would otherwise yield a node that looks successful and contains nothing.

Replanning mid-run

When the plan turns out not to hold while a task is running, the agent may propose a new flow revision, which goes back through compilation, permission and risk assessment.

The initial specification stays as it was. Later revisions are written to an append-only plan log, which retains the old topology, the reason for the change, its initiator and the approval record in full.

A completed node may be reused only when its invocation contract, inputs, upstream artefacts, node template and capability summary are all still compatible. Otherwise it is explicitly marked invalid and rerun.

Replanning must not delete historical evidence, and must not present not-yet-executed nodes as though they had been part of the initial plan.

Waiting for human input

An agent question marks only the current attempt as waiting for input. Dependent downstream waits, while independent parallel branches keep running.

A reply binds to the node, the attempt and the question identifier. Stale questions belonging to an invalidated attempt are closed automatically — otherwise a question already superseded by a new attempt would go on drawing attention.

A question waits for a configured period by default. On timeout the default is to fail the task, though it can be configured to continue on an assumption or to fail that node alone.

When a user sends a message unprompted and the agent is busy, the message enters a pending queue. An explicit forced insertion prefers the runtime's native insert capability, and otherwise interrupts the current turn and resumes. A runtime that cannot reliably resume must refuse forced insertion outright rather than silently queueing.

A pending message can be edited or withdrawn before it is delivered, and the input area always shows the target node, attempt and session.

Cancellation

A user can cancel a single attempt and rerun it by hand, or cancel the whole task.

Permanently skipping a required node and letting the dependency graph carry on automatically is not supported. The limitation is deliberate: skipping a required node breaks the integrity of the artefact contract, leaving downstream to consume something that was never produced.

Cancellation acceptance, process exit, durable terminal state and business success are distinct state facts. A side effect with no exit evidence stays unknown and is not replayed automatically.

When to rebuild rather than recover

A rebuild reads the original task's business inputs and creates a new task under the current operator's identity, re-freezing models and credentials, capability locks and runtime, and re-issuing the knowledge access policy. The old task is kept in full.

Rebuilding is the better choice when:

• the requirement itself has changed;
• you need to change the model or the gateway;
• the flow version the original task used has been deprecated and you want the newer one;
• you want the work carried out under another person's identity and permissions.

Recovery suits a plan that still holds and an execution that was merely interrupted. Rebuilding suits inputs or an environment that have already changed.