Host failures

A coding agent can work in a sandbox for hours, editing files, installing packages, running tests and committing. Hardware fails during that time. Whoever runs the agents then has to know what is left and what to do next.

This guide explains how Ortam handles a host failure, as Saves and recovery states it, and why each part works that way. Ortam is in private beta, and some of its numbers aren’t published yet: how often it saves, and how independent its storage is from a host. The guide says where those matter. The last part, what your own system should do, applies on any provider.

What a failure can lose

A sandbox holds three kinds of state, and a host failure treats each differently:

Files on the persistent filesystem are kept up to the last completed save. Changes after that save can be lost, “even writes Linux reported as finished”. Linux counts a write as finished before any save has copied it off the host. Temporary mounts such as /tmp and /run, and the credentials Ortam delivers for Connections, are never saved.
Memory, running processes and open sessions are lost. Agents that write their session to files, as Claude Code and Codex do, keep it as of the last save and can resume from there.
Command records survive. You can still read which commands were running and how they ended. Output Ortam had already kept survives too, marked as kept. The newest lines can be lost.

Which save Ortam recovers from

Ortam saves a running Sandbox on a schedule. A save counts only once it is complete and stored independently of the host. A save that was under way when the host failed never replaces the last completed one, so recovery never starts from a half-written disk. Until a Sandbox’s first save completes, only the base image or Snapshot it was created from can be recovered.

A running SandboxLost: since the last completed save

Host failsThe Sandbox is lost; running commands end infrastructure_lost

Next start waitsUntil the failed host can no longer write

Boots from the last completed saveNothing reruns; your system decides

Completed saveSave under way, doesn’t countHost failsWork lost

A save counts once it’s complete and stored away from the host. Ortam never recovers from a save that was under way when the host failed.

Saves are crash-consistent: a save holds what the disk would hold if the power had been cut at that moment. Postgres and most other databases recover from that state when they start, as they would after a power cut. If an application needs a clean state, such as a database you take a milestone Snapshot of, pause its writes first.

The schedule isn’t a promise about how much a failure can lose. A save can lag. The Sandbox’s status says when saving is delayed or failing. The interval itself isn’t published yet, so today Ortam makes no promise about the most recent work a failure can lose.

What happens after the failure

The Sandbox’s phase becomes lost. Commands that were running end infrastructure_lost, or unknown if Ortam can’t establish what happened.
Nothing restarts or reruns on its own.
The next start, or the next command sent to the Sandbox, brings it back from the last completed save. That start can wait while Ortam makes sure the failed host can no longer write.
Until then, Snapshots of the Sandbox are refused.
Ortam reports which save it recovered from and which commands were running, and never presents that save as the latest live state.

Recovery restores files only. Command records and output, settings, Connections and grants stay as they were.

Ortam waits before recovering because a host that looks failed may only be cut off from the network. If it could still write the Sandbox’s data while a new VM booted from it, two machines would be writing the same disk. Neither copy could then be trusted. So Ortam waits until the old host can’t write before the new VM starts. Recovery is slower as a result, and two hosts never write the same disk.

Why nothing reruns

An agent’s commands act outside the sandbox. They push branches, open pull requests, run migrations against shared databases, call APIs and spend model tokens. When a host fails partway through a command, Ortam can’t tell whether the command got as far as any of those. Running it again could open a second pull request or apply a migration twice.

Your system knows which of its commands are safe to repeat. Ortam doesn’t, so it leaves the decision to your system. The same rule applies to everything else that reports unknown: Ortam reports what it couldn’t establish instead of guessing.

What your system should do

Read each command’s state when it ends, and act on it:

State	What happened	What to do
`exited`	The program ended, with its `exit_code`	Use the result. A nonzero code is the program’s own
`infrastructure_lost`	The host failed while it ran	Check what it already did, then decide whether to run it again
`unknown`	Ortam can’t establish what happened	The same: treat it as possibly run
`not_launched`	It never started, for example because the Sandbox stopped first	Send it again, with a new idempotency key
`launch_failed`	The program couldn’t start, such as a missing executable	Fix the command
`timed_out`	Your timeout passed, and Ortam asked its processes to end	Check what it did before you send revised work
`resource_exhausted`	It ran out of a resource it needed	Use a larger size, or do less at once
`interrupted`	You or a Stop asked it to end	Expected after a Stop; otherwise check who stopped it

A command’s result doesn’t prove that every process it started has ended: services it started can keep running until the Sandbox stops.

These habits limit what a failure can cost, whichever provider you use:

Push the agent’s work to a branch. Have the agent commit and push to a work branch whenever its tests pass, for example with a line in its AGENTS.md. A host failure then costs at most the work since the last push, a limit you set, whatever the save interval is.
Keep the plan in a file. An agent that writes its plan and progress to a file in the repository can read it after a restart and continue from there, even if its session is lost.
Make your own steps safe to repeat. Send every request that changes something with an idempotency key, and before you rerun a command, check for its effects: whether the branch already has the commit, whether the pull request exists.
Check the outcome of every Stop. A Stop reports its final save separately from the end of the work. If that save fails, the next start uses the save before it, and the last changes aren’t there.

What isn't published yet

Three numbers in this design are still to be published: how often Ortam saves, how far apart its storage is from a host, and how long a Stop has to finish. Until they are, the habits above limit a failure’s cost. Keep them afterwards too, because no save interval brings the loss to zero.

If you are comparing providers, ask each one the same questions: where a running sandbox’s disk lives, what the most recent work a host failure can lose is, and whether anything reruns afterwards. What ten providers keep, compared covers what their docs state.