· 5 min read · Mert Ozgun
A coding agent can work in a sandbox for hours: editing files, installing packages, running tests, committing. Hardware fails during that time, and whoever runs the agents has to know what is left afterwards and what to do next.
This post explains how Ortam handles a host failure, as its docs state it, and why each part works the way it does. Ortam is in private beta, and some of its numbers aren’t published yet: how often it saves, and how independent its storage is from a host. The post says where those matter. The last part, what your own system should do, applies on any provider.
A sandbox holds three kinds of state, and a host failure treats each differently:
/tmp and /run, and the credentials Ortam delivers for Connections, are never saved. Changes after it can be lost, “even writes Linux reported as finished”. Linux counts a write as finished before any save has copied it off the host.Ortam saves a running Sandbox on a schedule, and a save counts only once it is complete and stored independently of the host. A save that was under way when the host failed never replaces the last completed one, so recovery never starts from a half-written disk. Until a Sandbox’s first save completes, the only thing that can be recovered is the base image or Snapshot it was created from.
Saves are crash-consistent: what a save holds is what the disk would hold if the power had been pulled at that moment. Postgres and most other databases recover from that state when they start, as they would after a power cut. If an application needs a clean state, such as a database you take a milestone Snapshot of, pause its writes first.
The schedule isn’t a promise about how much a failure can lose. Ortam’s docs say a save can lag, and the Sandbox’s status says when saving is delayed or failing. The interval itself isn’t published yet, so today Ortam makes no promise about the most recent work a failure can lose.
lost. Commands that were running end infrastructure_lost, or unknown if Ortam can’t establish what happened.Recovery restores files only. Command records and output, settings, Connections and grants stay as they were.
The wait in step 3 is there because a host that looks failed may only be cut off from the network. If it could still write the Sandbox’s data while a new VM booted from it, two machines would be writing the same disk, and neither copy could be trusted. Ortam waits until the old host can’t write before the new one starts. A slower recovery is the price of never having two writers.
An agent’s commands act outside the sandbox. They push branches, open pull requests, run migrations against shared databases, call APIs and spend model tokens. When a host fails partway through a command, Ortam can’t tell whether the command got as far as any of those. Running it again could open a second pull request or apply a migration twice.
Your system knows which of its commands are safe to repeat and Ortam doesn’t, so the decision is left to your system. The same rule applies to everything else that reports unknown: Ortam reports what it couldn’t establish instead of guessing.
Read each command’s state when it ends, and act on it:
| State | What happened | What to do |
|---|---|---|
exited | The program ended, with its exit_code | Use the result. A nonzero code is the program’s own |
infrastructure_lost | The host failed while it ran | Check what it already did, then decide whether to run it again |
unknown | Ortam can’t establish what happened | The same: treat it as possibly run |
not_launched | It never started, for example because the Sandbox stopped first | Send it again, with a new idempotency key |
launch_failed | The program couldn’t start, such as a missing executable | Fix the command |
timed_out | Your timeout passed, and Ortam asked its processes to end | Check what it did before you send revised work |
resource_exhausted | It ran out of a resource it needed | Use a larger size, or do less at once |
interrupted | You or a Stop asked it to end | Expected after a Stop; otherwise check who stopped it |
A command’s result doesn’t prove that every process it started has ended: services it started can keep running until the Sandbox stops.
Beyond that, a few habits limit what a failure can cost, whichever provider you use:
AGENTS.md. A host failure then costs at most the work since the last push, a limit you set, whatever the save interval is.Three numbers in this design are still to be published: how often Ortam saves, how far apart its storage is from a host, and how long a Stop has to finish. Until they are, the habits above are what limit a failure’s cost. They are worth having anyway, because no save interval brings the loss to zero.
If you are comparing providers, ask each one the same questions: where a running sandbox’s disk lives, what the most recent work a host failure can lose is, and whether anything reruns afterwards. The comparison of what ten providers keep covers what their docs state.
· 6 min read
A coding agent’s sandbox can start fresh, from a prebuilt image, or on the disk the last job left. What each one costs in waiting, failures, drift, parallel work and money, and how to measure it on your own jobs.
· 10 min read
Set an agent’s environment up once, keep it in a Snapshot, give each task its own copy, and rebuild it only when the lockfile or images change. With a script that waits for every step.
· 9 min read
Persistent can mean the disk, the memory, or a snapshot you take yourself. Six questions that tell them apart, answered from ten providers’ own docs.