Keep environments warm

Decide what one environment holds

There are two common choices:

• One Sandbox per repository. Every job for the repository runs in the same Sandbox, one after another, and finds the last job’s dependencies, build outputs and agent session files in place. This is the simplest setup, and the one with the most already in place. Jobs share one working tree and one set of services, though, so running them at the same time needs a separate worktree and service state for each.
• One Snapshot per repository, and a new Sandbox per task. You prepare the environment once and take a Snapshot. Each task gets its own Sandbox created from it. Tasks can then run in parallel without sharing a working tree, and a task that breaks its machine breaks only its own copy.

Most of this guide builds the second, which also covers the first. For one Sandbox per repository, skip the Snapshot and stop the Sandbox between jobs.

A helper that waits

Commands in a Sandbox run in the background. The API accepts a command and returns 202 with its ID. The command keeps running after the request ends. A script that sends git clone and then npm ci without waiting starts both at once, so npm ci runs before the repository exists.

These helpers wait. Save them as ortam.sh and source them from your scripts. They need curl, jq and uuidgen, and two variables: ORTAM_API, the API address, and ORTAM_API_KEY.

set -euo pipefail

# Sends a request and prints the response. A request that changes something gets one idempotency key, reused on
# every retry, so a retry after a lost response never acts twice. Retries network errors, 429 and 5xx.
ortam_api() {
  local method=$1 path=$2 body=${3:-} status attempt out key
  out=$(mktemp) || return 1
  local args=(--silent --show-error --output "$out" --write-out '%{http_code}' -X "$method"
    -H "Authorization: Bearer $ORTAM_API_KEY")
  if [[ $method != GET ]]; then
    key=$(uuidgen) || return 1
    args+=(-H "Idempotency-Key: $key")
  fi
  if [[ -n $body ]]; then args+=(-H "Content-Type: application/json" -d "$body"); fi
  for attempt in 1 2 3 4 5 6; do
    status=$(curl "${args[@]}" "$ORTAM_API$path") || status=000
    case $status in
      2??) cat "$out" || return 1; rm -f "$out"; return 0 ;;
      000 | 429 | 5??) sleep $((attempt * 2)) ;;
      *) echo "$method $path: HTTP $status $(cat "$out")" >&2; rm -f "$out"; return 1 ;;
    esac
  done
  echo "$method $path: no answer after $attempt tries" >&2
  rm -f "$out"
  return 1
}

# Reads a Snapshot or an operation until it's done, and prints it. Succeeds only if it completed.
wait_for() {
  local path=$1 record state
  while :; do
    record=$(ortam_api GET "$path") || return 1
    state=$(jq -r .state <<<"$record")
    if [[ $state != pending && $state != running ]]; then break; fi
    sleep 2
  done
  if [[ $state != completed ]]; then echo "$path ended $state" >&2; return 1; fi
  echo "$record"
}

# Runs a shell command in a Sandbox and waits for it. Succeeds only if it exited with code 0. The optional third
# argument is a JSON object of environment variables for this command.
run() {
  local sandbox=$1 command=$2 env=${3:-"{}"} body id record state
  body=$(jq -n --arg c "$command" --argjson env "$env" '{argv: ["/bin/bash", "-lc", $c], env: $env}') || return 1
  id=$(ortam_api POST "/v1/sandboxes/$sandbox/executions" "$body" | jq -er .resource.id) || return 1
  while :; do
    record=$(ortam_api GET "/v1/executions/$id") || return 1
    state=$(jq -r .state <<<"$record")
    if [[ $state != accepted && $state != running ]]; then break; fi
    sleep 2
  done
  if [[ $state == exited && $(jq -r .exit_code <<<"$record") == 0 ]]; then return 0; fi
  echo "\"$command\" ended $state, exit code $(jq -r '.exit_code // "none"' <<<"$record")" >&2
  return 1
}

# Stops a Sandbox and waits. Succeeds only if its final save completed, so the next start finds this run's files.
stop_sandbox() {
  local op record
  op=$(ortam_api POST "/v1/sandboxes/$1/stop" | jq -er .operation.id) || return 1
  record=$(wait_for "/v1/operations/$op") || return 1
  jq -e '.result.final_save == "completed"' <<<"$record" >/dev/null || {
    echo "final save: $(jq -r .result.final_save <<<"$record"); the next start uses the save before it" >&2
    return 1
  }
}

A command’s output can’t be read through the API yet: that route is to be published. Until it is, these helpers judge each step by its state and exit code. Read output in a terminal on the Sandbox.

Give it access to the repository without saving a token

A private repository needs a credential to clone and fetch. Anything written to the Sandbox’s disk is saved and copied into every Snapshot. So don’t write the token to a file, a remote URL or the git config. Pass it to each git command as an environment variable instead. Git reads configuration from GIT_CONFIG_COUNT and its numbered keys, and sends the header without writing it anywhere.

basic=$(printf 'x-access-token:%s' "$GITHUB_TOKEN" | base64 | tr -d '\n')
git_env=$(jq -n --arg auth "Authorization: Basic $basic" '{
  GIT_CONFIG_COUNT: "1",
  GIT_CONFIG_KEY_0: "http.https://github.com/.extraheader",
  GIT_CONFIG_VALUE_0: $auth
}')

Use a token that expires, such as a GitHub App installation token, which lasts an hour. Ortam doesn’t return a command’s environment when you read the command back. Git doesn’t write the header to any file, so the token isn’t in the Sandbox’s saves or Snapshots unless a program writes it out. Every process that command starts can still see it.

Set it up once

Create a Sandbox and run the setup an agent would otherwise repeat. Docker isn’t preinstalled on the base image, so install it first. Replace the repository and the commands with your project’s own:

source ortam.sh

golden=$(ortam_api POST /v1/sandboxes '{"name": "checkout-golden"}' | jq -er .resource.id)

run "$golden" "sudo DEBIAN_FRONTEND=noninteractive apt-get update"
run "$golden" "sudo DEBIAN_FRONTEND=noninteractive apt-get install -y docker.io docker-compose-v2"
run "$golden" "git clone https://github.com/you/checkout app" "$git_env"
run "$golden" "cd app && npm ci"
run "$golden" "cd app && sudo docker compose build --pull && sudo docker compose up -d --wait"
run "$golden" "cd app && npm test"
run "$golden" 'sudo docker image prune --force --filter "until=168h"'

Each run returns only when its command has ended. The script stops at the first command that fails. Running the tests is part of the setup: it fills the build and test caches, and it proves the environment works before you copy it. The last line removes dangling images created more than a week ago, so they aren’t copied into every task. Dangling images are the untagged leftovers of earlier builds (Keep the disk from growing).

You don’t save anything yourself. Ortam saves a running Sandbox automatically. A save covers the whole disk, including installed packages, Docker images and volumes (Saves and recovery).

Take a Snapshot and give each task a copy

Take a Snapshot of the prepared Sandbox and wait for it to complete. On a running Sandbox, it captures the files at some point after Ortam accepts the request, so take it only once setup has finished. Then stop the Sandbox so it isn’t billed while it waits for the next rebuild:

snapshot=$(ortam_api POST "/v1/sandboxes/$golden/snapshots" | jq -er .resource.id)
wait_for "/v1/snapshots/$snapshot" >/dev/null
stop_sandbox "$golden"

For each task, create a Sandbox from the Snapshot and check out the task’s commit. The checkout in the Snapshot is the one setup cloned, so the task’s first command fetches:

task=$(ortam_api POST /v1/sandboxes "$(jq -n --arg s "$snapshot" --arg n "checkout-task-$TASK_ID" \
  '{name: $n, snapshot_id: $s}')" | jq -er .resource.id)

run "$task" 'cd app && git fetch origin "$TASK_COMMIT" && git checkout --detach "$TASK_COMMIT"' \
  "$(jq --arg c "$TASK_COMMIT" '. + {TASK_COMMIT: $c}' <<<"$git_env")"

Images built from your own source, by a build: entry in the Compose file, are as old as the Snapshot, so each task builds them again from its commit. Docker reuses the cached layers, so only the layers after the first changed one are rebuilt.

Before the agent starts, check that the copy is warm. npm ci --offline fails if a package isn’t in npm’s cache, and --pull never makes Compose fail if an image it pulls isn’t on the disk:

run "$task" "cd app && npm ci --offline"
run "$task" "cd app && sudo docker compose up -d --wait --pull never --build"

If npm ci --offline fails, the cache isn’t where npm looks for it, or the lockfile changed since the Snapshot and the key should have caught it. If Compose fails, look for a service with pull_policy: always or an image that isn’t pinned. These checks cover npm’s cache and the pulled images. The image builds, install scripts and running services can still use the network.

The new Sandbox gets a copy of the files and changes independently from then on. It doesn’t inherit the source’s settings or Connection grants, so set those on each new Sandbox before the agent starts.

When the task is done and you’ve collected what it produced, delete its Sandbox. Deleting it doesn’t affect the Snapshot:

op=$(ortam_api DELETE "/v1/sandboxes/$task" | jq -er .operation.id)
wait_for "/v1/operations/$op" >/dev/null

For one Sandbox per repository, end each job with stop_sandbox instead, and check that it succeeds. If the final save fails, the next start uses the save before it, and the job’s last changes aren’t there.

Rebuild it when its inputs change

A Snapshot goes stale when something it was built from changes: the lockfile, a Dockerfile, the Compose file, the images it pulls, or the base image. Rebuilding on a schedule wastes work when nothing changed. It also keeps a stale copy when something did. Instead, give each Snapshot a key computed from its inputs.

Compute the key in your own system from a clone of the repository, at the task’s commit. Include package.json, whose install scripts can change without the lockfile changing. Pin images by digest in the Compose file (postgres:16@sha256:…), so a new image changes the file. A tag such as postgres:16 can point at a new image without any file changing.

# Prints a key for the environment at a commit. Fails if an input file is missing.
env_key() {
  local commit=$1 file sums="ubuntu-24.04-v1"
  for file in package.json package-lock.json Dockerfile docker-compose.yml; do
    sums+=" $file:$(git -C "$CLONE" show "$commit:$file" | sha256sum | cut -c1-64)" || return 1
  done
  printf '%s' "$sums" | sha256sum | cut -c1-16
}

Add any other file that changes what gets installed, such as .npmrc, and use the base image your Sandboxes are created with.

The key doesn’t include your application’s source, so a new commit doesn’t make a new Snapshot. Each task rebuilds the images built from that source instead, as shown above.

Store the key next to the Snapshot ID. When a task arrives, compute the key at the task’s commit. If a Snapshot has that key, create the task’s Sandbox from it. If not, build a new one: create a Sandbox from the newest Snapshot, check out the task’s commit, run the install and build steps again, and take a Snapshot under the new key. The package manager’s cache is on the disk, so the install downloads mostly what changed. If the base image itself changed, run the whole setup on a new Sandbox instead. A Snapshot keeps the operating system it was taken with.

Two tasks can arrive with the same new key at once. Build the Snapshot for each key once, holding a lock in your own system. The other task waits for it, or starts from the previous Snapshot and installs what changed.

Bring services back after a start

A save keeps files, not running processes. Memory, open sessions and anything in /tmp or /run are gone after a stop. Enabled services start again at boot, as on any Linux server. Packages such as docker.io and postgresql enable their service when they’re installed. Enable your own systemd services with sudo systemctl enable.

A container comes back if it has a restart policy and existed when the Sandbox stopped. Docker starts it again when its daemon starts. The setup above created the containers with docker compose up -d:

services:
  cache:
    image: redis:7
    restart: unless-stopped

A database that was writing when a save happened is in the same state as after a power cut. Postgres and most other databases recover from that when they start. Test data you can’t recreate belongs in a dump file, not only in the database’s own files.

Keep the disk from growing

Every file on the disk is saved. Saved data is billed per GB a month (Pricing and limits). Old images and caches add up across many Snapshots. In a terminal on the Sandbox, df -h / and sudo docker system df show what takes the space. The setup’s last step prunes images before the Snapshot is taken. If your own caches grow, prune them there too.

Keep the images and caches the tests need. Pruning those makes every task download them again, which is the work this setup avoids.

Keep other secrets out of the copies

Files are copied as they are. A Sandbox created from a Snapshot has the same SSH keys and tokens as the source. A file you delete is gone from later saves, but it stays in earlier ones and in Snapshots taken before you deleted it.

Pass secrets to the commands that need them as environment variables, as with the git token above. For a model account, grant the task’s Sandbox a Connection. The credentials it delivers live in temporary storage that is never saved. If a credential does end up on a disk you copied, rotate it.