Engineering · · 5 min read
Why we built our own agent sandbox
Daytona, Modal, and E2B can start isolated compute quickly. Our evaluation runtime needed a one-shot lease with tenant and execution binding, plus provider-confirmed destruction.
Daytona, Modal, and E2B each provide isolated compute for agent code. We built Sandboxwich because an evaluation run needed a narrower contract: claim a cell that has never served another tenant, bind it to one execution, and never return it to inventory. If teardown cannot be proven, the cell stays quarantined. That changed the state machine, retry protocol, in-pod credentials, and pool reconciler.
Documented warm-start and lifecycle behavior#
| Platform | Fast-start mechanism | Documented lifecycle |
|---|---|---|
| Daytona | Managed warm pools hand a pre-created sandbox to a matching request and replenish the pool. Warm claims require default resources and user, with no custom environment variables, volumes, or secrets. | A claimed sandbox supports the ordinary Daytona lifecycle. Containers can stop and start with their filesystem preserved; VMs can pause and resume with filesystem and memory preserved. |
| Modal | A published warm-pool example uses a queue, a scheduled function, readiness probes, and caller-kept expiry timestamps. | A running sandbox can be retrieved by object ID and reused. The default maximum lifetime is five minutes and can be configured up to 24 hours. |
| E2B | Templates snapshot a prepared sandbox, including its filesystem and running processes, and are documented to load in about 80 ms. | A sandbox can pause and resume with memory and files preserved. Paused sandboxes remain until explicitly killed. |
A clean filesystem at claim time cannot prove that no earlier process saw a credential, changed memory state, or influenced a daemon that survived cleanup.
A cell gets one lease#
A cell starts as ready(generation=1, never exposed). A successful claim atomically changes it to leased(generation=2), bound to the tenant and execution. The only terminal outcomes are destroyed and quarantined.
ready, generation 1, never exposed
|
| atomic claim
v
leased, generation 2, tenant-bound
|
| release or expiry
v
stopping
/ \
deletion proven deletion uncertain
/ \
v v
destroyed quarantined
No route transitions a leased, destroyed, or quarantined cell back to ready. The pool creates a replacement.
A retry cannot claim twice#
A caller can lose the response after the server commits the lease. Retrying a generic create request can then allocate a second sandbox, and changing the request under the same identifier can bind the wrong execution. Sandboxwich treats claim_id as a durable fence:
- The first request stores a digest of the complete claim.
- An exact retry returns the same lease and attestation.
- A retry that changes the binding or TTL receives
409. - If the original claim found no inventory, its retries stay empty even after a cell becomes available.
Without a durable empty result, a delayed retry could acquire authority after the caller moved on.
Daytona documents distributed locking in its control plane, Modal supports unique names for running sandboxes, and E2B tells webhook consumers to deduplicate lifecycle events by event ID. The public create-interface documentation we reviewed on 18 August 2026 has no equivalent of Sandboxwich's combination of a caller claim fence, full-request digest, generation check, and durable empty replay, so we built that protocol into our control plane.
The lease is scoped to the evaluation#
A lease binds:
- cell and lease IDs;
- lease generation and expiry;
- signed release set, runtime class, and policy digest;
- organization, workspace, thread, and runner session.
The control plane signs that tuple with HMAC-SHA256 and persists only the digest.
Inside a prewarmed pod, a credential-bearing control sidecar validates the lease and sends a sanitized activation over mTLS to a launcher with no Sandboxwich API credential. The raw attestation stays in the sidecar.
E2B's private-beta workload identity issues short-lived JWT-SVIDs for a project, sandbox, and execution, and its egress proxy can substitute a token without exposing the token value to sandbox code. Our attestation also carries the pool generation, release tuple, execution binding, and one-shot lease state.
Verify the admitted pod#
Sandboxwich places Kubernetes workloads behind gVisor or Kata Containers via runtimeClassName. Writing that field into a manifest proves nothing, because a mutating admission webhook can rewrite or remove it.
After Kubernetes admits the pod, the worker reads it back and checks spec.runtimeClassName, and checks again before command reuse. A missing or mismatched value ends in the terminal runtime_class_boundary_unverified failure and teardown.
Their hosted APIs do not expose runtimeClassName, so the check applies where you operate the Kubernetes layer, as we do.
Cleanup is a capacity rule#
A timed-out delete call does not prove the provider stopped the workload, and an immediate replacement could leave both alive.
The pool therefore counts stopping and cleanup_pending cells against capacity, and cleanup continues after an API restart or a zero pool target. The control plane marks the provider absent only when the completed stop matches the expected lease, generation, and disposition.
Selection boundary#
A workload that needs a persistent session conflicts with the sterile-cell state machine, because a released cell cannot resume. Our evaluation runtime needs a record of no prior tenant exposure, claim replay, execution-scoped authority, an admitted runtime class, and confirmed deletion.
Current limits#
Sandboxwich is pre-1.0. Its capability matrix marks the Kubernetes provider paths experimental, and Kata's virtual_machine isolation still needs a recorded live conformance run on nested-virtualization infrastructure. The HMAC assertion authenticates control-plane state and carries no TPM or confidential-computing evidence.
The implementation is public in the Sandboxwich repository. Start with the sterile-cell design, then read the pool reconciler and the HTTP contract tests.