Sandboxed code execution
Code from a model is untrusted, so it must not run inside the API that serves users. An isolated sandbox fixes that, but a cold sandbox needs about 300 ms to boot. A warm pool of 2 keeps booted sandboxes ready.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | No warm sandboxes | Warm pool of 2 sandboxes |
|---|---|---|---|
| p99 latency (Code) | under 250 ms | 368 ms Fail | 106 ms Pass |
| Error rate (Code) | under 0.1% | 0% Pass | 0% Pass |
| p99 latency (Chat) | under 100 ms | 69 ms Pass | 69 ms Pass |
| Monthly cost | 8 USD/mo | 8 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Default warm pool | 0 sandboxes |
| Runs per second, one sandbox part | 10,000 ops/s |
| Sandbox call, median | 0.3 ms |
Warm pool of 2 · warm sandboxes: 2 · the first run waits 0 ms for a boot
How it works
An agent that writes code and runs it is useful, and it is a risk. The code can read files, call the network or use up the CPU. If it runs in the same process as the API, a mistake or an attack reaches the data and the users of the API.
A sandbox is a boundary. The code runs in a separate, limited environment, for example a micro virtual machine, with no access to the API's files and secrets and with limits on time, memory and network. The API sends code in and gets output back.
The boundary has a start-up cost. After 5 idle minutes a sandbox needs about 300 ms to boot. In the mini board the agent runs code in bursts of 8 runs/s, with 5 idle minutes between bursts. The first runs of each burst pay the cold start, and code-run p99 goes to about 370 ms, against a limit of 250 ms.
A warm pool keeps a few booted sandboxes ready. The first runs of a burst use them at once, and p99 stays near 100 ms. The pool costs a little all the time, so keep it small. A sandbox should be used once and destroyed, or fully reset, so one run cannot leave something for the next.
In the levels you run generated code in a sandbox and keep the first runs fast.