Boot time is the wrong race. We timed the work instead.
We ran three real open-source CI pipelines on a Workpaper computer, then set the results beside 15 published sandboxes. We placed 3rd, 4th and 3rd. Here is the method, the data, and every place we lost.
- Better-Auth CI
- 3rd/16
- Mastra CI
- 4th/16
- OpenClaw CI
- 3rd/15
- Runs completed
- 56/56
Abstract
Sandbox benchmarks usually time how fast a VM starts. Agents and CI spend their time after that: cloning, installing, type-checking, testing. We ran the unchanged workloads of the open-source HPC Sandbox Benchmarks on 56 fresh Workpaper computers, one at a time, and compared them with Starsling’s published run of 3 October 2026. On end-to-end pipelines Workpaper placed 3rd of 16 (Better-Auth), 4th of 16 (Mastra) and 3rd of 15 (OpenClaw). Storage and Git work are a strength. Network is our weakness: we come last on cold DNS lookups and internet throughput. This is a self-run, not a leaderboard listing, and section 5 lists what that means for the numbers.
1Why time the work
A computer that boots in 200 ms and then takes four minutes to install dependencies is a slow computer. Start time matters for bursts. For an agent doing real work, what matters is how long until the tests come back.
The HPC Sandbox Benchmarks measure exactly that. They take pinned commits of real projects and run their actual CI steps inside a fresh 4 vCPU / 8 GiB sandbox, alongside the usual CPU, memory, disk, network and database tests. We asked two questions:
- How does a Workpaper computer compare on real CI pipelines with the providers in the published dataset?
- Where does it fall short, and why?
2Method
We followed Raj Jain’s ten-step systematic approach to performance evaluation (The Art of Computer Systems Performance Analysis, 1991). It forces every choice that shapes a result to be written down before the result is read.
| 1 · Goal | Compare time to finish developer workloads on a 4 vCPU / 8 GiB computer. |
|---|---|
| 2 · System | One Workpaper computer, a microVM on a dedicated host. Measurement starts after the computer is running; creation time is excluded. |
| 3 · Services | Clone, install, build, lint, type-check and test three projects; plus CPU, memory, disk, network, SQLite, Git and PostgreSQL tests. |
| 4 · Metrics | Wall time per task (s); throughput (MB/s, IOPS, TPS, runs/s); latency (ms). |
| 5 · Parameters | 4 vCPU, 8 GiB RAM, about 36 GiB root disk, one VM at a time, upstream toolchain image v8 pinned by digest. |
| 6 · Factors | One: the provider. Everything else is held fixed by the upstream scripts. |
| 7 · Technique | Direct measurement. No models, no simulation. |
| 8 · Workload | Upstream workload scripts at 7fbfde1df756, unchanged. |
| 9 · Design | Pipelines: 12 fresh VMs × 1 pass each. CPU: 5 VMs × 2 passes. Other tests: 3 VMs × 2 passes. |
| 10 · Analysis | Median of per-VM medians. A pipeline total is the sum of its task medians. Positions are simple ordering, not significance tests. |
Comparison values come from Starsling’s run 37073680030 (3 October 2026), which the upstream project marks as partial.
3Results
3.1 End-to-end pipelines
On Better-Auth, Workpaper finished in 147.6 s, 25% behind the fastest (Brezel, 118.1 s). Across all three projects, only Brezel, Namespace and boat finished ahead of us at least once.
3.2 Where the time goes
- Installs are fast. Mastra’s cold install (37.7 s) is 2nd of 16; Better-Auth’s is 4th of 16.
- Type-checks hold up. OpenClaw’s tsgo type-check is 3rd of 15; Node.js tooling throughput is 4th of 16.
- Git and the disk lead. Common Git operations are 2nd of 16; 1 MB sequential writes reach 7,697 MB/s, 3rd of 16.
- Clones are slow. Every project’s
git cloneis 13th of 16. Section 4 explains why.
3.3 Every measurement
All 64 measurements, including the ones that make us look bad. Filter by area or search.
| Measurement | Workpaper | Position | Evidence |
|---|
No measurements match.
3.4 What a run costs
Starsling also prices providers: the published hourly rate for 4 vCPU / 8 GiB, which, multiplied by the measured time, gives the cost of a run. Its method ranks only rates that are a fixed CPU-plus-memory charge with no plan attached. Of the 7 providers it ranks, the cheapest is boat at $0.036 an hour.
Workpaper’s Pro rate, $0.020 an hour, is the lowest in the comparison, but Pro is a $15-a-month plan, returned as usage credit, so Starsling’s rule would leave it unranked. Our plan-free rate, pay as you go at about $0.067 an hour, would rank 2nd of 8.
| Provider | $/hour | Seconds | $ per 1,000 runs |
|---|---|---|---|
| Workpaper Pro ¹ | $0.020 | 147.6 | $0.82 |
| boat | $0.036 | 135.8 | $1.36 |
| Workpaper pay as you go | $0.067 | 147.6 | $2.73 |
| tama | $0.074 | 220.6 | $4.53 |
| Novita | $0.233 | 181.9 | $11.79 |
| Daytona VM | $0.331 | 156.6 | $14.41 |
| Freestyle | $0.264 | 351.2 | $25.81 |
| E2B | $0.331 | 295.4 | $27.18 |
| Runloop | $0.634 | 559.6 | $98.48 |
4Discussion
Why we win where we win. Each computer gets dedicated cores and local NVMe through a short storage path, and nothing else was competing for them during the run. Work that is mostly CPU and disk (installs, type-checks, Git, database writes) shows it: PostgreSQL read-write throughput is 4th of 16.
Why we lose where we lose. Almost every bottom-of-table result is network:
- Cold DNS was last, and is now fixed. In this run github.com took 44.0 ms and pypi.org 68.0 ms (15th of 15), because computers resolved through a public resolver far from the host. Since the run, their DNS goes to a resolver in the same datacenter: in a spot check after the change, the same five lookups took 1 to 4 ms. The table above still shows the original run.
- Internet throughput is last. WAN download is 896 Mbit/s (15th of 15). The host has a 1 Gbit/s uplink, and the test saturates it.
- Clones pay for distance. GitHub serves repository data from far away from our host; the clone gap is round trips, not compute.
- Memory bandwidth is low. STREAM Triad is 14th of 16. The host has two memory channels; providers on server platforms have more.
The practical reading: for agents that clone once and then build and test for minutes, the pipeline numbers are the ones that matter, and those put Workpaper in the top four. Workloads that move a lot of data over the internet will see the network limits.
5Threats to validity
We use the four standard categories from empirical software engineering (Wohlin et al., Experimentation in Software Engineering).
Construct validity: do the numbers measure “time to finish the work”?
Pipeline totals sum per-task medians. They exclude computer creation, setup and waits between tasks, so a real end-to-end run takes longer than the total shown. Every provider in the dataset is summed the same way.
Internal validity: could something other than the provider explain the result?
We ran our own measurements, through our own runner, on a different date from the reference run. The runner executes the unchanged upstream scripts, but it is not the official submission harness. All of our VMs ran on one host, one at a time, so our repeats measure repeatability, not variation across a fleet. The reference run is marked partial upstream.
External validity: will a customer see the same?
Only under similar load. One VM at a time is a quiet host; a busy multi-tenant host may be slower. The run used about 36 GiB of disk because the suites need 30 GiB free, more than the 20 GB in our published large plan. Disk size can affect disk results, and you should weigh that when comparing price.
Conclusion validity: are the positions real differences?
Positions order medians; we ran no significance tests. With 12 VMs per pipeline, small gaps between neighbours (Mastra’s 4th place is within 17 s of Namespace) can swap on another day. Gaps of tens of seconds are more robust.
6Reproducibility
The workloads, the toolchain image and the reference data are public and pinned. Our raw results, every per-VM value, are on GitHub.