Performance results

Results from the nightly Performance run on 2026-10-30, at fb78964 (celld 0.6.3-ewhauser.0). Each number is the median over the repeats of a scenario.*

* Not real-world numbers

These numbers come from a small, shared GitHub-hosted runner (4 CPUs, linux x86_64). The load generator and every node share that one machine, and the bucket is a local store or MinIO that answers in well under a millisecond, where a real bucket takes tens. The runner’s CPU, disk, and neighbours also change from run to run.

Read them for the shape of a result and for how it moves between commits, not for what celld does in production or on your hardware. To get numbers for your own machine, run the scenarios yourself.

Column	Meaning
Offered/s, OK/s	Requests the open-loop generator sent per second, and how many succeeded
p50, p99, p99.9	Latency in milliseconds from each request’s scheduled start, so queueing counts
Bucket/OK, Core/OK	Bucket requests and core messages per successful request; these do not depend on the machine

Single node

One node on the dev backend (its local SQLite store) unless the scenario needs a bucket.

S1-stateless-ceiling

A stateless Worker at rising rates: peak requests per second, CPU per request, and latency at fractions of the peak. Sweep CELLD_MAX_STATELESS_ISOLATES and CELLD_TOKIO_THREADS with –env.

1 node, dev backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
noop@1000	1,000	1,000	0	0.23	0.33	0.56	0.003	0.00
noop@2000	2,000	2,000	0	0.23	0.29	0.59	0.001	0.00
noop@4000	4,000	4,000	0	0.22	0.34	0.85	0.001	0.00
noop@8000	8,000	8,000	0	0.25	1.31	7.04	0.000	0.00
noop@16000	16,000	11,620	0	672	5,374	5,898	0.000	0.00
cpu-10k@500	500	500	0	0.34	0.44	0.94	0.009	0.00
cpu-10k@1000	1,000	1,000	0	0.34	0.53	6.01	0.003	0.00
cpu-10k@2000	2,000	2,000	0	0.46	0.81	4.74	0.001	0.00
cpu-10k@4000	4,000	4,000	0	0.62	2.11	17.9	0.001	0.00

Count checks passed.

S2-warm-read

Reads of resident cells. A warm read makes no bucket request for its cell and exactly one core route and one output gate; testing.md quotes p50 ~1.1 ms and p99 ~7 ms on a fixed host.

1 node, dev backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
one-cell	1,000	1,000	0	0.66	0.94	2.37	0.003	3.00
1k-cells	1,000	1,000	0	0.75	1.95	3.52	0.003	3.00

Count checks passed. Verification: every acknowledged write read back (1,001 cells).

S3-write-bucket-proof

Writes on a single node, which proves each through the bucket: latency is about one PUT plus one ownership GET. The bucket gets an S3-like latency (needs a perf build); variants remove it or double it. Runs on the s3 backend: the dev store fsyncs every object under one SQLite writer, which caps a lone node at a few hundred uploads per second and would be what this measured.

1 node, s3 backend, 3 repeats.

local

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
write-100	200	200	0	4.74	8.45	14.6	2.015	3.01
write-4096	200	200	0	4.74	9.98	16.9	2.015	3.01
write-65536	200	200	0	5.25	10.5	22.0	2.015	3.01

Count checks passed. Verification: every acknowledged write read back (1,000 cells).

s3-like

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
write-100	200	200	0	67.6	117	135	2.016	3.01
write-4096	200	200	0	67.6	121	143	2.014	3.01
write-65536	200	200	0	67.6	121	143	2.020	3.01

Count checks passed. Verification: every acknowledged write read back (1,000 cells).

slow

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
write-100	200	200	0	225	483	606	2.017	3.01
write-4096	200	200	0	225	483	573	2.014	3.01
write-65536	200	200	0	225	483	606	2.016	3.01

Count checks passed. Verification: every acknowledged write read back (1,000 cells).

S4-write-coalescing

Many concurrent writes to one cell share one upload: PUTs per acknowledged write fall as concurrency rises.

1 node, dev backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
one-cell@20	20	20	0	67.6	125	135	2.153	3.06
one-cell@200	200	200	0	88.1	143	176	1.145	3.01
one-cell@1000	1,000	1,000	0	92.2	152	176	1.031	3.00
one-cell-coalesced	1,000	1,000	0	92.2	152	176	1.033	3.00

Count checks passed. Verification: every acknowledged write read back (1 cells).

S5-upload-slots

Writes spread over many cells at once, in bucket mode: each cell’s upload waits for one of the node’s 64 slots. durability.rs notes 4,000 cells committing together took ~9 s on a c3-standard-4. Runs on the s3 backend: the dev store fsyncs every object under one SQLite writer, which caps a lone node at a few hundred uploads per second and would be what this measured.

1 node, s3 backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
burst-100	100	101	0	67.6	109	121	2.025	3.02
burst-1000	1,000	1,000	0	836	1,212	1,278	1.675	3.00
burst-4000	4,000	4,000	0	5,374	7,733	7,995	1.477	3.00

Count checks passed. Verification: every acknowledged write read back (4,000 cells).

S6-hot-cell

One cell at rising rates, then the same cell while its isolate neighbours are busy (up to 32 cells share an isolate). Rejections past CELLD_MAX_CELL_REQUESTS show as http_5xx errors.

1 node, dev backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
alone@1000	1,000	1,000	0	0.59	0.88	1.82	0.003	3.00
alone@2000	2,000	2,000	0	0.66	1.44	3.77	0.001	3.00
alone@4000	4,000	4,000	0	1.38	4.22	6.53	0.001	3.00
alone@8000	8,000	3,761	42,390	32.3	129	152	0.001	6.38
with-neighbours	4,000	4,000	0	1.31	3.26	6.01	0.001	3.00

Count checks passed. Verification: every acknowledged write read back (32 cells).

S7-core-saturation

Small DO calls spread over 10,000 resident cells at rising rates, to find whether the serial core thread (loop.core_lag_us) or the V8 isolates saturate first.

1 node, dev backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
noop@1000	1,000	1,000	0	0.75	2.50	4.48	0.003	3.00
noop@2000	2,000	2,000	0	0.91	4.03	7.04	0.002	3.00
noop@4000	4,000	4,000	0	9.47	573	606	0.001	3.00
noop@8000	8,000	3,571	0	5,636	9,175	9,175	0.001	3.00
noop@16000	16,000	5,761	0	4,129	5,112	5,112	0.001	3.00

Count checks passed. Verification: every acknowledged write read back (10,000 cells).

S8-cold-activation

Restores from the bucket: the node restarts with its local state deleted, then reads touch a working set ten times the residency cap, so nearly every read evicts one cell and restores another. The ladder finds the churn a lone node sustains; beyond it, requests wait for residency capacity and fail with CapacityExhausted (about 38 per second on a 16-core Mac with S3-like bucket latency). Reports activations by restore path and bucket LISTs per download.

1 node, dev backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
cold-reads@10	10	10	0	225	319	336	13.520	4.07
cold-reads@25	25	25	0	401	541	639	19.144	3.96
overload	50	50	0	4,850	10,224	10,224	17.294	3.86

Count checks passed. Verification: every acknowledged write read back (2,000 cells).

S9-memory

Memory per resident cell, then after every cell is evicted. Read rss_bytes_max per phase against the resident count in end_state.

1 node, dev backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
resident	100	100	0	1.05	1.89	2.88	0.029	3.01
evicted	10	10	0	0.66	1.01	1.12	0.340	0.12

Count checks passed. Verification: every acknowledged write read back (10,000 cells).

S10-websockets

2,000 hibernatable sockets on 200 cells: echo at rising rates, auto-response pings (which must not reach the core), writes, and a 10-member broadcast. The main thread polls every socket pump, so watch loop.main_lag_us.

1 node, dev backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
echo@1000	1,000	1,000	0	0.33	0.50	1.01	0.003	3.00
echo@5000	5,000	5,000	0	0.33	0.88	2.37	0.001	3.00
echo@10000	10,000	10,000	0	0.56	2.11	3.65	0.000	3.00
ping	5,000	5,000	0	0.09	0.14	0.44	0.001	0.00
write	500	500	0	1.57	3.77	6.01	2.007	3.00
broadcast	100	1,001	0	0.72	1.44	2.50	0.005	0.30

Count checks passed. Verification: every acknowledged write read back (200 cells).

S11-alarms

2,000 alarms due at nearly one instant while reads continue. The collect step reports how late each alarm fired (alarm_late_* in ms, summed and maxed over cells).

1 node, dev backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
during-storm	200	200	0	1.18	1.95	2.88	2.020	3.34

Count checks passed. Verification: every acknowledged write read back (2,100 cells).

S12-services

Each platform service in turn: KV (one namespace is one cell), D1 run and batch, R2 by object size, Queue send (delivery lag in the after step), and Workflow creation.

1 node, dev backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
kv-get	1,000	1,000	0	0.66	1.05	2.11	0.003	3.00
kv-put	200	200	0	1.31	2.24	2.62	1.016	3.01
kv-put-large	10	10	0	17.9	27.1	35.8	5.000	9.13
kv-list	50	50	0	1.50	1.89	2.88	0.058	3.02
d1-run	200	200	0	2.37	6.01	8.45	2.046	3.01
d1-batch-10	100	100	0	2.88	6.78	8.45	2.082	3.02
d1-read	500	500	0	0.81	1.18	3.01	0.006	3.00
r2-get-1k	500	500	0	0.53	0.75	1.57	1.007	0.00
r2-get-1m	50	50	0	1.76	4.22	5.25	1.082	0.03
r2-list	20	20	0	9.98	11.5	11.5	1.145	0.06
queue-send	500	500	0	12.5	22.0	28.2	3.072	3.12
workflow-create	20	20	0	10.5	22.0	24.1	22.740	12.06

Count checks passed.

S12-workflow-steps

Workflow completion time as the step count grows: run() replays from the top on each wake, so a long workflow can cost O(steps^2). Each after step reports creation-to-completion time for all workflows so far.

1 node, dev backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
10-steps	5	5	0	11.0	27.1	27.1	23.040	12.10
100-steps	2	2	0	11.0	22.0	22.0	24.000	12.40

Count checks passed.

S14-overheads

The warm read and write paths with optional features on: compare each variant’s p99 and cpu with the base variant.

1 node, dev backend, 3 repeats.

base

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
read	2,000	2,000	0	0.94	3.52	6.27	0.001	3.00
write	200	200	0	67.6	117	152	2.017	3.01

Count checks passed. Verification: every acknowledged write read back (1,000 cells).

export

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
read	2,000	2,000	0	0.91	3.39	6.01	0.002	3.00
write	200	200	0	67.6	117	135	2.019	4.01

Count checks passed. Verification: every acknowledged write read back (1,000 cells).

otel-bucket

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
read	2,000	2,000	0	0.97	4.22	7.29	0.002	3.00
write	200	200	0	67.6	117	135	2.016	3.01

Count checks passed. Verification: every acknowledged write read back (1,000 cells).

S16-activation-rate

Fresh cells activated at fixed rates. Each activation opens its cell’s SQLite database inside the core thread’s StartRuntime effect (RuntimeManager::start_cell -> Worker::own_cell -> storage::open_at_epoch), so a burst of activations delays the core’s timers, node lease renewal among them; on a 16-core Mac a lone dev node’s core lag reached 18 s at 150 activations/s and the node fenced itself at 300/s. The error-rate check fails until activation leaves the core thread.

1 node, dev backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
fresh@50	50	50	0	8.96	23.0	54.3	6.028	4.02
fresh@150	150	150	0	9.98	30.2	35.8	5.970	4.00
fresh@300	300	300	0	71.7	336	401	5.898	3.98

Count checks passed.

Fleet and injected faults

Three nodes on MinIO: fleet proofs, forwarding, takeover, a slow follower, a frozen owner, a throttling bucket.

F1-fleet-proof

Writes in a fleet are proven by followers’ fsync rather than the bucket: testing.md reports ~25 ms for a fleet proof against ~600 ms for a bucket proof. Variants change the ensemble size and log transport.

2 nodes, s3 backend, 3 repeats.

2-nodes

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
write@100	100	100	0	4.03	58.4	79.9	0.457	3.53
write@300	300	300	0	3.65	58.4	84.0	0.306	3.51
write@1000	1,000	1,000	0	18.9	160	209	0.138	3.50

Count checks passed. Verification: every acknowledged write read back (1,000 cells).

3-nodes-stream

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
write@100	100	100	0	22.0	84.0	125	0.689	3.70
write@300	300	300	0	6.27	64.5	96.3	0.484	3.67
write@1000	1,000	1,000	0	37.9	217	303	0.264	3.67

Count checks passed. Verification: every acknowledged write read back (1,000 cells).

3-nodes

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
write@100	100	100	0	23.0	84.0	176	0.690	3.71
write@300	300	300	0	8.96	64.5	92.2	0.492	3.68
write@1000	1,000	1,000	0	50.2	233	352	0.258	3.67

Count checks passed. Verification: every acknowledged write read back (1,000 cells).

5-nodes

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
write@100	100	100	0	25.1	75.8	135	0.896	3.86
write@300	300	300	0	24.1	67.6	92.2	0.667	3.82
write@1000	1,000	1,000	0	113	336	418	0.422	3.80

Count checks passed. Verification: every acknowledged write read back (1,000 cells).

F3-forwarding

Requests sent to one node for cells spread over three: about two thirds are forwarded through the peer tunnel. Compare with the local phase.

3 nodes, s3 backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
spread	2,000	2,000	0	4.22	217	287	0.317	3.67
one-entry	2,000	2,000	0	2.24	33.8	79.9	0.142	3.67

Count checks passed. Verification: every acknowledged write read back (3,000 cells).

F5-takeover

An owner is killed mid-load (SIGKILL), and its cells move: the timeline shows how long its cells were unavailable, and the verification sweep that no acknowledged write was lost. Traffic enters through node 0, which is never killed. A second phase kills a node and deletes its disk.

3 nodes, s3 backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
kill	300	287	781	3.90	1,802	2,294	0.856	3.40
kill-disk-loss	300	289	681	3.39	1,933	2,425	0.720	3.03

Count checks passed. Verification: every acknowledged write read back (1,000 cells).

F9-idle-cost

Bucket requests an idle fleet makes on its own: node lease renewals, capacity samples, rebalance, waker scans. A deterministic count. The check bounds requests per second per node.

3 nodes, s3 backend, 3 repeats.

10-nodes

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
idle	1	1	0	0.78	13.6	13.6	16.133	11.92

Count checks passed.

3-nodes

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
idle	1	1	0	1.18	13.1	14.6	3.675	3.58

Count checks passed.

F9-idle-cost-resident

F9 with 10,000 resident cells: a resident cell should cost no bucket request while idle.

3 nodes, s3 backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
idle	1	1	0	1.18	13.6	15.1	3.675	3.56

Count checks passed. Verification: every acknowledged write read back (10,000 cells).

Network faults

Three nodes on MinIO, with the harness delaying, cutting, or resetting links between nodes and to the bucket.

N2-partitioned-follower

Node 2 loses every peer link (blackhole, both ways) for 30 s but keeps the bucket. Leaders must stop waiting on it as a follower and prove writes through the bucket; node 2 keeps its lease, so its own cells are unreachable from node 0 while the partition lasts. No acknowledged write may be lost.

3 nodes, s3 backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
partition	150	150	2	39.9	29,884	30,933	0.979	3.70

Count checks passed. Verification: every acknowledged write read back (600 cells).

N3-bucket-cut

Node 1 loses the bucket (blackhole) while its peers stay reachable. A node that cannot renew its lease must fence itself: the await_exit step times how long that takes, the timeline how long its cells were unavailable, and the verification sweep that no acknowledged write was lost. The node then returns on a healed network.

3 nodes, s3 backend, 3 repeats.

Phase	Offered/s	OK/s	Errors	p50	p99	p99.9	Bucket/OK	Core/OK
bucket-cut	150	150	0	7.04	606	1,475	0.607	3.19

Count checks passed. Verification: every acknowledged write read back (600 cells).

Run it yourself

The scenarios are in crates/perf. Build celld-perf and a node in the optimized lab profile; a debug build is too slow to measure.

cargo build --profile lab -p celld -p celld-perf --features celld/perf
target/lab/celld-perf list
target/lab/celld-perf run S2 --repeat 3     # one scenario, one dev node
target/lab/celld-perf run all --repeat 3    # every single-node scenario

The fleet and fault scenarios start several nodes on an S3-compatible bucket. Start MinIO, create a bucket named perf, then:

AWS_ACCESS_KEY_ID=perf AWS_SECRET_ACCESS_KEY=perf-disposable-password \
  target/lab/celld-perf run F1 F5 N2 --repeat 3 \
  --backend s3 --bucket s3://perf --endpoint http://127.0.0.1:9000

Each run prints a summary and writes target/perf/<run-id>/result.json. To compare two runs, or to see yours on this page:

target/lab/celld-perf compare before/result.json after/result.json
cd site && PERF_RESULTS=../target/perf pnpm dev

For steady numbers, use Linux on bare metal with a fixed CPU frequency and nothing else running. macOS fsyncs are full flushes of 10–15 ms, so its durability numbers are an order of magnitude slower. Performance tests has the scenario catalog, the injected and network faults, and how to read each number.