Performance ledger#

Numbers measured on this host (WSL2, kernel 5.15, ext4 on the virtual disk /dev/sdd, 32 cores). Re-measure after any storage-layer change; just bench fails on >20 % regression once bench/thresholds.toml exists (M9).

M0 fsync latency (2026-10-15)#

Two instruments, same 4 KiB-file-per-iteration workload, each variant in its own fresh temporary directory under ~/.local/state/swarmops-bench:

• crates/swarmops-store/tests/fsync_percentiles.rs (cargo nextest run -p swarmops-store --run-ignored all -E 'test(fsync_percentiles)' --no-capture): 1,000 independent writes per variant, every write timed on its own — the table below is from this run; percentiles are nearest-rank over the 1,000 per-write durations.
• crates/swarmops-store/benches/fsync_latency.rs (cargo bench -p swarmops-store): criterion means for regression tracking only (criterion reports per-sample means over tens of iterations, which smooths tails; it cannot give per-write percentiles).

Criterion means for the same variants: 49 µs, 4.07 ms, 4.01 ms, 8.81 ms (consistent with the medians above).

Decision. p99 of a durable single-file write is 5.4 ms (max 7.4 ms over 1,000 writes), under the 20 ms threshold the M0 plan set, so no --durability fast flag is introduced now; the default stays full fsync (objects sync_all + the fsync of each touched fan-out directory, batched once per commit in flush() + the metadata store's syncs: SQLite synchronous=FULL in the M0 plan, and since the decision of 2026-10-15 for a native metadata store, the journal's two fdatasyncs per transaction plus two fsyncs per moved ref (one per deleted ref), storage spec §3). The rename + directory-fsync path has a 16.8 ms max, so a commit's directory fsyncs can occasionally approach the threshold; they are paid once per commit, not per object.

Consequence for M1. A commit that writes N new objects pays ~4 ms per object if the fsyncs are serial (100 files ≈ 0.4 s), which cannot meet the "≤ 5 ms + 0.3 ms per changed file" commit target. FsObjectStore::put_batch must issue the per-file fsyncs in parallel (rayon) and fsync each touched fan-out directory once at flush(). M1 measures put_batch throughput at 1, 8 and 32 concurrent fsyncs (benches/put_batch.rs); if the virtual disk serializes flushes (throughput does not scale), the --durability fast option (skip per-object fsync, keep the metadata journal's syncs — the SQLite WAL fsync in the M0 plan —, documented as "safe against process crash, not power loss") is reconsidered with that measurement attached here.

M1 store level (2026-10-15)#

The store-level half of the M1 Task 7 experiments; the CLI-level contention test follows with the CLI. Same host as M0: WSL2 (kernel 5.15.153.1), ext4 (data=ordered) on the virtual disk /dev/sdd, AMD Ryzen 9 7950X3D (16 cores, 32 threads), 106 GiB RAM; Rust 1.99.0, optimized builds (cargo bench; the contention test with --release). Every store lives in a fresh temporary directory under ~/.local/state/swarmops-bench.

The host was not quiet. Three JetBrains remote-dev-server processes kept about 8.5 cores busy throughout, and other worktrees were compiling and testing. Before each run, the harness waited up to 10 minutes for the 1-minute load average to drop below 8. It did so three times, before put_batch runs 1, 2 and 4; every other run started at load 8.5–20.1. Each table gives the load at the start of each run, and the range sampled every 15 s during it. Threads blocked in fsync count towards the load average, so the I/O-bound runs raise it themselves. Every measurement ran twice. Where the two runs agree within 20 % the better one is shown; otherwise both are.

put_batch fsync scaling#

crates/swarmops-store/benches/put_batch.rs (cargo bench -p swarmops-store --bench put_batch). Every iteration puts 256 new 4 KiB objects into a fresh store whose 256 fan-out directories already exist, as they do in a created store. The objects are incompressible, so each file holds 4 KiB. The batch runs on a pool of 1, 8 or 32 threads (FsObjectStore::set_parallelism). Per object that is a temp file, sync_all and a no-clobber link. flush then fsyncs each fan-out directory the batch touched (about 162), one after another. Each figure is criterion's median of 10 samples; objects/s = 256 / median.

The "busy writer" variants keep a thread appending 64 bytes to a file of its own every 100 µs, throughout, as some process always does on a swarm host. The other variants see whatever the host did meanwhile. These are runs 3 and 4, which agree within 12 % everywhere:

• run 3 started at load 12.0 (10.9–23.1 during the run);
• run 4 started at load 7.8 (7.8–18.6).

The "vs 1" ratios divide each cell's better run. Run by run (run 3 / run 4), 32 threads against 1 give 13.8× / 14.6× for put_batch + flush, 2.56× / 2.45× with the busy writer, and 14.9× / 13.2× for put_batch alone; 8 threads against 1 give 4.3× / 4.3×, 2.0× / 2.0× and 4.4× / 4.4×.

flush alone, after a 256-object batch: 4.1 ms (4.2 / 4.1 ms), and 571 ms with the busy writer (571 / 580 ms).

Runs 1 and 2 came before the busy-writer variants existed, and the host put them in different regimes. They disagree, so both count:

The per-object fsyncs scale: 14.8× on 32 threads, because the virtual disk and ext4's journal commit concurrent fsyncs together. flush scales only when nothing else is writing to the filesystem. On this kernel, a directory fsync becomes a full journal commit of its own whenever any process has metadata pending on the filesystem. That happens even for a directory that has not changed.

A side check bears this out (the two scripts are in the appendix below; the load was 11–12 throughout). 162 fsyncs of unchanged directories took 0.1 ms per round on a quiet filesystem. With one shell loop appending to a file, they took 567–616 ms per round over 5 rounds, every fsync a 3.7 ms commit (p50). The same fsyncs issued from 32 threads took 44–47 ms, against 567–585 ms serially in the same session.

So on a busy host a commit's flush costs about 3.5 ms per touched fan-out directory: 571 ms for 256 objects. That breaks the "≤ 5 ms + 0.3 ms per changed file" commit target, which is 82 ms for 256 files, although put_batch alone on 32 threads (67 ms) meets it.

Index lookups#

crates/swarmops-store/benches/index_lookup.rs (cargo bench -p swarmops-store --bench index_lookup). One index per size is built outside the timed loops: a chain of real (BLAKE3) changeset ids, each changeset with a delta hash of its own. Its lookups are timed twice. First the last 1,000 records sit in the unsorted tail and the hits ask for tail records: the binary search misses, then the tail is scanned. Then the index is compacted whole and the hits ask for any of its records. The files are in the page cache, as a hot store's are. Each lookup cycles through 1,024 targets in a scattered order. p50 is criterion's median per lookup. Run 1 started at load 12.3 (12.1–20.3 during the run), run 2 at load 10.3 (10.3–11.6).

The one cell with two values is where the runs differ by more than 20 % (run 1 / run 2). Run once more on the integration branch, where the bench lays the index out with Index::create (Index::open creates nothing), at load 23.7 (21.9–27.4 during the run): every median within 12 % of the value shown. Every lookup takes under 11 µs, so the spec's read target (< 1 ms) holds by two orders of magnitude, at 200,000 changesets as at 10,000. A lookup is about log2 n read_ats of one 32-byte key (14 at 10,000, 18 at 200,000), so going from 10,000 to 200,000 changesets adds about 2 µs. A full tail adds 0.5–3.2 µs: one 32 KiB read_at and a linear scan. find_prefix pays the tail scan even when the sorted view holds the match. find miss is what every insert of a new changeset asks, under the lock.

fsync fan-out (Index::sync)#

crates/swarmops-store/benches/fsync_fanout.rs (cargo bench -p swarmops-store --bench fsync_fanout): five files in a fresh directory. Before every iteration (untimed), each gets 64 KiB of new data appended, as the index's appends grow its files. The iteration then makes them durable:

• serially, one sync_data per file, each file opened for it, as Index::sync does;
• from five threads at once (std::thread::scope, thread start included);
• for scale, one such file alone.

Criterion median. Run 1 started at load 11.5 (11.5–12.6), run 2 at load 17.1 (17.1–22.3).

Serial costs five fsyncs (5 × 3.8 ms). Parallel costs about two, because ext4 commits the five files' metadata in a shared journal commit: 2.5× faster (2.48× in run 1, 2.97× in run 2). Index::sync runs at every 32nd transaction and before a compaction, under the lock. Parallel would save about 11 ms there, about 0.35 ms per transaction on average.

32 writer processes on one metadata store#

crates/swarmops-store/tests/contention.rs, ignored: cargo nextest run --release -p swarmops-store --run-ignored all --test all -E 'test(/^contention::/)' --no-capture; just check-slow runs it unoptimized.

• 32 child processes start together, and each runs 1,000 MetaStore::write() transactions. Each transaction inserts the next changeset of the child's own chain and commits, with the default 10 s lock timeout. A Busy is counted and the transaction tried again.
• The lock wait is the time until write() returns: the wait for meta/lock, plus the tail checks and replay under it. Those alone take 0.06–0.09 ms (p50; p99 0.12 ms) uncontended, on the final store.
• Two pacings were measured:
  • back to back, the default: a writer that loops, as a bulk import or a long-lived process does;
  • 2 ms between a child's transactions (SWARMOPS_TEST_CONTENTION_PAUSE_US=2000), as an agent leaves at least a process start between two commits.

The two runs of each pacing disagree by more than 20 % on max and on the Busy count, so all four are shown. Under just check-slow (unoptimized, beside the other slow tests) the default run failed the same way, twice. The first gate run started at load 14.8: 270 Busy, p99 lock wait 8.45 s, max 49.7 s, spread 1.47× and 111.0 transactions/s. The run on the final commit of the measurement started at load 12.9: 260 Busy, p99 lock wait 8.11 s, max 40.7 s, spread 1.55× and 108.6 transactions/s. Neither gate run sampled the load during the run.

The journal at the end of every run holds the same:

• 32,031 frames: 32,000 transactions and 31 Compacted markers;
• 1 frame past journal.ckpt;
• journal.log of 3,711,565 bytes (116 bytes per one-parent changeset frame).

The four runs above checked the store only after the Busy assertion, so its checks ran there only in 2 ms run 2, where they passed. The test now checks the store first. A third back-to-back run made that way:

• passed every store check: 32,000 changesets, each chain intact, one frame per transaction, and seqs contiguous in journal order, so the Busy retries lose nothing;
• then failed on Busy: 449 of them, p99 lock wait 9.29 s, max 78.7 s, spread 1.74×, 111.9 transactions/s, starting at load 12.3 (11.1–25.0).

A transaction holds the lock for its two journal sync_datas: 7.5 ms p50 when it does not wait, M0's 2 × 3.8 ms. Every 32nd transaction adds the index sync and the checkpoint (uncontended p99 30 ms), and about every 1,025th adds a compaction. The lock is never idle, so throughput stays at 100–121 transactions/s whatever the pacing.

What differs is who waits, and MetaLock's poll makes that unfair. A writer that fails try_lock sleeps 10 ms, so the writer that has just released the lock takes it again, a few microseconds later, before any poller wakes:

• Back to back, that lets one writer keep the lock for runs of transactions while the others time out. The lock-wait p50 is 0.08 ms, but p99 is 8.2 s, and the worst transaction waited through seven 10 s timeouts.
• 2 ms apart, the lock goes to whichever poller wakes first: a lottery rather than a queue, and pollers that sleep a fixed 10 ms can stay unlucky. p50 is 130–140 ms and p99 1.9–2.3 s. Every child waited over 1 s at least once in both runs, 20 and 5 of them over 5 s, and in run 1 one wait reached the 10 s timeout.

So the spec's "32 processes are serialized fairly by the flock queue" does not hold for this lock. Nor does the plan's "Busy only after the 10 s lock timeout, 0 failures": Busy comes at either pacing.

Side experiment (the program is in the appendix below): 32 processes × 300 transactions, each two small appends with sync_data under the lock, taken either by std's File::try_lock every 10 ms (what MetaLock did when these runs were made) or by the blocking File::lock (flock(LOCK_EX)).

Each cell gives run 1 / run 2, and the first row the load at the start of each. These runs did not wait for the load to drop, and nobody sampled it during them.

The polling column reproduces MetaLock's behavior above.

A blocking flock wakes a waiter the moment the lock is released. That removes the 10 ms window in which a poller sleeps through the release. 2 ms apart, the lock rotates almost as a queue would: p50 225 ms ≈ 31 × 7.3 ms, and spread 1.00×. Back to back, the releaser still often takes the lock again before the woken waiter runs (p50 0.01 ms), but the tail drops from 7–9 s to about 1 s. The kernel's flock is not FIFO; a looping writer would need an explicit handoff, such as a ticket file, to be strictly fair.

Decisions#

• (a) --durability fast: not justified by this data; parallel directory fsyncs in flush come first. Applied in Task 7c, which parallelizes the directory fsyncs and adds no flag ("M1 store fixes (7c)" below). The rule: reconsider the flag if 32-way put_batch throughput is below 4× the 1-way.
  • The trigger is met only with another writer on the filesystem, which is the normal state of a swarm host: put_batch + flush on 32 threads against 1 then reaches 2.5× (405 against 161 objects/s; 2.56× and 2.45× in runs 3 and 4), below 4×. While nothing else writes it reaches 14.5× (3,666 against 253 objects/s; 13.8× and 14.6×).
  • The shortfall is not what a fast mode would skip. The per-object fsyncs scale 14.8× (14.9× and 13.2×); flush's serial directory fsyncs take 571 of the 632 ms. A fast mode that skipped the object fsyncs would leave those in place.
  • Issuing the directory fsyncs in parallel keeps full durability, and took 162 of them from 575 ms to 45 ms in the side check. That comes first, and this data does not justify a --durability fast flag, which exists only with a measurement here that does.
• (b) Parallel Index::sync(): recommended. Applied in Task 7c (below). The rule: only if five parallel fsyncs are at least 2× faster than serial.
  • Measured: 7.5 ms against 18.8 ms, 2.5× (2.48× and 2.97× in the two runs).
  • It saves about 11 ms at every batched sync (every 32nd transaction, and before each compaction), all of it time under the lock.
• (c) A blocking lock instead of polling: recommended. Applied in Task 7c (below). The rule: if the fairness spread exceeds 2× or the p99 lock wait exceeds 1 s. The p99 lock wait carries the decision: it exceeds 1 s in every run, while the spread exceeds 2× only in back-to-back runs 1 and 2.
  • Back to back: p99 lock wait 8.2 s and 9.4 s, spread 2.18× and 2.01×, and 386 and 427 Busy, so both criteria are exceeded. The third back-to-back run (p99 9.3 s, spread 1.74×) and the two just check-slow runs (p99 8.5 s and 8.1 s, spreads 1.47× and 1.55×) exceed the p99 criterion alone.
  • 2 ms apart: p99 2.3 s and 1.9 s, with spread 1.19× and 1.09×. The p99 criterion is still exceeded, and Busy still occurs.
  • At 2 ms apart the spread understates the problem: over a whole run the lottery evens out, so every child finishes close to the others while single waits run to seconds. The wait tail is the indicator.
  • The side experiment says the change works: with the blocking File::lock, Busy disappears, and the p99 wait drops from 7–9 s to 0.8–1.1 s back to back and to 0.3–0.8 s 2 ms apart. The 10 s timeout can stay, with a thread that blocks in File::lock while the caller waits for it with a timeout; these runs never came near it. A strictly fair order for a writer that loops (a bulk import) needs a FIFO handoff on top of it.

M1 store fixes (7c)#

Task 7c applied the three decisions above, all at full durability:

• (c) MetaLock waits in the kernel's flock queue. An acquire tries the lock once without blocking; if another handle holds it, a helper thread blocks in File::lock (flock(LOCK_EX)) on the acquire's own open file, and the caller waits for it up to the 10 s timeout. A helper that gets the lock after its caller gave up lets it go at once.
• (b) Index::sync makes its five sync_datas from five threads at once (MetaStore::set_parallelism, 5 unless set).
• (a), the first step: FsObjectStore::flush makes its directory fsyncs at once, on the object store's pool (FsObjectStore::set_parallelism, else rayon's global pool). The flag stays out.

At parallelism 1 both kinds of sync run one after another in their old order. LocalRepo sets that under SWARMOPS_TEST_CRASH_AT, so crash sweeps keep meeting the same call at the same count.

The disk was 1.6–3.7× slower than M0's. Another process on the host was writing about 110 MB/s near the end of the lock-change run (seen at its 13th minute, gone 9 minutes later), and other worktrees' tests and benches ran during some runs (named with each). Beside the load, every run gives a raw fdatasync of a 120-byte append (the median of 40) at its start and end: 13–15 ms while that writer ran, 6.6–7.9 ms after it stopped, against M0's 4.1 ms. So absolute times here do not compare with the 7a section; waits counted in transactions, Busy and the spread do.

32 writer processes, back to back#

crates/swarmops-store/tests/contention.rs as above (32 × 1,000, --release), run once with the lock change alone and twice with all three changes. The 7a polling runs are the back-to-back runs 1 and 2 above. "In transactions" divides each wait by the run's mean transaction time (wall / 32,000), the time the lock takes to pass once.

Tests of other worktrees ran during some of these runs, on the same disk: Task 6's crash sweeps in 9 of the 39 load samples of run 1, and Task 7b's fsync_latency bench in 10 of the 29 samples of run 2. Task 7b's 32-process CLI contention test was running 8 minutes before the lock-change run started and had ended 11 minutes into it; that run did not log other processes, and neither did the 7a runs.

Busy is gone, in every run. Whether the writer that has just committed takes the lock straight back depends on the host. At load 32–33 the woken waiters won: the lock rotated, and the median wait was one round of the 31 other writers (33–37 transactions). At load 10–16 (run 2) the committing writer usually asked again and got the lock before a woken waiter ran (median 0.11 ms), as the blocking flock did in the 7a side experiment. Either way every child finished within 16 % of the others (spread 1.06–1.16×, from 2.0–2.2× with polling). Counted in transactions, the p99 wait is 6–8× shorter, and the longest 19–48× shorter.

The tail is bounded, not gone. At each release the kernel wakes every waiter and the first to run takes the lock, so a waiter can lose many releases in a row. A uniform random winner among 31 waiters predicts p99 140 and, over 32,000 waits, a longest wait of about 316 transactions; the rotating runs show 131–157 and 283–359. So the longest wait is a few hundred transactions: 2.6 s at run 2's 14 ms per transaction (it would be about 2.5 s at 7a's 8–9 ms), but 5.2–8.9 s at 18–25 ms. That stays inside the 10 s timeout, but on a slow disk not by much. An order fair to the transaction (at most 31) needs a FIFO handoff on top of flock.

Throughput follows the disk: an uncontended transaction took 13.2–15.3 ms at the end of these runs (7.4 ms in 7a), and the lock passed every 14–25 ms. The lock's own hand-off (a helper thread woken in the kernel, then a channel to its caller) does not show at that scale. In one process, 32 threads × 16 turns of 2 ms (tests/lock.rs, twice each) wait at most 0.12–0.13 s with the kernel queue (load 16.0), against 1.22–1.25 s with the polling lock (load 17.0).

Index::sync#

crates/swarmops-store/benches/fsync_fanout.rs, once, with two new variants that time Index::sync itself: an index of its own gets the 32 records of a checkpoint's frames appended before every iteration (untimed), then syncs at parallelism 1 or 5. It started at load 13.1 (12.4–13.7 during the run), raw fdatasync 8.0 ms at its start and 7.1 ms at its end; nothing else was benchmarking.

One sync_data took 6.9 ms here against 3.8–4.0 ms in 7a, so the disk ran 1.8× slower; in those units the raw variants repeat 7a (serial 5.6 single syncs, 7a 4.9–5.6; parallel 2.2, 7a 1.9–2.0), 2.5× apart (2.48×; 7a 2.48× and 2.97×). Index::sync itself, after a checkpoint's 32 records, takes 24.1 ms one file after another and 9.2 ms at the default parallelism: 2.6× faster, and 14.9 ms less time under the lock at every batched sync (0.47 ms a transaction on average). It costs less than the raw serial variant because 32 records add about 1 KiB to a 32-byte column, not 64 KiB.

flush#

crates/swarmops-store/benches/put_batch.rs, once, unchanged: flush now makes its directory fsyncs on the store's pool, so the 8- and 32-thread variants fsync 8 and 32 at a time and the 1-thread variants as before. It started at load 13.1 (13.3–24.9 during the run), raw fdatasync 7.2 ms at its start and 7.9 ms at its end; nothing else was benchmarking.

Each row is one median, and the host changed between rows: the two 8-thread rows disagree the wrong way round (481 ms, and 304 ms with the busy writer). The object fsyncs cost what they did in 7a: put_batch alone takes 984 / 228 / 66.3 ms on 1 / 8 / 32 threads (7a: 999 / 230 / 67.0 ms). But every directory fsync was a journal commit of its own today even without the bench's busy writer, so one thread's serial flush took about 870 ms (1,857 − 984 ms).

From 32 threads at once, flush takes 6 ms of the 72.3 ms that put_batch + flush take, and alone, with the busy writer, 65 ms against 571–580 ms one after another in 7a: 8.8–8.9× faster (the 7a side check had 575 against 45 ms).

Against decision (a)'s rule, put_batch + flush on 32 threads now reaches 25.7× the 1-thread throughput (3,542 against 138 objects/s; 7a 13.8× and 14.6×), and 15.9× with the busy writer (2,588 against 163 objects/s; 7a 2.56× and 2.45×). That is above 4× in both, so nothing here calls for a --durability fast flag. A 32-thread commit's 256 new files take 72 ms, inside the "≤ 5 ms + 0.3 ms per changed file" target (82 ms), and 99 ms with the busy writer on this slower disk.

Appendix: the side-experiment programs#

The side experiments above ran outside the workspace; here they are, so that they can be run again.

Directory fsyncs (Python 3). python3 dirsync.py <dir> times 162 fsyncs of unchanged directories in a fresh directory under <dir>, in 5 rounds; python3 dirsync_par.py <dir> times the same serially and from 32 threads. For the busy case, a shell loop appended to a file on the same filesystem meanwhile (while :; do echo x >> f; done).

dirsync.py:

"""Times fsync of directories that have not changed: 162 in a row, 5 rounds."""
import os
import sys
import tempfile
import time

base = sys.argv[1]
with tempfile.TemporaryDirectory(dir=base) as d:
    dirs = [os.path.join(d, f"{i:02x}") for i in range(162)]
    for p in dirs:
        os.mkdir(p)
    fd = os.open(d, os.O_RDONLY)
    os.fsync(fd)
    os.close(fd)
    for rnd in range(5):
        times = []
        for p in dirs:
            fd = os.open(p, os.O_RDONLY)
            t = time.perf_counter()
            os.fsync(fd)
            times.append(time.perf_counter() - t)
            os.close(fd)
        times.sort()
        print(f"round {rnd}: total {sum(times)*1e3:.1f} ms, per fsync p50 {times[81]*1e6:.0f} us, "
              f"p90 {times[145]*1e6:.0f} us, max {times[-1]*1e6:.0f} us, "
              f"{sum(1 for x in times if x > 1e-3)} of 162 over 1 ms")

dirsync_par.py:

"""Times fsync of 162 unchanged directories, serially and from 32 threads (5 rounds each)."""
import os
import sys
import tempfile
import time
from concurrent.futures import ThreadPoolExecutor

base = sys.argv[1]


def sync(p):
    fd = os.open(p, os.O_RDONLY)
    os.fsync(fd)
    os.close(fd)


with tempfile.TemporaryDirectory(dir=base) as d, ThreadPoolExecutor(32) as pool:
    dirs = [os.path.join(d, f"{i:02x}") for i in range(162)]
    for p in dirs:
        os.mkdir(p)
    sync(d)
    for mode in ("serial", "32 threads"):
        totals = []
        for _ in range(5):
            t = time.perf_counter()
            if mode == "serial":
                for p in dirs:
                    sync(p)
            else:
                list(pool.map(sync, dirs))
            totals.append((time.perf_counter() - t) * 1e3)
        print(f"{mode}: 162 directory fsyncs take " + ", ".join(f"{x:.1f}" for x in totals) + " ms")

Lock fairness (Rust, std only). Built with rustc --edition 2024 -O flock_fairness.rs, and run as flock_fairness <dir> <poll|block> 32 300 <pause-us>.

//! Side experiment for decision (c), in Rust with std's file locks (what MetaLock uses).
//! usage: flock_fairness <dir> <poll|block> <processes> <per-process> <pause-us>
//! Each process runs <per-process> transactions: open <dir>/lock, take it (poll: try_lock every
//! 10 ms, a Busy counted after 10 s and the wait restarted; block: File::lock), append + sync_data
//! 120 B to journal.log and 8 B to journal.off, unlock (drop). Pause between transactions.
use std::fs::{File, OpenOptions, TryLockError};
use std::io::Write;
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};

fn append(path: &Path, bytes: &[u8]) {
    let mut f = OpenOptions::new().append(true).create(true).open(path).unwrap();
    f.write_all(bytes).unwrap();
    f.sync_data().unwrap();
}

fn child(dir: &Path, mode: &str, per: u64, pause: Duration) {
    while !dir.join("go").exists() {
        std::thread::sleep(Duration::from_millis(1));
    }
    let started = Instant::now();
    let mut waits = Vec::new();
    let mut busy = 0;
    for i in 0..per {
        if i > 0 && !pause.is_zero() {
            std::thread::sleep(pause);
        }
        let begun = Instant::now();
        let lock = OpenOptions::new().write(true).create(true).truncate(false).open(dir.join("lock")).unwrap();
        if mode == "poll" {
            let mut deadline = begun + Duration::from_secs(10);
            loop {
                match lock.try_lock() {
                    Ok(()) => break,
                    Err(TryLockError::WouldBlock) => {}
                    Err(TryLockError::Error(e)) => panic!("{e}"),
                }
                let now = Instant::now();
                if now >= deadline {
                    busy += 1;
                    deadline = now + Duration::from_secs(10);
                }
                std::thread::sleep(Duration::from_millis(10).min(deadline - now));
            }
        } else {
            lock.lock().unwrap();
        }
        waits.push(begun.elapsed().as_micros());
        append(&dir.join("journal.log"), &[0x78; 120]);
        append(&dir.join("journal.off"), &[0x79; 8]);
        let _ = lock.unlock();
        drop(lock);
    }
    let total = started.elapsed().as_micros();
    let w: Vec<String> = waits.iter().map(u128::to_string).collect();
    println!("{total} {busy} {}", w.join(","));
}

fn pct(sorted: &[u128], p: usize) -> u128 {
    let rank = (sorted.len() * p).div_ceil(100).max(1);
    sorted[rank - 1]
}

fn main() {
    let args: Vec<String> = std::env::args().collect();
    let dir = PathBuf::from(&args[1]);
    let (mode, procs, per, pause_us) = (&args[2], args[3].parse::<u64>().unwrap(), args[4].parse::<u64>().unwrap(), args[5].parse::<u64>().unwrap());
    if std::env::var_os("FLOCK_CHILD").is_some() {
        child(&dir, mode, per, Duration::from_micros(pause_us));
        return;
    }
    std::fs::create_dir_all(&dir).unwrap();
    let exe = std::env::current_exe().unwrap();
    let children: Vec<_> = (0..procs)
        .map(|_| Command::new(&exe).args(&args[1..]).env("FLOCK_CHILD", "1").stdout(Stdio::piped()).spawn().unwrap())
        .collect();
    std::thread::sleep(Duration::from_millis(500));
    let started = Instant::now();
    File::create(dir.join("go")).unwrap();
    let outs: Vec<String> = children.into_iter().map(|c| String::from_utf8(c.wait_with_output().unwrap().stdout).unwrap()).collect();
    let wall = started.elapsed().as_secs_f64();
    let (mut waits, mut totals, mut busy) = (Vec::new(), Vec::new(), 0);
    for out in &outs {
        let mut parts = out.trim().split(' ');
        totals.push(parts.next().unwrap().parse::<u128>().unwrap());
        busy += parts.next().unwrap().parse::<u64>().unwrap();
        waits.extend(parts.next().unwrap().split(',').map(|w| w.parse::<u128>().unwrap()));
    }
    waits.sort_unstable();
    let (min, max) = (*totals.iter().min().unwrap(), *totals.iter().max().unwrap());
    println!(
        "mode={mode} procs={procs} per={per} pause={pause_us}us: wall {wall:.1} s, {:.1} txn/s, Busy {busy}; lock wait p50 {:.2} ms p99 {:.1} ms max {:.1} ms; spread {:.2}x (min {:.1} s, max {:.1} s)",
        (procs * per) as f64 / wall,
        pct(&waits, 50) as f64 / 1e3, pct(&waits, 99) as f64 / 1e3, *waits.last().unwrap() as f64 / 1e3,
        max as f64 / min as f64, min as f64 / 1e6, max as f64 / 1e6
    );
}

M1 (CLI level) (2026-10-15)#

The CLI-level half of the M1 Task 7 experiments: 32 agents committing through the real swarmops binary, and the lock-free read path after the journal hardening. Same host as M0: WSL2 (kernel 5.15.153.1), ext4 (data=ordered) on the virtual disk /dev/sdd, 32 hardware threads; Rust 1.99.0, optimized builds (the contention test with --release, the bench with cargo bench). Every store lives in a fresh temporary directory under ~/.local/state/swarmops-bench. just perf re-measures in three steps, the store-level contention test, this CLI test at 32 × 1,000, then every store bench, and prints only their summary lines.

Every number here is provisional. The host was never quiet: three JetBrains remote-dev-server processes (about 8.5 cores), another project's test suite, and other SwarmOps worktrees: Task 7c's 32-process store-level contention runs overlapped the end of contention run 2 and both bench runs before the merge, and Task 8's multi-process tests overlapped the runs in just perf after it ("After 7c" below). The 1-minute load average was 15.7–23.1 when each of the four 32 × 1,000 contention runs started and 15–39 during them (32 processes blocked in fsync count towards it), and 12–61 during the other runs. Every measurement ran twice, except the read snapshot after the merge (once, in just perf); where two runs disagree by more than 20 %, both are shown. The controller re-runs just perf in a quiet window before the final review.

32 agents committing through the binary#

crates/swarmops-cli/tests/contention.rs, ignored. just perf runs it optimized (cargo nextest run --locked --release -p swarmops-cli --run-ignored ignored-only --test all -E 'test(/^contention::/)' --no-capture), just check-slow unoptimized, both at the spec's 32 × 1,000 (at 32 × 250 for a while; see below).

• The parent inits a store, commits a base changeset and creates main on it. Then 32 child processes start together behind a barrier (ready-<n> files, then go). Each runs swarmops commit 1,000 times in a row as agent contention/agent-<n>: one small file (agents/<n>.txt, about 20 bytes) per commit, on --parent its own previous changeset. So no child's commit depends on another's, and all they share is the store, and its meta/lock.
• Every commit is a process of its own: start, store open, two metadata snapshots (resolving --parent, checking the parents), the blob, two trees and the changeset written and synced (put_batch + flush), then the metadata write transaction under meta/lock, then the envelope.
• Per commit the child records the envelope's meta.elapsed_ms (all of that but process start, exit and printing) and the wall time from spawn to exit; the tables give nearest-rank percentiles over every commit's value. An E_BUSY (exit 4, retryable) is counted, and the same commit run again.
• A control comes first: one commit while the test itself holds meta/lock gets E_BUSY after 10,013 ms (10,014 in run 2), so the harness does see a Busy when there is one. The test also checks that every E_BUSY came after the 10 s lock timeout.
• After the children: info must count every changeset; 100 uncontended commits in a row time a commit alone on the final store; then verify --deep must find nothing, and log -n 5 (from main, moved to child 0's last changeset) must list child 0's last five commits.

The two runs disagree by more than 20 % on nearly everything but the spread, so both are shown. The host's I/O load differed: Task 7c's store-level contention test started during run 2's last few minutes, and run 2's uncontended commits afterwards, timed while it ran, cost twice run 1's (58 against 30 ms). A 32 × 100 calibration run before them (load 24.4, 22.1–27.7 during) gave run 1's throughput: 63.3 commits/s, p50 261 ms, p99 2.96 s, max 6.78 s, no E_BUSY, and an uncontended commit of 20 ms.

Throughput is the lock's. The lock never idles: each of the 32 processes needs it once per commit, and between two of its turns does only its own work, which takes 20–60 ms alone (an uncontended commit), against the 0.4–0.9 s it waits on average. So commits/s is one over the time a commit holds meta/lock: about 15.6 ms in run 1 and 27 ms in run 2 (15–20 ms in the 32 × 250 runs below). That is 2–3.6× the store-level transaction of the M1 store-level section (Task 7a: 7.5 ms p50 alone, 100–121 transactions/s with 32 writers). It is the same transaction, with the journal's two sync_datas (and every 32nd commit the index sync and the checkpoint), but here ext4 commits each of them along with 31 other processes' new object files and fan-out directory fsyncs.

The wait is the latency. A contended commit is mostly its wait for the lock: p50 235 and 374 ms, p99 2.8 and 5.3 s. With the lock always busy, the mean is fixed by the throughput: each process takes one turn per round of all 32, so a commit takes 32 ÷ commits/s on average, 0.50 s in run 1 and 0.87 s in run 2, whatever the lock's discipline. The 32 × 250 runs, which print the mean, bear this out: 624 and 461 ms against 32 ÷ 49.4 = 648 and 32 ÷ 65.6 = 488 ms. The discipline only decides who waits how long, and the 10 ms poll makes it a lottery: the median commit waits half the mean, while one commit in a hundred waits 3–5 s. Every child waited over 1 s at least once, and 24 and 32 of them over 5 s. A commit that loses the lottery for 10 s straight gets E_BUSY: 2 and 19 of the 32,000. Its retry then commits, so nothing is lost and verify --deep stays clean. But the plan's "0 failures, Busy only after the 10 s timeout" holds only in its second half, and the test fails, as the store-level one does.

The spread is not the indicator. 1.11× and 1.10×: over 1,000 commits each child's luck evens out, while single commits wait for seconds. That matches the store-level runs 2 ms apart, the pacing closest to a CLI agent's.

An uncontended commit costs 20–30 ms, nearly all of it fsyncs. 20–30 ms p50 when the disk is calmer (58 ms in run 2). Every commit makes at least four fsyncs one after another: its new objects' (in parallel), then its fan-out directories' (one after another before Task 7c, at once since), then the journal's two sync_datas under the lock. At M0's ~4 ms per fsync on this disk, that alone is about 15 ms. The snapshots are not the cost: a commit takes two, about 0.1 ms (below). The design spec (§15) states the target ("≤ 5 ms + 0.3 ms per changed file") excluding the syncs, and a full durable commit at 20–30 ms on this host; nothing here measured a commit without its syncs.

verify --deep checked 32,101 changesets and about 128,300 objects in 2.0–2.1 s, about 63,000 objects/s. log -n 5 took under 1 ms.

32 × 250 while the lock polled#

With the polling lock the test took 516.5 s and 886.1 s at 32 × 1,000, over 10 minutes in run 2, so its default was cut to 32 × 250 until the lock changed; just perf kept the spec's 32 × 1,000. These are its runs at 32 × 250 then.

No other SwarmOps test ran during either; run 1's load rose to 61 with other work on the host. The runs disagree by more than 20 %, so both are shown. Both passed, but with the polling lock that was luck: their slowest commits waited 9.8 s and 8.4 s of the 10 s timeout. At the 32 × 1,000 runs' rates (2 and 19 E_BUSY in 32,000 commits) 8,000 commits meet 0.5 to 4.75 on average, so just check-slow would have failed now and then until the lock changed. Under just check-slow then (unoptimized, beside every other slow test, load 18.9 at the start and 19.6–25.9 during) the test passed in 138.6 s.

Read snapshot after the journal hardening#

crates/swarmops-store/benches/read_snapshot.rs (cargo bench -p swarmops-store --bench read_snapshot). Since the hardening, a reader measures journal.off before journal.log, and Journal::tail_frames_from finds the checkpoint's frame by a binary search of journal.off, one 8-byte read_at per probe. MetaStore::read() calls it twice.

• One store per size, 10,000 and 100,000 changesets, each inserted by a write transaction of its own: a journal frame each, as commits make them. The stores are built outside the timed loops, through a filesystem that skips fsyncs, since a read does not depend on them; their files are in the page cache, as a hot store's are.
• Each store is timed twice through a handle on the real filesystem:
  • ckpt_end: after MetaStore::flush, the checkpoint at the end of the journal;
  • tail31: after 31 more transactions, 31 frames past the checkpoint, the most that the batched checkpoint leaves.
• The variants cycle through 1,024 scattered changesets:
  • read_changeset: MetaStore::read(), then changeset(id) on the snapshot;
  • read: MetaStore::read() alone;
  • tail_frames: one Journal::tail_frames_from(journal.ckpt);
  • off_search: that call's binary search of journal.off alone (copied from Journal, whose search is private): what caching the checkpoint's frame would save.
• Every timed call is recorded, so these are nearest-rank percentiles over 57,000 to 999,000 calls per variant, warm-up included. Criterion's medians run 0–16 % above these p50s: they are medians of per-sample means, which the tail pulls up.

Run 1 started at load 11.7 (12.4–16.3 during), with Task 7c's 32-process store-level contention test running throughout; run 2 at load 22.6 (17.0–24.6), Task 7c's processes starting up again during it. p50 / p99:

Each cell is run 1, which run 2 matches within 20 %, except the bold cell, where they differ by 22 % (p50) and 23 % (p99): run 1; run 2.

Task 2b's figure is a find p99 of 16 µs (tests/index.rs, index_on_wide_history_is_fast: find alone on an open 200,000-record index, unoptimized). That test, re-run on this code at load 17–29 with Task 6's tests running, gives p50 9.1 µs and p99 18.3 µs (2b: 10.3 and 16.1 µs), so the lookups did not change. The snapshot is the larger cost: read_changeset's p99 of 84–128 µs is 5–8× that find p99, and read alone is most of it. The lookup on the snapshot adds 6–12 µs (p50).

Should the journal.off lookup be cached? Not now.

• The search costs 3.9–5.3 µs (p50; 14 probes at 10,000 frames, 17 at 100,000). read() runs it twice, 8–11 µs, about a fifth of the snapshot.
• A snapshot with its lookup is 46–70 µs p50 and under 130 µs p99, about 8× under the spec's read target (< 1 ms) at 100,000 changesets. A CLI command takes one or two: two for a commit, about 0.1 ms of a 20–60 ms commit, and less than its own process start.
• The rest of the snapshot is opening and reading files: journal.ckpt, the index's nine files and the refs, about 25 µs. The 31 frames past the checkpoint add 14–15 µs more: read and parsed twice, then overlaid. Each costs more than the search.
• Caching pays only in a long-lived process that takes many snapshots, such as swarmops mcp or a network server after v1 (umbrella spec §17). There, a per-handle cache of the checkpoint's frame (its seq, checked with one 8-byte read that its entry still holds the checkpoint) would turn both searches into one read each.

The lock: decision#

The rule: recommend a blocking lock with a timeout thread if the fairness spread exceeds 2× or the p99 lock wait exceeds 1 s. Against the polling lock's runs above:

• Spread: not exceeded. 1.11× and 1.10×.
• p99 lock wait: exceeded. About 2.8 s and 5.3 s (estimated from meta.elapsed_ms, whose p99 is 2.87 and 5.35 s; the commit's own work is 20–60 ms of it).
• E_BUSY occurs. 2 and 19 of 32,000 commits, every one after the 10 s timeout. The CLI contention test fails on it, as the store-level test does.

Recommended: a blocking lock (File::lock on a thread of its own, the caller waiting for it with the 10 s timeout and returning Busy when that passes. In a CLI process the blocked thread dies with the process; in a long-lived one, a thread that gets the lock after its caller gave up must release it at once).

• A blocking flock wakes a waiter the moment the lock is released, and so removes the 10 ms window in which every poller sleeps through the release. The store-level side experiment shows the result: no Busy, and a p99 wait of 0.3–0.8 s with writers 2 ms apart.
• CLI commits are further apart than 2 ms: each process spends its 20–60 ms of own work between two lock requests.
• What it will not change is throughput: the lock stays busy, so the mean wait stays about 0.5 s at 32 back-to-back committers on this disk. A fair lock spreads that wait over every commit, as a queue would, instead of piling it on a few. More commits per second need a shorter transaction under the lock: 7.5 ms alone, 15–27 ms here.

Task 7c applied it, with the parallel Index::sync and flush; "After 7c" below has the CLI-level result.

After 7c: the kernel-queue lock#

Task 7c's changes (the store-level section's "M1 store fixes (7c)") are merged under the CLI: MetaLock waits in the kernel's flock queue, a helper thread blocked in flock while the caller waits up to the 10 s timeout; Index::sync makes its five syncs at once; and flush its directory fsyncs at once. The same test, 32 × 1,000, --release, twice:

• run 1 on its own, with no other SwarmOps test running;
• run 2 inside just perf, right after the store-level contention test, with Task 8's multi-process tests running for about 2 of its 8.4 minutes (33 processes, from its start) and in four later 15 s samples (2–46 processes).

7c's raw fdatasync probe (the median of 40 appends) took 3.8 ms at the start and end of run 1 and 3.7 ms at the start of just perf: M0's speed, unlike the 6.6–15 ms of the 7c runs. Percentiles are nearest-rank over every commit. "In lock passes" divides a wait by the run's mean time per commit (wall / 32,000), the time the lock takes to pass once. The polling runs did not print the mean; 32 ÷ commits/s stands in for it. The polling maxima leave out the E_BUSY waits, each past 10 s.

The two kernel-queue runs agree within 20 % on throughput, the median and the spread, and disagree on the tail, so both are shown: run 2's p99 and longest wait are about twice and three times run 1's, with Task 8's tests on the disk.

• E_BUSY is gone, from 2 and 19 in 32,000 commits to none.
• The tail is cut. p99 0.66 and 1.22 s, from 2.87 and 5.35 s. In lock passes, p99 45 and 76 from 181 and 195, and the longest wait 196 and 503 passes from 367–638 (and the E_BUSY waits past 10 s).
• The median rose and the mean did not move. The mean is fixed by the throughput (455 and 498 ms against 32 ÷ 69.7 = 459 and 32 ÷ 63.7 = 502 ms). The lock now rotates: a commit waits about one round of the other 31 writers (27–29 passes), where the poll lottery let most commits through in a fraction of a round and made the unlucky ones wait for seconds. So p50 is 441–450 ms, from 265–432 ms.
• Every child finished within 1 % of the others (spread 1.01×, from 1.10–1.11×).
• Throughput is still the lock's. 63.7–69.7 commits/s, 14.3–15.7 ms per lock pass, on a disk at M0's fsync speed: about four raw fdatasyncs for a transaction that makes two. What the rest of a pass costs under 32 writers is not measured here. Whether the commits' own object writes set that pace is for the controller's quiet-window just perf to show: the store-level contention test in this just perf (70.9 transactions/s) overlapped Task 8's 33–57 processes for about 6 of its 7.5 minutes, so it does not compare.
• The tail is bounded, not gone, as 7c found at store level. Run 1's longest wait, alone on the host, was 196 passes (2.8 s); run 2's about 500 (7.9 s at 15.7 ms a pass), with Task 8's tests on the disk for part of it. On a disk a third slower, 500 passes reach the 10 s timeout, which is why both contention tests now run one after the other (.config/nextest.toml). The tail is to be re-measured in the controller's quiet-window just perf. An order fair to the commit needs a FIFO hand-off on top of flock (7c).
• An uncontended commit took 21 ms p50 in both runs (p99 30 and 54 ms), about 5.5 raw fdatasyncs. The four rounds in a row (the objects, their directories, now at once, then journal.log and journal.off) make about 15 ms of it.

The read snapshot after the merge. The merged reader also checks the frame listed just before where it reads. just perf's read_snapshot run on it (load 15–28 during the benches, the disk at M0's speed) gives the same numbers within 3 % with the checkpoint at the journal's end (100k: read_changeset p50 54 µs, p99 97 µs). With 31 frames past the checkpoint they are 5–6 % slower (100k: 73 and 126 µs), as tail_frames takes 12.7–13.1 µs p50 against 10.5–11.7 µs before. off_search is unchanged (4.0–5.2 µs), and so is the answer on caching it.

Size. The test took 473.2 s and 517.4 s at 32 × 1,000, under 10 minutes both times. So its default is the spec's 32 × 1,000 again, and just check-slow runs that. Under just check-slow, unoptimized, it first passed in 513.5 s beside the store-level contention test (499.1 s; load 22.6–36.2 during). .config/nextest.toml now runs the two one after the other: the suite then took 725.1 s, this test 449.3 s and the store-level one 274.3 s after it (load 16.9 at the start and 12.0–37.1 during, raw fdatasync 3.8 ms at both ends).

M1 — just perf after all merges (2026-10-16, controller run)#

Head a4da77d (every M1 task merged), just perf (store-level contention → CLI contention → store benches, one after another), start load 16.4 → end load 14.6 (one concurrent debug build of another worktree during the benches; no other 32-process test). Raw log: the SDD workspace (just-perf-after-merges.log). Nearest-rank percentiles.

The lock pass, store against CLI. At store level the lock passes every 8.27 ms (264.8 s / 32,000), about two raw fdatasyncs, and an uncontended transaction takes 7.24 ms. The 3.8 ms probe is itself a 120-byte append and its fdatasync, so the inode size update that each journal append makes is already in it: growing the files is not the difference. Through the binary the lock passes every 13.75 ms (440.0 s / 32,000). The roughly 5.5 ms more appears only when the commits' own object writes run, outside the lock, on the same filesystem. Task 7b's "about 14 ms per pass at both levels" came from a store-level run that overlapped Task 8's processes, and is withdrawn. Whether the lock holder's two fdatasyncs wait behind those object syncs on ext4's shared journal is not measured yet: the M1.6 measurement of the lock holder under object syncs will record it in a section of its own.

Decisions confirmed on this run: the kernel-queue lock keeps Busy at 0 with every longest wait under 1.2 s, so lock_timeout stays at 10 s (FIFO hand-off remains an M6 experiment); parallel Index::sync and parallel flush directory fsyncs hold their 7c gains; 32-thread put_batch scales 13.8× (15.5× with a busy writer), so a --durability fast flag is not justified by this data. The CLI harness now writes each change list to a file (7b fix round); the 21 ms uncontended commit matches the earlier stdin-based harness within noise.

M1.5 — native store vs embedded databases (2026-10-17)#

bench/compare/ measures the native metadata store (swarmops_store::MetaStore, through its public API) against SQLite, redb, fjall and sled on SwarmOps' own workloads (M1.5 Task 8): just bench-compare, or just bench-compare --quick for a smoke run of every workload at small sizes (88.6 s of wall time here, 84.5 s inside the harness after a 4 s rebuild, at load 10.4–11.5 and a raw fdatasync of 3.0–5.1 ms, on b4d2208; the brief asks for under 2 minutes). It is a cargo workspace of its own, like fuzz/: publish = false, outside the root workspace, never built by just check. SwarmOps depends on none of these databases; they exist only in that harness, to measure the store the project chose to build instead. Its README gives the method in full.

The run. One hold of the host's lock for heavy runs, 2026-10-17 02:49–06:07 UTC (11:49–15:07 KST), 11,865.8 s, no errors, on tree dcf4fa7, at the plan's sizes: W1 10,000 timed transactions after 200 untimed, W2 32 writers × 1,000 transactions, W3 and W5 on 100k and 1M changesets, W4 on 1M, W6 on 100k (20,000 lookups of each kind, 5 opens and 3 kills per run, 10 s of W6 writer), the sync count over 64 and 1,088 transactions, every measurement 3 times (the raw record, bench-compare-1792217222.json, is kept under bench/compare/target/results/ on the benchmark host, git-ignored). The branch has moved to b4d2208 since (Task 6, test code): its store source differs from dcf4fa7's only in doc comments and in gating the CrashAtFs test double behind the test-hooks feature, which the harness does not enable, so the code measured is the code of b4d2208. The final fix wave then moved it past that: its store source differs from b4d2208's in the read loop's comparison (store.rs: the frames read again are compared whole instead of counted, fa2d0e3), in the error of a failed Compacted marker append (store.rs, 9ede780), in two stderr lines on the journal's abort paths (journal.rs, 6624cf7; the wave's third such line is in swarmops-repo, which the harness does not build), and in the reads of the store's files: every open of a store file is preceded by a stat that refuses anything but a regular file (fs_ops.rs, journal.rs, column.rs, sorted.rs, ed0eff5), journal.ckpt is read through a 9-byte cap and each ref file read is preceded by a stat of its length (journal.rs, refs.rs, 0f90b51); the rest is doc comments (index.rs, order_check.rs). So a snapshot pays one stat per journal file and sorted view it opens and one per ref file it reads, no sync and no allocation, and nothing was re-measured. The same host as M0 (WSL2, kernel 5.15, ext4 on /dev/sdd, 32 threads), under the host's lock for heavy runs: no other benchmark beside it, but the lock does not cover builds and tests, and the load shows that some ran. Load 11.57 and a raw fdatasync of 3.77 ms at the start (M0 measured 4.1 ms on a quiet disk). Outside W2, the 1-minute load was 7.7–15.2 at the start of every measurement and 7.7–25.6 during them, and the raw fdatasync probe 3.7–5.6 ms in every one of them but the first native W5 build (7.1–7.2 ms). W2 is the exception, in two ways. Its probe read 6.7–9.7 ms at the start and end of every run but its first's start (4.1 ms) and four readings under a load event (17–26 ms): the disk stayed there for the whole 2.8-hour W2 phase. And two events the harness's own work does not explain (runs 2 and 3 of the same workloads stayed at load ≤ 13) raised the load, to 89 during the native threads run 1 and to 85 during redb's threads run 1 (76 at the start of fjall's). The per-run rows of the raw record show both; a median over three runs is one of the other two runs' values, and the [min–max] beside it shows the outlier.

Engines, durability and method#

Every engine runs the same transaction on the same seeded data, in the layout a competent user of it would choose, with its defaults apart from durability. W1's transaction (which W2, W4 and W6 reuse) checks that the changeset is new, looks up its parent and computes its generation, stores it under its 32-byte id numbered in insertion order, indexes it by its 32-byte delta hash, moves main to it by compare-and-swap and records the move, and commits durably. The native store records the move in its journal frame; the others write a reflog row, as the M0 SQLite schema did. SQLite's schema mirrors the native index (rows in seq order, a unique index by id, an index by delta hash); redb, fjall and sled keep changesets, by_delta_hash, refs, reflog and meta (the counters).

Every version was released at least 7 days before the harness was written, and so was every crate in its Cargo.lock, except zstd 0.14.2 (5.7 days), zstd-safe 8.1.0 and zstd-sys 2.1.1 (6.6 days), which swarmops-store's own zstd brings in. sled builds cleanly on Rust 1.99, so it is measured, as a reference.

• Equalized durability. Every engine but two makes each commit durable before it returns: the native store, SQLite at synchronous=FULL, redb at Durability::Immediate and fjall at PersistMode::SyncAll. SQLite at synchronous=NORMAL syncs only at checkpoints, so a commit survives a process crash but not a power loss. sled's flush syncs with sync_file_range, which writes no metadata and does not flush the disk's cache. Those two are reference lines, never contenders.
• Multi-process access. Only the native store and SQLite let several processes open one store. redb, fjall and sled lock the database to one process, so W2 runs them on threads, and runs the native store and SQLite on threads too, which measures what the asymmetry is worth.
• fjall's settled state. fjall writes a memtable to a table file only once it holds 64 MiB, and drops a journal only when every keyspace has flushed past it. The small keyspaces (meta, refs, reflog) never fill a memtable, so a bulk-loaded store keeps every journal until 512 MiB of them force a flush (216 of the 1M store's 385 MiB). The fjall rows settle as the other engines do, by moving the log into the main structure as SQLite's checkpoint does: the journal is synced, then every keyspace's memtable is flushed into table files (rotate_memtable_and_wait, public but doc-hidden), and fjall deletes the journals no keyspace needs. The active journal stays: fjall rotates it only once it passes 64 MB at a flush, and every open replays it whole, flushed or not. So the flush shrinks the 1M store (W5); at open it saves only the first open after the load, which on the unflushed store replays all 216 MiB of journals, flushes what the load left in memtables and deletes the sealed journals (2.9 s in the second provisional hold, where W4 ran before any W3, against 0.55 s for the flushed store); every later open replays the same 43 MiB active journal (W4). A 100k store keeps its whole journal beside the new tables. The fjall-unflushed rows (W3–W5 only) show the store as the load left it in W5; W3 and W4 see it after fjall's own first open settled it.
• Method. Same host and filesystem as M0 (ext4 data=ordered on WSL2's virtual disk), stores in fresh directories under ~/.local/state/swarmops-bench, warm page cache. Seeded, deterministic data with real BLAKE3 ids, in chains. Warm-up before timing (W1 200 transactions; W3 one untimed pass of every lookup kind; W6 one of the reader's). Each measurement runs 3 times, the sync count and W5's builds of each store included, the engines in a rotated order; every cell is the median over the runs with [min–max]. None of the open, first-lookup, first-write or commit timers includes a close (SQLite's open hands its connection to the first session: closing a WAL store's last connection checkpoints the WAL and deletes it); W4's process ms (a process from spawn to exit) and W5's load s do span one. Beside every row: the 1-minute load at the start of each run (and its range, sampled every 2 s) and a raw fdatasync probe (the median of 40 appends of 120 B, each sync_data'd) at the start and end of each run. No network.
• This run's course. The sync count first (0–191 s into the run), then the breakdown (to 252 s), W1 (to 1,028 s), W2 (to 10,991 s: 2 h 46 min, most of the run), W5's three builds of every store (to 11,455 s), W3 (to 11,580 s), W4 (to 11,668 s) and W6 (to 11,866 s). The raw fdatasync probe read 4.1 ms at the start of W2's first run (native, processes) and 7.0 ms at its end, then 6.7–9.7 ms at the start and end of every W2 run (the four readings under the second load event apart) until the first native W5 build (7.1–7.2 ms), after which it was back to 3.9 ms; the records do not say why. So W2 ran on a disk about twice as slow as W1's: its engines compare with one another, not with W1's rates. Two load events, which the harness's own work does not explain, fell into W2's first round of threads: one reached 89 during the native threads run 1 (its 59 Busy, its 28.8 s longest wait and its 24.5 transactions/s are that run's), the other 36 during SQLite's threads run 1, then 85 during redb's and 76 at the start of fjall's (probes of 17–26 ms; their run 1 rows: redb 92.7 transactions/s against 228–230 in runs 2–3, fjall 72.9 against 156–160). The medians of W2's rates are the two unaffected runs'.
• What changed since the provisional run (2026-10-16). That run, on the Task 8 tree in two holds of the lock on a busy host, had W1 at 1,000 transactions and W2 at 32 × 50, at a raw fdatasync of 3.7–18 ms. Its headlines, for the record: W1 29.9 commits/s against SQLite's 135 and redb's 218 (4.5–7.3×, at a 7.6–8.3 ms probe); W2 27–59 transactions/s; W3 3.3–12× SQLite's p50 and 8.2–57× redb's; W4 open + lookup 0.108 ms against SQLite's 0.377, recovery 17.2 ms against 5.14 (3.3×), 301 against 6.63 ms at worst (45×); W6 90.8 µs p50 beside the writer; W4's slow kills were there too (the second hold's run 2: a median of 192 ms and a worst of 301; the first hold's: 261, 310 and 190 ms). The harness changed in four ways between those holds and this run: SQLite's open hands its connection to the first session (the first hold's SQLite rows, open + lookup 0.679 ms and recovery 23.7 ms, came from an open that made a connection and closed it, which checkpoints a WAL store, with two syncs, and deletes its WAL: a tie that was the harness's; fixed before the second hold); W5's builds and the sync count run three times, as every measurement does (also before the second hold); each record names the tree it measured (host.git in the record, the report's tree line), where the provisional records named the base commit of an uncommitted harness; and W2 reaps its writers on an error, and fjall's settle fails at its timeout instead of returning. The store's source changed too: Task 8's tree predates the Task 2 and 3 store fixes and the code minors batch, and the native store now makes 165 system calls per commit where it made 157; its sync counts are the same.

Sync calls per commit: durability, checked#

strace and perf are not installed on this host, so the harness counts system calls with a ptrace tracer of its own (bench/compare/src/trace.rs: about 230 lines of safe Rust over nix's ptrace calls, x86_64 Linux only). A child process creates a store, commits 32 untraced W1 transactions and waits on its stdin. The tracer seizes every thread, follows the threads they create, and counts every system call while the child makes n more commits, closes the store and exits, naming each sync call's file through /proc/<pid>/fd. It runs twice, with n = 64 and n = 1,088; the difference over 1,024 is the cost of one commit, with the batched work averaged in (the native index sync every 32 frames and a compaction every 1,024 records, SQLite's checkpoint every 1,000 WAL pages) and the open, the warm-up and the close cancelled out. The pair runs three times, once per run, the engines in a rotated order. Per commit, the median [min–max] of the three counts (the run's first 191 s, load 11.0–12.4 at the start of each count). Every count but sled's system calls came out the same three times:

• Every durable contender makes one sync per commit: SQLite an fsync of its WAL (1.02 with its checkpoints averaged in; this SQLite build has no fdatasync), redb one fdatasync, fjall one fsync of its journal.
• The native store makes 4.23, as storage-core §3 specifies ("2 journal syncs per transaction, + 2 per moved ref"; every W1 transaction moves main): the frame's fdatasync (the commit point), the journal.off entry's, the ref file's temp-file fsync and the fsync of refs/, plus the batched index sync (5 files every 32 frames: 0.16), the checkpoint's replacement (2 every 32 frames: 0.06) and the compaction's syncs (every 1,024 records, under the table's 0.01 cut-off).
• Not durable as the others are: sled's 2.26 sync_file_range calls per commit write neither the file's metadata nor the disk's cache, and SQLite at NORMAL syncs only at its checkpoints (0.018 per commit).
• The native store writes the fewest bytes per commit (470, against SQLite's 24,031 in whole WAL pages and redb's 49,752 in copy-on-write pages) and makes the most system calls (165: the lock file and the index's nine files are opened in every transaction, and the ref file gets a temp file of its own). Neither matters beside its syncs (below).

Where a native commit's time goes: the journal syncs#

breakdown runs W1's transaction on a native store whose filesystem is RealFs with every call timed (through the store's public MetaStore::create_with_fs): 1,024 commits per run after a 200-commit warm-up, so that one compaction falls among them, 3 runs. RealFs::append_durable (the journal's append + fdatasync) and atomic_replace of a ref file (temp file, fsync, rename, directory fsync) are done step by step with RealFs's calls in RealFs's order, so that each sync is timed on its own; a kind of call counts for the time its calls cover, overlaps once (the index sync's five sync_datas run at once). Means per commit, so the parts add up. Load 11.9/12.1/11.5 at the start of the runs (11.5–12.6 during them), raw fdatasync 3.8–4.7 ms.

• 94 % of a native commit is its four syncs, about a quarter each: journal.log's fdatasync 23.7 % [23.5–23.7], journal.off's 23.4 %, the ref temp file's fsync 24.2 % [23.9–24.2] and the fsync of refs/ 22.6 % [22.4–23.0] (93.8 % together, 93.7–93.9 in each run), each about one raw fdatasync (3.70–3.87 ms against a probe of 3.8–4.7 ms in these runs). The batched work takes 2.8 % (the checkpoint 1.55 %, the index sync 1.10 %, the compaction 0.17 %), and everything else (the lock, opening the index, the lookups, the encoding, the appends, the rename) 3.3 %.
• The second journal sync is a quarter of a commit, not a half. An independent code review suggested that the fdatasync of journal.off may be droppable. Without it, a W1 commit would take about 12.3 ms instead of 16.1 ms: 1.3× the commits/s. The ref file's replacement costs as much as both journal syncs together (its two syncs 46.7 %, the journal's two 47.1 %), although storage-core §2 rule 3 calls the ref files, like the index, a projection of the journal that replay restores from the frames past the checkpoint, and the index's syncs are batched every 32 frames. A commit with one sync, where the other engines are, needs both gone. Whether either can go is a durability question for the spec (what a reader and the tail check may trust after a power loss), which this measurement does not settle; it prices the options.

W1: one writer, durable commits#

W1's transaction back to back, 10,000 timed commits after 200 untimed. Bytes per transaction: the closed store's allocated size over the 10,200 commits.

Verdict: the native store loses, with 6.6–7.6× fewer commits/s than the durable engines (59.5 against fjall's 394, SQLite's 417 and redb's 450). Its p50 of 15.5 ms is four raw fdatasyncs (3.87 ms each; the probe read 3.9–5.1 ms): the frame per transaction with two journal fdatasyncs, plus the ref file's atomic replacement with two more, where every other durable engine makes one sync per commit. And that one sync costs the others about half of a raw append + fdatasync (SQLite's p50 1.82 ms, redb's 1.88, fjall's 2.34, against the probe's 3.8–4.7), where each native sync costs about one (3.70–3.87 ms in the breakdown above); most likely because every native sync follows an append or a newly created file, whose new length or inode ext4 must log through its own journal before the sync returns, while redb rewrites pages in place inside a file that does not grow at every commit, SQLite, after each checkpoint, rewrites its WAL from the start, and fjall writes into a journal it pre-sized to 64 MiB. The native tail is the longest too: p99 34.8 ms [31.9–70.0] and a worst commit of 106 ms [84.6–472] (run 3's 70.0 and 472 fell where the load rose to 25.6), against p99s of 4.90–6.24 ms and worst commits of 14.0–38.4 ms at the median for the three durable engines (fjall's run 3: 96.0, above the native store's best run, 84.6). sled (1,791/s) and SQLite at NORMAL (8,920/s) return before the commit is on the disk. On disk, the native store takes 397 bytes per transaction, between redb (382) and fjall (502); SQLite takes 290.

W2: 32 writers#

32 writers × 1,000 transactions each, every writer moving main with a compare-and-swap against its value inside the transaction. Processes for the native store and SQLite, threads for the others; the native store and SQLite on threads as well. Latency includes Busy retries (a writer that gets Busy after the 10 s timeout tries again); spread is the slowest writer's total time over the fastest's. No writer failed, and no engine reran a transaction. The raw fdatasync probe read 6.7–9.7 ms through W2 (above), against 3.8–5.1 during W1.

Verdict: the native store loses on throughput, 2.8–2.9× to SQLite and 4.9–7.1× to fjall and redb, and wins on fairness.

• Throughput. Every durable engine serializes its writers, and none batches several commits into one sync, so W2's rate is set by each engine's per-commit sync cost on the disk it had (0.54 of the native store's W1 rate, 0.51 of redb's, 0.40 of fjall's, 0.22 of SQLite's): the native store at 32.0–32.1 transactions/s (four syncs at 7.8 ms, where W1's 59.5/s were four at about 4 ms), against SQLite's 90.5–92.6 (2.8–2.9×), fjall's 156 (4.9×) and redb's 228 (7.1×). The four run-1 rows under the load events are lower (native threads 24.5, SQLite threads 73.2, redb 92.7, fjall 72.9); the medians are the other two runs'.
• Fairness. The native store's writers wait in the kernel's flock queue (MetaLock, M1 Task 7c): spread 1.03–1.06 in all six runs, p50 1,043–1,111 ms over the six runs (medians 1,089 and 1,108; about one rotation of the 32 writers: 1.0 s at 32 commits/s), and the longest wait 1.57–1.75 s in four of the six runs, 3.34 s in the processes run 1 (its load rose to 29). The one run with Busy is the threads run 1 under the load of 89: 59 of them, after 10 s in the queue, and a longest wait of 28.8 s; the other five runs had none. SQLite's busy handler polls, with sleeps that grow to 100 ms: 71–139 SQLITE_BUSY per run after the 10 s timeout, on processes and threads alike, p99 5.1–7.0 s, longest waits 27–52 s, and a spread of 1.6–3.0 in five runs and 54 in one (the processes run 3, where a writer finished 54 times sooner than the slowest). redb's and fjall's in-process writer locks (a condition variable in redb, a mutex in fjall), neither first-come-first-served: spread 1.2–4.1, longest waits 49–118 s, p99 0.89 s [0.018–8.1] and 3.5 s [0.99–11.7]. sled's transactions are short and fair (spread 1.02, p50 17.2 ms, the longest 40 ms), at a durability the others do not have.
• The asymmetry. The native store runs the same on threads as on processes (32.0 against 32.1 transactions/s, spread 1.04 either way; every acquire flocks a file description of its own, so threads queue as processes do), and so does SQLite (90.5 against 92.6). redb, fjall and sled would need a server process in front of them to take SwarmOps' 32 agent processes.

W3: lookups, warm#

Bulk-loaded stores of 100,000 and 1,000,000 changesets (one chain, in transactions of 50,000, then settled and closed), opened in the measuring process; 20,000 lookups of each kind, after one untimed pass of every kind. On one open snapshot: by id (get), by the first 6 bytes of the id (prefix), by delta hash through the secondary index (delta), and an id the store does not hold (miss, what every insert asks first); and by id in a fresh read transaction each (fresh, what a CLI command pays; sled has no read snapshots, so its lookup reads the latest state and pays no snapshot cost). Every answer is checked. µs:

The native store's p50 over each engine's, kind by kind: the median over the three runs of each, as the harness prints it under each W3 table so that a verdict starts from every kind (report::ratios; the two tables of the report joined here, their cells as printed). Above 1, the native store is slower:

Verdict: the native store loses every lookup kind to SQLite and to redb, by 3.7–11× against SQLite and 7.4–73× against redb, the fresh snapshot by the most; against fjall at 1M it wins the prefix, delta-hash and miss lookups, ties the get and loses the fresh snapshot.

• Against SQLite. A get or a delta-hash lookup is 3.7–3.8× slower, a miss 3.8–4.7×, a prefix lookup 7.3–8.7× (14.8 µs p50 against 2.03 at 1M) and a fresh snapshot and lookup 9.1–11× (56.3 µs against 6.22 at 1M).
• Against redb. 18–73× at 100k (a get 10.0 µs against 0.390, a fresh snapshot 51.4 against 0.702) and 7.4–29× at 1M (a get 13.5 against 1.40, a fresh snapshot 56.3 against 1.97).
• Against fjall. At 1M the native store is the faster on the prefix, delta-hash and miss lookups (0.14–0.37× fjall's p50: fjall's prefix lookup takes 46.5 µs, its miss 60.8), about even on the get (13.5 against 12.7 µs, 1.07×) and the slower on the fresh snapshot (4.0×). At 100k it is the slower on every kind but the miss, where fjall's 14.6 µs is the slowest of all (the native store's 6.25). So at 100k the native store is the slowest engine at every kind but the miss; at 1M it is the slowest at the get and the fresh snapshot, and fjall at the other three.
• Why. A native lookup binary-searches a sorted view's key file with one pread per probe (about 20 at 1M), then reads the record's columns with four more. It keeps no cache of its own, nor maps the files (unsafe is forbidden), where redb and sled answer from a 1 GB in-process cache and SQLite from its page cache. And a native snapshot opens the index's nine files and reads the checkpoint, the journal tail and the ref files each time, because readers take no lock and share nothing with writers (the lock-free reader).
• What the columnar index buys. A prefix or delta-hash lookup costs about what a point lookup does (each kind has its sorted view), and a miss less. Going from 100k to 1M adds 31–41 % to a get, a prefix lookup, a delta-hash lookup and a miss (about three more probes) and 10 % to a fresh snapshot; SQLite's lookups grow 32–63 % over the same step, redb's 2.5–3.6×, and fjall's 4.2–23× (get 23×, fresh 15×, prefix 11×, delta 9.1×, miss 4.2×), past the native store's on the prefix, delta-hash and miss lookups: its 32 MiB block cache holds a sixth of its 1M tables (192 MiB).
• Every native lookup at 1M stays at or under 30 µs p99 (get 27.0, prefix 28.8 [27.4–30.0], delta 26.7), 0.11 ms with a fresh snapshot, nine times inside the spec's 1 ms read target. The gap matters to a long-lived process that takes many snapshots, not to a CLI command that takes one or two.

W4: opening a 1M store, and recovery after a kill#

Open: a fresh process opens the 1M store (warm: the files are in the page cache, and W3 has opened every store three times before W4, so these are re-opens; the first open after a load is not timed here) and looks one changeset up, 5 times per run (the JSON's open_max_ms holds the worst of the 5). Recovery: a writer process commits W1's transaction in a loop on the 1M store and is killed with SIGKILL 0.1–1 s after its first acknowledged commit (a seeded draw), most likely inside a commit; a fresh process then opens the store and commits one transaction (where most engines repair what the kill left), 3 kills per run. A killed process loses nothing the kernel holds, so this is recovery from a process crash, not from a power loss. None of the open, first-lookup or first-write timers includes a close: SQLite's open hands its connection to the first session, and the lookup's session is closed after the timing (process ms, the opening process from spawn to exit, spans its close). The last acknowledged commit was present after every kill, on every engine (each engine's ordered log implies the earlier ones). open + lookup ms is the sum of a run's median open and median first lookup; open + write ms is the median of the per-kill sums. ms:

Verdict: the native store opens fastest by far; it loses recovery to SQLite, 4.3× slower at the median and 40× at worst, and recovers 13–22× faster than redb and fjall at the median and 1.5–2.3× at worst (235 against 342 and 529 ms).

• Open. 0.091 ms to open the 1M store and answer a first lookup, against SQLite's 0.368 ms (4.0×), redb's 1.79 ms and fjall's 468 ms (sled, a reference line: 147 ms). The native open reads nothing (0.022 ms); the first snapshot (0.071 ms) reads the checkpoint, the index's files and at most 31 frames past the checkpoint, because the index is synced and the checkpoint moved every 32 frames (the batched index sync). SQLite's open makes its first connection, which creates the WAL and its index again (a clean close deleted them), 0.334 ms; its first lookup and redb's are the fastest of all (0.034 and 0.035 ms). fjall replays its active journal (43 MiB here) into its memtables at every open, flushed or not; by W4 the unflushed store holds the same active journal, its first open (in W3) having replayed and flushed its 216 MiB of journals and deleted the sealed ones, so the two open alike (463 ms against 468). That first open is not in the table: in the second provisional hold, which ran W4 with no W3 before it, it took 2.9 s (fjall-unflushed, run 1's worst open), and sled's first open 1.2 s against 162 ms at the median.
• Recovery. 22.4 ms [16.2–231] for the next process's open and first write, against SQLite's 5.21 [5.03–5.50] (4.3×), redb's 287 (without quick-repair, its default, an open after a crash walks the database to rebuild the allocator state) and fjall's 499 (its journal replay): 13× and 22× faster than redb and fjall (sled, a reference line, 172). SQLite's open after a kill rebuilds the WAL's index from the WAL (0.879 ms: a read, no sync), and its first write is one ordinary commit appended to that WAL: an fsync of the WAL, which also fsyncs the directory, as every connection's first sync does (4.31 ms, about one raw fdatasync). The native open reads nothing (0.022 ms), and its first write is an ordinary commit too: its four syncs (16.2 ms in run 3, at a 3.8–3.9 ms probe), with the tail repair and the replay of at most 31 frames costing next to nothing when the kill left no torn frame.
• The worst case. 235 ms for the native store against SQLite's 5.86 ms (40×): ahead of redb's (342) and fjall's (529); sled, a reference line, took 188. Run 1's median kill took 231 ms and its worst 235; run 2's worst 198 ms against a median of 22.3; run 3's worst 20.9: at least three of the nine kills. Most likely kills that left a torn or unlisted frame: before it cuts or lists one, the next writer checks every listed frame (storage-core §2 rule 3; check_listed_chain), and at 1M that reads the whole 101 MiB journal. The check grows with the journal, in bytes and in listed frames (two reads, a CRC and a decode each), and only a crash's repair pays for it. The seeded store was bulk-loaded in 20 transactions of 50,000 changesets, each followed by its Compacted frame, so its journal holds 40 frames (plus the killed writers' own commits); a store built one commit at a time holds a frame per commit, about a million at 1M commits, and its repair would take longer than this worst case (not measured; docs/known-limitations.md, "Recovery after a crash walks all of journal.off").

W5: size on disk#

The bulk-loaded stores, settled (native: flush; SQLite: wal_checkpoint(TRUNCATE); redb: nothing to do; fjall: memtables flushed, fjall-unflushed: the journal synced only; sled: flush) and closed; allocated bytes, as du counts them. Each store was built three times, a run each (the first build is the store W3, W4 and W6 read). The native store, SQLite, redb and fjall came out the same to the byte in all three builds (the native store 29,839,360 bytes at 100k and 298,065,920 at 1M), fjall-unflushed's 1M store within one 4 KiB block, and sled 2.3 % apart at 100k (64.0–65.5 MiB) and 1.0 % at 1M (534–539 MiB); fjall's 1M store holds 35 files in every build. Log: the native journal (journal.{log,off,ckpt}), SQLite's WAL, fjall's journals; redb has none, and sled's log is its data file.

Verdict: the native store is third of the four durable engines: 284 MiB for 1M changesets, 298 bytes each: 1.45× SQLite's 197 MiB (206 bytes), 1.21× fjall's 235 after its flush (247), 0.62× redb's 460 (483) (sled, a reference line: 538, at 565 bytes). Over a third of it is the journal (101 MiB), which the native store keeps whole as the record of truth, the index being a projection of it that replay can rebuild (storage-core §2). The rest is the columnar index, which stores each id and each delta hash twice more, in its column and in its sorted view: 112 bytes of columns and parents and 80 of sorted views per record (from the 1M seed's file list: meta/idx/{ids,dh}.col and meta/parents.log 32 bytes each, meta/idx/{gen,par}.col 8; meta/sorted/{ids,dh}.key 32, meta/sorted/{ids,dh}.rec 8), the same per record at 100k. SQLite's B-trees hold each key once per index; redb's copy-on-write file keeps free pages (514 MiB apparent, 460 allocated); fjall's active journal is 18 % of its flushed 1M store, and at 100k all of its journal stays beside the new tables (40.9 MiB, against 21.6 MiB unflushed, all journal).

W6: a reader while a writer commits#

On a copy of the 100k store: a reader looks a changeset up by id in a fresh read transaction each time (sled has no read snapshots: its reader looks up the latest state), 20,000 times idle, then for 10 s while a writer commits W1's transaction back to back. Reader and writer are two threads of one process, each with a session of its own (SQLite: two connections), for every engine. µs:

Verdict: the native reader is never blocked, and stays the slowest. Idle, a fresh snapshot and lookup takes 51.4 µs p50 (SQLite 4.37, redb 1.01, fjall 0.721; sled, a reference line, 0.861). Beside the writer's 64 commits/s it takes 87.3 µs p50, 179 µs p99, 1.3 ms p99.9 and 5.0 ms at worst [4.4–5.3]. Readers take no lock, so the writer never stalls them; what the writer adds is work per snapshot: the frames past the checkpoint (up to 31) are read and overlaid on every snapshot, and the ref files and the frames are read again when a commit lands meanwhile. The other engines' readers stay at or below 1.1× their idle p50 (SQLite 0.73×, redb 1.02×, fjall 1.08×; sled's 1.34 against an idle 0.861 [0.661–1.42]): SQLite's WAL readers 3.21 µs, redb's 1.03, fjall's 0.782 beside their writers, with worst cases of 0.47–1.2 ms at the median of the runs (fjall's 5.8 ms in one run).

The verdicts in one place#

At the loads and raw fdatasync latencies in the tables above: outside W2, load 7.7–15.2 at the start of the runs and 3.7–5.6 ms per fdatasync (7.1–7.2 in one W5 build); W2 at 6.7–9.7 ms, with the two load events named above in its first round of threads.

The journal syncs. Of a native W1 commit (16.1 ms here), the fdatasync of journal.log takes 23.7 % and that of journal.off 23.4 %; the ref file's temp-file fsync and directory fsync take 24.2 % and 22.6 %.

What would change the picture#

• Fewer syncs per commit. 94 % of a native commit is its four syncs, and the storage-core spec's own target is two per transaction (§3.1, principle 7). Making the ref files durable at the batched checkpoint, as the index already is, would take 46.7 % off a commit (the temp file's fsync and the directory's, 24.2 % and 22.6 % in the table, 46.7 before rounding); dropping the fdatasync of journal.off, as the code review suggests, 23.4 %. Both would leave one sync per commit, where SQLite, redb and fjall are, and a commit of about 5 ms here (16.1 ms less three syncs of 3.70–3.87). Each touches what a reader and the next writer's tail check may trust after a power loss, so each is a spec decision; these shares are what it would buy.
• Group commit. No durable engine here puts several writers' commits under one sync, so each one's W2 rate is bounded by its own per-commit sync cost (sled, a reference line, runs W2 above W1: 2,110 against 1,791 transactions/s). With 32 writers that is where the next factor lies, for the native store as for the others.
• The disk. Every durable engine here waits on fdatasync; the ratios follow the number of syncs per commit for as long as a sync costs more than the rest of a commit (94 % of it here), and the other engines' one sync costs about half of a raw append + fdatasync (W1: 1.82–2.34 ms against 3.8–4.7) where each native sync costs about one (3.70–3.87 ms). Most likely every native sync pays for a new length or a new inode that ext4 logs before the sync returns, while redb rewrites pages in place, SQLite rewrites its WAL from the start after each checkpoint and fjall writes into a journal it pre-sized to 64 MiB; journal files preallocated in place would test that. W2 ran on a disk about twice as slow as W1's (a 6.7–9.7 ms probe against 3.8–5.1); the native store's rate fell to 0.54 of its W1 rate and redb's to 0.51, about as much as the disk, fjall's to 0.40 and SQLite's to 0.22, so the ratios between the engines moved (against the native store: SQLite 7.0× in W1, 2.8–2.9× in W2; fjall 6.6×, 4.9×; redb 7.6×, 7.1×); the records do not say why.
• A long-lived reader. A long-lived server process (swarmops mcp today, a network server after v1) could keep the index's files open and cache the sorted views' upper levels, closing most of W3's and W6's gap; mapping the files is ruled out while unsafe is forbidden. A CLI command, which takes one or two snapshots, pays 0.1–0.2 ms.

M1.6 — publishes under 32 writers: one transaction against two (not measured yet)#

commit --advance makes a publish one write transaction: the changeset and the move of its ref in one journal frame, where commit and then ref set --expect take two. The CLI contention test measures what that saves under 32 writers (crates/swarmops-cli/tests/contention.rs): with SWARMOPS_TEST_CLI_CONTENTION_ADVANCE=1 each child publishes its 1,000 commits in one transaction each, commit --parent <its previous> --advance release/agent-<n>, and with =0 the same children publish in two, commit --parent <its previous>, then ref set release/agent-<n> <it> --expect <its previous>; the parent creates the 32 refs at the base changeset before the children start. The summary gives publishes/s and the journal frames each publish wrote, so its lock passes per publish: 1 against 2 by construction (a compaction's frame every 1,025 changesets aside), which the test asserts.

The per-child refs are the measurement's artefact, not a ref model: agents publish to main (umbrella decision 7), where every publish in a round but one loses its compare-and-swap, records nothing and runs again on the winner. So these numbers are never landing throughput; the shared-main compare-and-swap loop is M4's to measure.

Not measured yet: the two runs wait until after the pilot, when they run once, in a quiet window under the host's heavy-run lock, and fill this section. Each is

SWARMOPS_TEST_CLI_CONTENTION_ADVANCE=1 cargo nextest run --locked --release -p swarmops-cli \
  --run-ignored ignored-only --test all -E 'test(/^contention::thirty_two/)' --no-capture

with =1, and again with =0.

M4 — a land's incremental re-merge (not measured yet)#

A land that loses its race re-merges incrementally (operations spec §3 (g)): what it merged before, L, onto what the target gained since, over L's delta paths alone (Repo::merge_trees), where the naive retry merges the whole change again from its base (Repo::merge); operations spec §9 item 1 asks for the two side by side. benches/remerge.rs of swarmops-repo times both, incremental/<files>x<gained> and full/<files>x<gained>, for a change of 1, 10 and 100 files out of a base of 1,000, against a target that gained 1, 10 and 100 changesets elsewhere since the land's first attempt (a store built without syncs, the objects in the page cache).

Not measured yet: M4's measurements wait until after the pilot, when they run once, in a quiet window under the host's heavy-run lock (the M4 plan's execution policy), with

cargo bench --locked -p swarmops-repo --bench remerge

v0.1.0 — just perf in a quiet window (2026-10-21)#

Head: the v0.1.0 release commit (the one tagged v0.1.0), 556 commits after the 2026-10-16 run's, with M1.5, M1.6, M2, M3, M4, M8 and the v0.1.0 release batch merged in between. just perf ran once (store-level contention → CLI contention → store benches, one after another) under the host's lock for heavy runs, its three steps built beforehand so that the timed run compiled nothing. Wall time 1,463.7 s: 255.6 s, 463.6 s and 744.5 s for the three steps (the benches: 7 bench binaries, 92 benchmarks). Same host as M0: WSL2 (kernel 5.15.153.1), ext4 (data=ordered) on the same virtual disk, AMD Ryzen 9 7950X3D (16 cores, 32 threads), 106 GiB RAM; Rust 1.99.0, optimized builds. Every store lives in a fresh temporary directory under ~/.local/state/swarmops-bench.

The window was quiet: start load 0.93 → end load 2.27, and no other test, build or bench on the host. The 1-minute load of 7–12 during the CLI test is its own 32 fsyncing processes, and the disk under the stores was busy 99 % of the store-level test (813 writes/s) and 97 % of the CLI test (1,086 writes/s), its 32 writers' own load, and 45 % of the benches. The raw fdatasync probe (the median of 40 appends of 120 B, each sync_data'd) took 3.56 ms at the start and 3.71 ms at the end, M1's quiet-disk speed (3.7–3.8 ms; M0 measured 4.1 ms). One run (the M1 experiments above ran each measurement twice), so every p99 and max below is a single run's. Nearest-rank percentiles; criterion figures are medians.

compaction, history, and read_snapshot's store with refs have no earlier number in this file. compaction times one Index::compact, every sync on, as a writer makes it while it holds meta/lock once the unsorted tail has passed its threshold: it rewrites both sorted views whole, 80 bytes per record. history times, on linear chains of 200k and 1M changesets, Index::find (an interpolation search) against a plain bisection the bench copies, on the same reads, an insert that scans an empty and a full (1,024-record) unsorted tail, 100 first-parent steps by whole records (changeset) and by parent entries (first_parent), and the reflog of main's last 100 moves. read_snapshot's third store holds 0, 16 or 256 release/* refs besides main; show main resolved main in one snapshot and read its changeset in a second, and makes both reads in one now.

The lock pass, store against CLI. At store level the lock passes every 7.94 ms (254.2 s / 32,000), 2.2 raw fdatasyncs of the 3.56 ms probe; an uncontended transaction takes 7.01 ms and a contended one 7.16 ms in the lock. Through the binary it passes every 13.94 ms (446.1 s / 32,000), 3.9 raw fdatasyncs: 6.0 ms more (2026-10-16: 8.27 and 13.75 ms, 5.5 ms more). The 2026-10-16 section puts that difference down to the commits' own object writes on the same filesystem; this run did not isolate it.

The wait. The lock rotated at both levels. At store level the wait's p50, 280.74 ms, is 35.3 passes (about one round of the other 31 writers), its p99 51.4 passes and its max 66.3. Through the binary, estimated as meta.elapsed_ms minus 19 ms: p50 29.7, p99 43.2, max 49.7 passes. 7c's rotating runs had a p99 of 131–157 passes and a longest wait of 283–359, and a uniformly random winner among 31 waiters gives a p99 of 140 and a longest wait near 316 ("M1 store fixes (7c)"). Both tails stayed under 0.7 s, against the 10 s timeout. The 2026-10-16 store-level run was in the other regime, the committing writer taking the lock straight back (wait p50 0.11 ms, p99 108 passes, max 145). The lock's queueing is the code it was then (only its open and its error messages changed), and 7c found that the regime depends on the host.

Against the 2026-10-16 run.

• Unchanged within 5 %: the store-level and CLI throughput (125.9 transactions/s against 120.9, 71.7 commits/s against 72.7), the uncontended transaction (7.01 ms p50 against 7.24), fsync_fanout, put_batch with and without the busy writer, and flush. The read_snapshot medians moved by at most 7 %, and the uncontended commit went from 21 to 19 ms (whole milliseconds). The disk ran at M1's quiet-disk speed; the host's start load was 0.93, against 16.4 then.
• The lock wait changed regime, not throughput, on the same queueing code (above). Its tails are the shortest recorded for either test (store p99 0.41 s, longest 0.53 s; CLI p99 0.60 s, longest 0.69 s, estimated), at the lowest start load of any run (M1's runs started at 7.6–33).
• index_lookup is about 2× faster (−43 % to −63 %). Two commits between the runs aim at it, 72f9112 (a guarded interpolation search over the sorted views) and 092d077 (find without re-reading the key it found, and lookups that read one field), and the 2026-10-16 run had a busier host (start load 16.4, a debug build of another worktree during its benches); this run does not separate the causes.
• read_snapshot's p99 is higher with 31 frames past the checkpoint (132.7 and 152.9 µs, against ≤ 126 µs): single-run tails, while its p50s moved by at most 7 %.

M0's fsync, inside just perf. The criterion means, 22.4 µs, 3.54 ms, 3.58 ms and 7.07 ms, are 54 %, 13 %, 11 % and 20 % below M0's (49 µs, 4.07, 4.01 and 8.81 ms): the same speed class. M0's no-fsync mean was twice its own per-write p50 (23 µs), where this run's mean and median agree (22.4 and 22.5 µs). just perf takes no per-write percentiles, so M0's per-write table is not re-measured; the probe's 40 appends at the start had a p90 of 4.09 ms and a worst of 5.56 ms.

Decisions confirmed on this run: the kernel-queue lock keeps Busy and E_BUSY at 0 of 32,000 at both levels, the longest waits 0.53 s (store) and 0.69 s (CLI, estimated), so lock_timeout stays at 10 s; parallel Index::sync is 2.47× serial (decision (b) asks at least 2×); 32-thread put_batch + flush reaches 13.1× the 1-thread rate, and 14.9× with the busy writer (decision (a) reconsiders the flag below 4×), so a --durability fast flag is not justified by this data.

Not in just perf, so not refreshed: M0's per-write percentiles (fsync_percentiles), the M1.5 comparison (just bench-compare), and the M1.6 publish pair and M4's re-merge bench above, which stay not measured.