7 · The number Draft
Repeatability and measurement setup
On this page
Not repeatable
A measurement that cannot be repeated cannot be compared, and comparing is what a swap is for. The rule is that the same binary, the same replay file and the same flags give the same decisions. Three things break it: an input that changes, which is why the capstone reads a file and not the network; a program that draws on something besides its input, such as Rust’s default HashMap, seeded randomly in each process so that anything decided by walking one changes from run to run; and timing, which moves where a batch ends, so two capstone runs are compared at the sequence numbers where both decided, and there their lines must match exactly.
Without that check a timing difference may not be the swap’s: a TreeBook with a bug that loses levels quotes less often and shows a faster think for a reason that has nothing to do with trees. Identical runs also differ by a few per cent at p50 and more in the tail, and a difference smaller than that spread is not a difference.
The fix is to record the binary’s hash, the file, the flags and the machine with every result, run each variant three to five times, and join two variants’ decisions on sequence number before comparing their histograms.
What it costs
The fixes cost setup time and some of the machine:
| Fix | What it takes | What it costs while running |
|---|---|---|
| Warm-up, discarded | a counter and a branch | 100,000 messages, a paced run’s first second |
performance governor, turbo off | two commands as root, lost at reboot unless set at boot | full power on the hot cores; 10–30% below turbo’s single-core speed |
| Pinned and isolated | sched_setaffinity per thread; boot parameters and a reboot | three cores out of general use, each spinning at 100% |
| Memory locked | one mlockall, with the locked-memory limit (ulimit -l) raised | the 112 MB file and the whole heap held in RAM |
| A lean harness | a build feature for the stage timers | 20–30 ns per clock read; one t3 per batch in the default build |
| Enough samples | longer runs | 10 s for a p99.99 resting on 100 samples; minutes to meet once-a-minute events |
| Open loop | a replay file with arrival times, 56 bytes a record | the feed core spins between arrivals; a run takes 20 s, not a fifth of one |
| Repeatable | a decisions file and a join | three to five runs per variant |
None of it is expensive beside a number that is wrong in a direction nobody can see.