Warm-up, frequency, and core placement

7.5 Keeping the number true

A number is a claim, and it can be false without any error being raised. “Application-arrival-to-decision p99 is 1.2 µs” claims that on this machine, with this binary and this input, 99 messages in 100 went from arrival to decision in 1.2 µs or less, and that the same run tomorrow would say the same. This section lists the ways that claim fails, in the order a first harness meets them. Each is explained as a mechanism, then by what it does to the number, then by its fix in a sentence or two; the fixes are gathered again among the rules near the end of the chapter. Nothing below is a trick.

No warm-up

A program’s first iterations run on a machine that has not seen it yet. The kernel supplies each 4 KB page of memory on the first write to it, through the OS chapter’s page fault, a trap into the kernel of 0.5–2 µs for a page that must be found, zeroed and mapped; the capstone’s ArrayBook alone is 256 pages, and its rings and histograms add 270 more. Every cache line of the working set, several hundred of them, comes first from DRAM at 80–100 ns. The branch predictor, the part of the core that guesses which way each branch will go before it is computed, guesses wrong at 15–20 cycles a time until it has seen the loop a few thousand times. A Vec that grows by doubling copies its contents at every growth, the last growth of a million samples 4 MB in one iteration, 0.5–1 ms. And some work happens only once: the first call to each shared-library function goes through the dynamic linker, and a OnceLock initialises on first use.

So the first thousands of samples are 10 to 1,000 times slower than the rest. In a run of 2 M they sit above p99.9 and set the max; in a benchmark of 10,000 iterations they are the p99. The opposite failure exists too: a benchmark that applies one update to one level a million times is warmer than any feed, every line in L1 and every branch predicted, which is one reason the capstone replays a recorded feed.

The fix is to discard the first N iterations and say how many: the capstone discards 100,000 messages, a paced run’s first second. A warm-up is long enough when page faults have stopped and the p50 of successive 10,000-message windows has stopped falling. It warms only what it touches: a page of the book the warm-up’s prices never reached still faults when the run first reaches it.

The compiler removed the work

A result nothing reads is dead code; the optimiser deletes it, and with it the work the benchmark meant to time. The data-structures chapter met this and answered it with std::hint::black_box, which makes the compiler assume a value is used, best effort and for benchmarks only. The number comes out too good: a loop body measured at 0.25 ns, one cycle at 4 GHz, did nothing. Two checks catch it. No cost can be below the floor of the memory the work must touch: an apply cannot beat its one L1 load and store, about 1 ns, nor a hop one line crossing between cores, 50–100 ns. And perf stat’s instruction count divided by the iteration count should be the tens to hundreds the code needs, not three. The capstone needs no black_box: its results leave the thread.

The clock rate moved

The core’s clock is not fixed. A core with little to do drops to 0.8–1.5 GHz; one busy core among idle ones may turbo to 4.5–5.5 GHz, above the advertised base, and falls back as other cores get busy or the package heats. The kernel’s governor, its policy for choosing each core’s frequency, raises the clock under load within 1–50 ms, or hands the choice to the CPU’s own hardware. The TSC follows none of this. So 200 cycles of work take 50 ns at 4 GHz and 167 ns at 1.2 GHz, 170 and 567 ticks of a 3.4 GHz TSC.

The number then moves with things that are not the program. A variant measured on an idle machine at a single-core turbo of 4.8 GHz beats one measured during a compile on the other cores, at 3.9 GHz, by 23% for no reason of its own. A loop that sleeps between messages lets its core enter an idle state, which takes 1–100 µs to leave; the capstone’s loops spin.

The fix is the performance governor, which holds the highest clock the limits allow (cpupower frequency-set -g performance), with turbo off, so that every run gets the base clock the part is rated to sustain: on Intel with the intel_pstate driver, echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo as root; on AMD, cpupower set --turbo-boost 0 with a recent cpupower, or echo 0 > /sys/devices/system/cpu/cpufreq/boost under the acpi-cpufreq driver. Then say which: “performance governor, turbo off, 3.4 GHz” belongs beside every number.

The wrong core

A thread that is not pinned runs wherever the scheduler puts it, and the scheduler moves it. A migration lands it on a core whose caches hold none of its lines, and refilling several hundred lines from L3 or DRAM costs 5–50 µs, spread over the next messages. A thread pinned but not isolated still shares its core, with other processes for scheduler slices of 1–4 ms and with kernel threads for tens to hundreds of microseconds. Something on the core’s second hardware thread, which Intel calls Hyper-Threading, shares its L1, L2 and execution units, and both run 10–40% slower. Interrupts land where the kernel steered them: 2–20 µs for a network interrupt and its follow-up work, 1–5 µs for the timer tick, 250–1,000 times a second. And a clock reading is taken on whatever core the thread is on at that instant: within one socket the TSCs agree to a few ticks, but across sockets or in a VM a hop computed across cores can come out negative.

The p50 moves with where the threads landed, two cores sharing an L3 or two on different dies; the p99 with migrations; the p99.9 and max with everything else that ran. Unpinned, the max is 50–1,000 µs; pinned and isolated, 5–30 µs.

The fix is to pin each hot thread to its own core, isolate those cores with isolcpus and nohz_full and move interrupts off them, as the OS chapter set out, and prove it: perf stat -e context-switches,cpu-migrations,page-faults must read zero after warm-up, and rdtscp, which returns the core number with the time, shows two readings came from the cores intended.