Measurement decisions for a hot path

7.9 What follows for a hot path

The chapter’s mechanisms leave ten rules for measuring a hot path, and nothing in them is a trick: each rests on a number measured earlier in the chapter, and a reader who disagrees with one should be able to point at that number. They apply to any program whose slow moments matter, and the capstone follows every one of them.

1. Record every event, and report p50, p99, p99.9 and max, never an average. A record into a histogram costs 10–30 ns and a histogram from 1 ns to 1 s at 3 significant figures costs about 170 KB, while one 10 ms stall among a million samples of 300 ns moves the mean by 10 ns and the max by 10 ms, so the mean hides exactly the event the max reports.
2. Take the first timestamp at the earliest point the program owns, and measure in open loop, from when each message arrived or was due rather than from when the loop got to it. A closed-loop harness charges a 10 ms stall once, as one slow sample, where at 100,000 messages a second the market sent about 1,000 messages during it; measured open loop, those 1,000 messages lie between 0 and 10 ms and the p99.9 of the capstone’s 19 measured seconds moves from microseconds to about 9 ms. Report service time beside response time, and call only response time the product.
3. Pin, isolate and lock memory before trusting any percentile above the p50. The capstone’s max is predicted at 5–30 µs pinned, isolated and locked and at 50–1,000 µs unpinned, and a single page fault adds 0.5–1 µs, or 100 µs and more when the page must come from disk; perf stat -e context-switches,cpu-migrations,page-faults on the hot threads must read zero after warm-up, or the histograms describe a different program from the one intended.
4. Warm up, leave the warm-up out of the numbers, and say how long it was. The first iterations pay for first touches of pages, cold caches, branch predictors still learning the loop and collections still growing; the capstone discards its first 100,000 messages, its first second, and a result that does not say what it discarded cannot be repeated.
5. Keep the measurement under 2–5% of the budget, 20–50 ns of a 1 µs iteration: one clock read per stage, samples stored at about 1 ns into a preallocated array, and the records made after the timed region. Three reads on the path and four records cost 100–150 ns a message, so the stage timers belong in a separate build, the headline comes from t0 and one read of t3 per batch, and the p50 of two back-to-back reads, 20–30 ns for the fenced read, is subtracted from every stage.
6. Keep the identity of the slow samples and reproduce them by replay. A histogram keeps neither order nor names, but a 1.6 KB array of the worst 100 samples with their sequence numbers turns “the p99.9 is 6 µs” into a message that can be cut from the replay file and run alone: slow again alone, the cause is in the data; fast alone, it was the machine.
7. Profile the body with a flame graph and the tail with stage timers. A flame graph shows each function’s share of all the samples, an average over the run, so a p99.9 set by one message in a thousand is not in it; the stage timestamps of the slow messages, replayed by sequence number, say which interval they were slow in, and the deltas of page faults, context switches and /proc/interrupts over the run say whether the kernel was the reason.
8. Same binary, same input, same flags: the same decisions, proved by a join on sequence number. A replay file gives every run the same messages, and a decisions file joined on sequence number shows that a swap changed the time and nothing else; without the join, a faster variant may be faster because it decides differently, or decides wrongly.
9. Let the loop measure itself in production the way it does in the lab, with the same points and the same histograms, read by another thread. Once a second the hot thread hands its histogram to a reporting thread through a small SPSC ring and takes back a cleared one, so it never shares a line with its reader and never spends the 5–10 µs of clearing 170 KB itself; a reader that read the live histogram instead would cost the writer a line transfer of 50–100 ns on each line it had touched.
10. Report the count and the machine with every percentile. A p99 of 100 samples is one of its two largest, a p99.9 needs thousands and a p99.99 hundreds of thousands, and a percentile means something only beside the CPU model and its TSC rate, the cores and whether they share an L3, the governor and the turbo setting, the kernel and its isolation settings, the build, the mode and the input file; without them two p99s, even from one program, cannot be compared.

Four things are deliberately not done. There are no hardware timestamps on the network card, where the card stamps each packet as it arrives against a clock that the Precision Time Protocol (PTP) keeps within well under a microsecond of other machines’ clocks, which is how a desk measures the kernel’s receive path and the wire itself; it needs a card that supports it and a timing network of its own. There is no distributed tracing, which carries an identifier and timestamps with each request across processes and machines so that one slow request can be followed through everything it touched, and which needs clocks that agree across machines, the problem PTP solves. There is no continuous benchmarking, which runs the capstone on every change to the code, on a dedicated machine tuned like the lab’s, and fails the change when a percentile moves by more than the run-to-run noise; it needs a quiet machine and a statistical rule for what “moved” means. And there is no live latency dashboard, which ships the per-second histograms to a time-series store and draws the p99 across the day; it is operations work, and its only demand on the hot path is the one already made, that the loop measure itself. The capstone needs none of them, and each is a chapter of its own.

The book comes together in one sentence: a hot loop is a cache-line budget, a kernel kept out, one writer per line, bytes read where they land, state held in arrays, a book kept true, and a number taken honestly. Each part of that sentence was a chapter, and each chapter came down to a mechanism with a price on it, measured on an ordinary machine with ordinary tools: perf, rdtsc, a histogram and a recorded feed. None of it is magic. It is the same work every performance-sensitive program does, a game holding its frame time, a database its p99, a web service its slowest page, with a market as the example. The final test is the one the book began with: read a hot loop, any hot loop, and say why each line is there.