7 · The number Draft

Chapter 7 — The number

4 min read

On this page

Every chapter so far has ended in a number. A load that misses every cache and goes to DRAM costs 80–100 ns. A cache line that another core wrote last costs 50–100 ns to fetch. A push into the SPSC ring costs 5–20 ns while its cached copy of the other side’s counter is valid, an apply on the dense array 2–5 ns, and a send through the kernel 2–5 µs. Each was predicted, then measured, and reported as a p50, a p99 and a max against the budget of 1 µs per hot-path iteration, 4,000 cycles at 4 GHz. None of those chapters said where a number comes from: what counter was read, what reading it cost, how a million readings became three figures, or when the three figures deserve to be believed. This one does.

The shape of the problem is the same as before. The common case is cheap and the tail is the product: the update that waits 10 ms is the one that arrived in the burst that moved the market, and the order it should have caused is the one that mattered. A measurement has a tail of its own, and it hides in the same place. The slow iteration may be one the harness never timed, because the harness was waiting for the program and sent nothing while the program was stuck. The two timestamps subtracted may have come from two counters that do not agree, so a hop between cores reads as 40 ns, or as minus 40. The average reported may be a number no event ever had: a million updates at 300 ns and one at 10 ms average 310 ns, and no update took 310 ns. Each of these produces a clean, plausible figure and raises no error. A number that is wrong in this way is worse than no number, because a decision gets made on it.

None of this is particular to trading. A web service that promises 99% of requests in under 200 ms, a service-level objective, has to measure the 99th percentile from the moment a request arrived, not from the moment a worker thread was free to take it, or the promise is checked against the wrong number. A game running at 60 frames a second has 16.7 ms per frame, and one frame of 50 ms is a stutter every player sees, though the average frame rate barely moves. A database’s p99 query time decides how many replicas a team buys. The clock, the histogram and the ways a number lies are the same in all of them; a trading loop is a convenient place to look because its budget is small enough that the measurement itself becomes a visible part of the cost.

The chapter starts with what a latency is to a program: two points in time, chosen by the program, and the distribution that a run of events produces, with percentiles as the way to read it. Then the clock: the counter in every core that a timestamp is read from, what reading it costs, and when two readings on two cores can be subtracted. Then the ladder of what each act of measuring costs and does to the thing measured, and the histogram, the structure that holds a million samples in fixed memory. Then the ways a number lies, in the order they bite, and how to find where the time goes once the number is true. Then the capstone, which wires the preceding chapters’ pieces into one loop and measures it end to end; the loop itself, line by line, with the reason for each line; the rules that follow for a hot path, collected in one place near the end, after the understanding that justifies them; and the lab. The code is in the companion repository under ch07-capstone/; the repository holds the code and this chapter holds the reasons.

Readings in this chapter