Measurement and the toolbox

0.11 Measuring latency

Latency is how long one thing takes, from a defined start to a defined end: packet arrival to order departure, say. Throughput is how many things complete per second. A system can have high throughput and poor latency, and a trading system cares about latency, one message at a time.

Measure many of them and you have a distribution, and the right way to describe it is by percentiles. The p50, the median, is the value half the measurements fall below; the p99 is the value 99% fall below; the max is the worst one seen. The slow end, above the p99, is the tail. A mean hides it: a thousand messages at 1 µs and one at 1 ms average to 2 µs, a number nothing actually took, while the one slow message is the one that missed a trade. Every lab in the book reports p50, p99 and max and never a mean.

A histogram is how you keep a distribution without storing every sample: count how many measurements fell into each bucket. Buckets spaced on a log scale (1–2 µs, 2–4, 4–8, …) cover microseconds to seconds in a few dozen counters, and hdrhistogram offers configurable precision over a preallocated range. Disable automatic resizing and handle out-of-range samples explicitly to keep recording allocation-free.

The traps are in the setup. The first pass through any code is slow for reasons that have nothing to do with the code: page faults, cold caches, an untrained branch predictor; warm up before timing. A core’s clock speed changes with load and temperature, so pin the governor. The OS can pause your thread mid-measurement, so pin the thread and measure on an isolated core. Reading a clock costs 20–30 ns, so do not time things that take 4. And when a benchmark stops generating requests while waiting for a reply, it omits the requests that would have arrived during the stall; Gil Tene calls this coordinated omission and Chapter 7 explains the fix.

Two tools do most of the work. perf stat reads the CPU’s hardware counters over a whole run: cycles, instructions, cache misses, branch mispredicts; instructions per cycle (IPC) is the first number to look at, high for compute-bound code and low for memory-bound. perf record samples where the core was every few thousand cycles, and a flame graph draws those samples as a stack of boxes, wider for functions that took more time. Together they say how fast, and where the time went; the predict-first rule says what you expected before you looked.

0.12 The toolbox

Every tool the labs use, what it shows, and the chapter where it is used. Install them before Chapter 1: perf comes with linux-tools, numactl, strace and tcpdump from the distribution, cargo-show-asm as a Cargo-installed tool, and hdrhistogram as a dependency of the lab programs.

Further reading

• Randal Bryant and David O’Hallaron, Computer Systems: A Programmer’s Perspective: chapters 1, 3, 6, 9 and 11 cover sections 0.1–0.7 of this chapter properly.
• David Patterson and John Hennessy, Computer Organization and Design, chapters 1–5, for the instruction cycle and the pipeline from first principles.
• Nand2Tetris, a free course that builds a CPU from logic gates; the first six projects make section 0.3 concrete.
• Beej’s Guide to Network Programming, the Chapter 4 reading, for sockets and the TCP/UDP calls in practice.
• The Wikipedia articles on the instruction cycle, von Neumann architecture, cache lines, system calls and the order book as quick reference.
• Brendan Gregg, Systems Performance, chapter 2, for the measurement vocabulary of section 0.11.