The network latency ladder

4.2 The latency ladder of the network

This ladder runs from the cheapest way two pieces of code can exchange a message to the most expensive, on the same 4 GHz core, against the same 1 µs budget. The first ten rows are round trips: the time from sending a one-byte message to receiving a one-byte reply, with the far side doing nothing in between but answer. A blocking read puts the reading thread to sleep until data arrives. A reader busy-polls when it never sleeps: it asks for data in a loop, with a non-blocking recv that returns at once when nothing is there, or with the kernel’s busy-poll option, under which the kernel checks the card’s ring itself inside the call. Kernel bypass means the program, through a library, reads and writes the card’s rings directly and runs its own protocol code, with the kernel out of the path. The protocol walk is the kernel’s code that checks and strips each envelope in turn. Interrupt coalescing is the card holding its interrupt back for a set time after a packet lands, so that one interrupt can announce several packets. A retransmit is TCP sending a segment again because it believes the first copy lost, and the retransmission timeout is how long it waits for an acknowledgement before it decides so. Real parts vary by a factor of two either way; the shape does not.

Read the table against the budget. Only the first two rows fit inside it, and neither uses the network: they are Chapter 3’s cross-core transfers, set here for scale. Every row that touches a socket costs several budgets: three to six for the best that two machines can do, with the kernel out of the way at both ends, and twenty to fifty for the ordinary kernel path with a sleeper at each end. Below the two-machine rows distance takes over, and the cross-region row costs seventy thousand budgets whatever the software does. The last five rows are not rungs of their own. Each adds to whichever rung a program sits on, whenever its event happens.

Direct and indirect cost

Each rung has two costs, and the table shows the first. The direct cost is the round trip a one-byte ping-pong measures. The indirect cost is paid by other work. The core that takes the card’s interrupt and runs the protocol walk spends a few microseconds per packet or batch, and leaves the kernel’s code and buffers in its caches, so the next thread to run there misses on what they evicted. Every wake-up interrupts the reader’s core. TCP data segments generate acknowledgements, often one covering several segments, a packet the sender’s card and kernel must take in. And the path is a queue: while one message is held, by a wake-up, a coalescing timer or a lost segment, the messages behind it wait in the socket’s receive queue. A 20 µs stall on a feed carrying an update every 10 µs delays the next two as well; a 70 ms stall delays seven thousand. The direct cost is paid once per message. The indirect costs are paid by the core that does the kernel’s work, and by every message that arrives behind a slow one.

Which rungs matter

The rungs down to the bypass row are paid on every message, and two choices decide which of them a program sits on: whose code walks the protocol, the kernel’s or the program’s own, and whether the reading thread sleeps or spins. Those rungs set the p50. The tail rows are paid only when something particular happens, and they set the p99 and beyond, with two twists of their own.

The first is that one tail row grows as traffic falls. A strategy thread on a feed that sends 200 updates a second waits 5 ms between them, long enough for its idle core to reach its deepest sleep, so in a quiet market the 100–200 µs wake-up is not the tail but the median. The second is that TCP turns one rare event into many slow messages, because every byte behind a lost segment waits for it. A good path that loses one segment in 100,000, carrying 10,000 segments a second, loses one every ten seconds. TCP’s fast retransmit, which resends as soon as the receiver’s acknowledgements show that three later segments arrived without the missing one, recovers it in about one round trip: 70 ms to a venue 35 ms away. The 700 segments that arrive in that time all wait behind the hole, 0.7% of the traffic, delayed by anything from 70 ms down to almost nothing. The p99 does not move, the p99.5 is about 20 ms and the max is 70 ms. The expensive rungs live in the tail, and from one event this one reaches a long way into it.

Reading your own machine

Which rung a box sits on, and how often it visits the tail rows, is written in four places. The first is the card. ip link lists the interfaces with their MTU and state, and ip -s link adds packet, error and drop counts. ethtool -i <iface> names the driver, its version and the card’s PCIe address, its bus-info; lspci -vv -s <bus-info> then prints the LnkSta line, the speed and width the slot actually gave the card, which can be less than the card wants.

The second is what happens to packets before the protocol walk. ethtool -c prints coalescing: rx-usecs is how long the card may hold its interrupt, and Adaptive RX: on means the driver picks that delay itself, and picks longer ones as traffic rises. ethtool -g prints the receive and transmit rings, current size against maximum; a ring is where packets wait when the kernel falls behind, and its size is how far behind the kernel can fall before the card drops. ethtool -k lists offloads, batching done by the card or the kernel, such as merging consecutive segments of one connection before the protocol walk (generic receive offload, GRO). ethtool -S prints the card’s own counters, per queue; the names vary by driver, and the ones to find count frames missed because a ring was full.

The third is which core hears about each packet. A server card has several receive queues, each with its own ring and its own interrupt, and puts every packet of one connection on the same queue by hashing its 4-tuple, which is called receive-side scaling. /proc/interrupts has a row per queue and a column per core: read it, send traffic on one connection, read it again, and the cell that moved names the queue and the core that connection uses. /proc/irq/<n>/smp_affinity_list says which cores that interrupt may be sent to.

The fourth is the path. ss -ti prints each TCP connection with what the kernel keeps for it: rtt, the smoothed round trip in milliseconds; rto, the current retransmission timeout (commonly at least 200 ms under default settings); cwnd, the congestion window, how many unacknowledged segments the sender allows itself; retrans, the segments sent again; and, in the Recv-Q and Send-Q columns, the bytes waiting in the two queues. ping times an ICMP echo, a small control message carried directly in IP that the far kernel answers inside its receive path without waking any program, so a program’s own ping-pong adds the far side’s wake-up and copies to what ping shows. mtr sends the same probes to every router on the path and prints the latency and loss at each; loss at one router that does not carry on to the routers after it is that router declining to answer probes, not loss on the path.