4 · Talking between machines Draft

Chapter 4 — Talking between machines

5 min read

On this page

Everything so far has happened inside one machine: one core and its caches, then the operating system around it, then several threads agreeing about memory. A trading program’s work starts and ends outside that machine. Every input is a packet: a depth update from a venue’s market-data feed, a trade, the acknowledgement of an order, a fill. Every output is a packet too: a new order, a cancel, a cancel-replace. So the hot path does not begin when the strategy thread starts deciding. It begins when a packet lands in DRAM, written there by the network card while no core was looking. What the program asked for was simpler: it called read on a socket and asked for the next bytes of a stream the far end wrote. Between those bytes, sitting in a buffer the card filled, and the program’s own buffer lies a chain of hand-offs that the source code does not show: an interrupt, the kernel’s protocol code checking the envelopes and peeling them off, a queue that belongs to the socket, a thread that may be asleep and must be woken, and a copy. Further out still are the cable, the switch and the distance to the venue, which no code touches at all.

That path has prices, and they sit badly against the 1 µs budget. The card’s part alone, from the last bit off the wire to the packet in DRAM with its interrupt raised, is 1–2 µs, before any software has run. The kernel’s receive path, from that interrupt to read returning in a thread that was asleep waiting for it, is 5–20 µs on a quiet box. If the reader’s core had dropped into a deep C-state, a power-saving sleep it takes when it has nothing to do, the wake-up alone is 100–200 µs. Those are the costs when everything works. When TCP’s own habits come into play they grow by three or four orders of magnitude. Nagle’s algorithm, TCP’s rule that a small segment, one packet’s share of the stream, waits while an earlier one is unacknowledged, can hold a write for 40 ms when the far side is deliberately delaying its acknowledgement. A lost segment that TCP’s quicker recovery misses waits for a retransmission timeout, the timer TCP falls back on before it resends, and a common Linux default minimum is 200 ms, though current kernels allow it to be configured. A fresh TLS connection to a venue 35 ms away costs 140–210 ms of handshakes, the exchanges that open the connection and agree its keys, before the first byte of application data moves. Under all of it is the floor nobody removes: light in glass takes 70 ms to go from Tokyo to Singapore and back, seventy thousand budgets, and no card, kernel or language changes it.

None of the expensive cases is the common one, and that is the shape of the problem again. The common case is cheap by comparison: a benchmark that sends a message to the box next door and times the reply reports 20–50 µs, much the same every time. The running system’s histogram carries the rest in its p99.9 and its max: the kernel waking a reader from a deep sleep, a stall while one side waits for the other’s acknowledgement, a lost segment and every message queued behind it. The second price is correctness, and it lies in what the program assumes about the bytes. A TCP connection delivers a stream, not a sequence of messages: one read can return half a message, or three and a half, and code that assumes one message per read passes every test on a quiet link and fails under load, when bytes arrive faster than the reader drains them. A UDP feed loses packets, and nothing reports the loss except a gap in the sequence numbers that the program must notice for itself. And the most convenient way to turn bytes into a struct, a JSON parser, allocates for each string field it meets, breaking the hot path’s rule of no allocation without a line of the program saying so. None of this is particular to trading. A web server, a database replica and a game server meet every one of these mechanisms; a trading program only measures them in smaller units.

The chapter starts with what the network is to a program: the socket and the calls that drive it, the hardware beneath it, the envelopes counted in bytes, and the physical floor under every number that follows. Then the ladder of what each kind of exchange costs, from two threads on one machine to two machines on opposite sides of a sea. Then one mechanism per section, the anatomy of a receive, the anatomy of a send, the two kinds of delivery that TCP and UDP offer, the ways a thread can wait for a packet, the turning of bytes into messages, encryption, and taking the kernel out of the path altogether, asking of each what it is, why a machine shared by many programs does it that way, and what it costs. What follows for a hot path is collected in one place near the end, after the understanding that justifies it. The labs specify programs to implement under ch04-network/; no companion binaries are included here. The numerical examples assume a stated setup and are predictions, not measurements. Driver, kernel and library versions can change both costs and behaviour.

Readings in this chapter