NIC and socket configuration

4.10 Getting the network out of the way

Each of the mechanism sections ended on what its mechanism costs. This section turns those costs into how a machine, its sockets and the program on them are set up, and how the result is checked, and ties each choice to the mechanism it answers. It is the only place in the chapter that reads as a list of things to do, and that is deliberate. The principle under all of it fits in one line: poll, never wait; parse in place; and keep the kernel off the hot path’s core, except for the hot socket’s own work.

How the machine is set up

The machine starts as Chapter 2 left it: cores 3–7 isolated and 0–2 doing the housekeeping, every interrupt line steered to 0–2, irqbalance stopped, C-states capped. The feed thread runs on core 3 and the order gateway on core 4. What follows is what the network adds, on the card that carries the hot traffic, <iface> below.

1. The receive queue that carries the hot socket’s packets has its interrupt steered, on purpose, to the hot thread’s own core when that thread busy-polls, through the queue’s /proc/irq/<n>/smp_affinity_list. The busy-polling recv runs that queue’s poll on the hot core anyway; with the interrupt there too, the queue’s descriptors, buffers and socket never cross between cores. napi_defer_hard_irqs and gro_flush_timeout under /sys/class/net/<iface>/, with SO_PREFER_BUSY_POLL on the socket, keep that interrupt masked while the thread polls. A thread that spins on a plain non-blocking recv is the other case: its queue’s interrupt goes to a housekeeping core that shares the hot core’s L3, so that the protocol walk runs beside the thread and not on it.
2. Every other queue’s interrupt stays on cores 0–2, and irqbalance stays off, since its next pass would undo item 1.
3. Interrupt coalescing is off on the hot card: ethtool -C <iface> adaptive-rx off rx-usecs 0. Adaptive coalescing lengthens the card’s hold as traffic rises, when the feed matters most. The possible increase in interrupts, mitigated by NAPI batching, falls on the housekeeping cores, and on the hot queue item 1’s deferral masks it.
4. GRO is off for the hot card, and so is LRO, large receive offload, the card’s own merging of segments: ethtool -K <iface> gro off lro off. Whether small messages merge or incur a hold depends on packetisation and the driver; compare offload settings with a latency histogram, since bulk flows lose throughput, one reason the hot traffic gets a card or a port of its own.
5. The receive rings are large enough that a burst does not drop: ethtool -G <iface> rx 4096, where the card allows it. At 10 GbE’s rate of minimum frames, 1,024 slots fill in 69 µs and 4,096 in 275 µs, and a drop on a TCP feed is a retransmit. No larger, since the card cycles through every buffer and fewer stay in cache.
6. Each hot thread has a receive queue of its own. An ntuple rule, a filter in the card that matches a flow’s addresses and ports, fixes the hot connection’s queue instead of leaving it to the hash: ethtool -K <iface> ntuple on, then ethtool -N <iface> flow-type tcp4 src-ip <venue> src-port <port> action <queue>. ethtool -X weights the indirection table, the card’s map from hash values to queues, so that no other flow lands there. RPS and RFS stay off, as by default: with threads pinned and interrupts placed by hand they only add a hop. On the send side XPS gives each core a transmit queue of its own through xps_cpus.
7. C-states stay capped on every core, not only the hot ones, because a reply whose interrupt lands on a housekeeping core pays that core’s exit latency.
8. The MTU stays at 1,500 unless every device on the path takes jumbo frames. A 48-byte message gains nothing from a larger one; a 9,000-byte frame ahead of it delays it by 7.2 µs at 10 GbE instead of 1.2 µs; and one device that refuses jumbo frames drops them, so the connection works for small messages and hangs on the first large one.

How the sockets are set up

1. TCP_NODELAY is on for every order socket, and for every TCP socket that sends small messages: set_nodelay(true). Nagle against a peer that delays its acknowledgements holds a write for up to 40 ms.
2. Every hot socket is non-blocking: set_nonblocking(true). A receive on an empty queue returns WouldBlock. A send can accept a short prefix or return WouldBlock for local, congestion-window or peer-window reasons. Retain unsent bytes and an offset in a bounded output buffer; resume in order, and define what happens when that buffer fills. write_all can fail after partial progress, so retrying the entire message can duplicate bytes.
3. SO_BUSY_POLL is set to 50 µs on the hot socket, or the thread spins on a plain non-blocking recv. On a non-blocking socket the option makes each empty recv run one pass of the driver’s poll before returning, so the thread’s loop is the spin and the kernel’s poll rides inside it. Raising the value above net.core.busy_read needs CAP_NET_ADMIN.
4. SO_RCVBUF and SO_SNDBUF are sized for the burst and set before connect, which sets explicit limits and disables receive-buffer autotuning when SO_RCVBUF is set, so the first burst after a connect does not meet a window still growing. The receive buffer covers the burst rate times the round trip, about 440 KB for 50 Mbit/s over 70 ms, plus the kernel’s charge for its own structures, up to 2 KB a segment however small the payload. The receive setting is capped by net.core.rmem_max (the send setting by wmem_max), which usually needs raising first, and doubles it to allow for that charge.
5. Each hot thread has one hot socket and is pinned to the core its queue’s interrupt is steered to. Two threads on one socket contend for its lock and move its lines between their cores on every call.
6. SO_TIMESTAMPING is on, for measurement: once hwstamp_ctl from linuxptp has turned the card’s stamping on, recvmsg returns the card’s receive stamp beside the data, and phc2sys keeps the card’s clock in step with the system’s, then validate the clock conversion, uncertainty and packet-to-message association. A hardware stamp may refer to a frame boundary different from this book’s last-bit convention; TCP, TLS and WebSocket boundaries do not correspond one-to-one. A TSC delta is not directly comparable to a NIC clock. See timestamping.