NIC rings and receive processing

4.3 Anatomy of a receive: from the wire to read returning

When the last bit of a frame reaches the card, its bytes are still several hand-offs from the program that wants them. The card must put them somewhere and say so; the kernel must notice, check every envelope, find the socket and queue the bytes there; a sleeping reader must be woken, and its own system call must copy the bytes out. Each hand-off exists for a reason that has nothing to do with trading: many programs share one card, and an idle machine should not burn a core watching for packets. This section follows one TCP segment from the cable to read returning and prices each step.

The ring and the descriptor

The card and the kernel meet in a receive ring: an array of descriptors in DRAM, used in a circle, one ring of a few hundred to a few thousand descriptors per receive queue. A descriptor, 16–64 bytes, is written twice. Before a frame arrives, the driver writes into it the address of an empty buffer, usually 2 KB; this is posting the buffer. After a frame arrives, the card overwrites it with what it learned: the frame’s length, a done bit, the hash of the 4-tuple, whether the checksums were good and, if asked, the arrival time by the card’s own clock.

The driver hands posted descriptors to the card by writing the ring’s new tail index to the card’s doorbell register, a store that the processor routes across PCIe instead of to DRAM and that the core does not wait for. Reading a card register would stall the core for a PCIe round trip, most of a microsecond, so the driver never reads the card’s position; it reads the done bit in the descriptor, in ordinary memory.

When a frame arrives, the card checks its FCS and destination MAC address, hashes the 4-tuple to choose a queue, writes the frame by DMA into that queue’s next posted buffer, and then writes the descriptor back with the done bit set. The NIC’s descriptor protocol and DMA ordering rules let a driver consume the frame after observing completion with the required DMA read barrier; a plain unchecked load of a done bit is not a portable substitute. On Intel’s server processors a feature called DDIO steers these writes into a portion of the L3, so the kernel’s first read of the headers hits there; elsewhere it misses to DRAM, about 90 ns. If the kernel falls so far behind that the card reaches the tail, the next frame has nowhere to go, and the card drops it and counts a miss: the ring’s size is how long a burst can wait for the kernel.

Read more: Joe Damato’s “Monitoring and Tuning the Linux Networking Stack: Receiving Data” (packagecloud, 2016) follows this path through the kernel source function by function, and the kernel’s Documentation/networking/scaling.rst describes how packets are spread across queues and cores.

The interrupt and the poll

With the frame in memory, the card tells a core. A server card uses MSI-X, message-signalled interrupts: it writes a small value across PCIe to a reserved address, and the processor turns that write into an interrupt on the core the message names. The card has one such vector per queue, each with its own affinity, so which core hears about a segment is decided twice, and never by the program: the card’s hash picks the queue, and the queue’s affinity, set by irqbalance on a stock box, picks the core. The interrupt lands a few hundred nanoseconds after the descriptor unless the card is coalescing, holding it for rx-usecs microseconds so that one interrupt announces several frames; with adaptive coalescing, on by default in many drivers, a stock box adds anything from nothing to 100 µs here.

The top half masks the queue’s interrupt, puts the queue’s NAPI context, the driver’s per-queue polling state, on this core’s list to poll, and raises the NET_RX softirq, which runs as the interrupt returns. The softirq calls the driver’s poll function, which takes up to 64 frames a call. For each descriptor with its done bit set, the poll builds an sk_buff, the kernel’s record of one packet on its way through the stack: about 230 bytes of metadata, nearly four cache lines, holding pointers into the buffer, the offset of each header, the length, the checksum verdict, the arrival time and, once known, the socket. Stripping a header moves a pointer rather than copying bytes. A poll that finds fewer than 64 frames has emptied the ring and unmasks the interrupt; one that takes all 64 is called again; and after 300 frames or about 2 ms the softirq leaves the rest to ksoftirqd, where the network’s work waits on the run queue like any thread’s.

On the way out of the poll each sk_buff is offered to generic receive offload, GRO, which keeps one packet under construction per flow for the length of a poll and merges into it each full-sized segment that continues it: the same 4-tuple, the next sequence number, the same options. A short segment, or one carrying PSH, a flag most senders set on the last segment of each write, closes the merge and sends the packet up. For a bulk transfer the protocol walk then runs once per 45 segments instead of 45 times, at the price that the first segment of a burst waits while those behind it are merged. A feed of small updates gets neither the gain nor the delay: each update goes up alone and pays only for GRO’s lookup.

The walk up the stack

From GRO the packet enters netif_receive_skb, the door to the protocol code. Packet taps see it first: a running tcpdump gets its copy here, which is why it shows GRO’s merged packets rather than the frames on the wire, and why its timestamps come after the card, the interrupt and the poll. Ethernet’s part is one field: the driver has already read the EtherType, so the stack calls the handler for 0x0800, IPv4.

IP checks the header’s version, lengths and 16-bit checksum, a few nanoseconds over 20 bytes. The firewall’s hooks run here, and on a box with a stateful firewall its table of connections costs a second hash lookup per packet. Routing then decides whether the packet is for this machine, and for TCP the answer comes early: a step called early demultiplexing finds the established socket by its 4-tuple, a hash lookup of tens of nanoseconds, and reuses the route cached on it. Reassembly of fragments, the pieces IP cuts a packet into when it exceeds a link’s MTU, is in the common case a check that two fields are zero, because TCP sizes its segments to fit the path.

TCP starts with its socket found and its checksum already checked by the card. If a thread holds the socket’s lock at that moment, usually a reader inside recv on this very socket, the segment waits on the socket’s backlog until the lock is released. The heart of the work is the sequence check. The socket knows the next byte it expects, and the common segment starts exactly there, so a fast path of a few comparisons accepts it. A segment that starts beyond it means something before it was lost or reordered: it goes into the out-of-order queue, a tree sorted by sequence number, and the socket at once repeats its last acknowledgement, listing the ranges it holds beyond the hole. When the hole is filled, everything contiguous behind it moves to the receive queue together. The segment’s own acknowledgement number is processed too, freeing whatever this socket had sent that the far side now confirms.

Two decisions and an append end the walk. The first is whether to acknowledge at once, as the socket does for data out of order or for more than one full-sized segment since its last acknowledgement, or to arm the delayed-ACK timer in the hope that a reply will soon carry the acknowledgement for free. The second is whether the socket has room under its receive-buffer limit; if not, the segment is dropped, to be sent again. Then the data is appended to the receive queue and the socket’s data-ready callback runs. For a lone packet the walk takes 1–2 µs, much of it cache misses on the socket’s own state; in a batch, with the code and the socket warm, 0.5–1 µs a packet.

All of this runs on the core that took the interrupt unless the kernel is told otherwise. Receive packet steering, RPS, repeats the card’s spreading in software: it hashes the flow, picks a core from a set in /sys/class/net/<iface>/queues/rx-<n>/rps_cpus, puts the sk_buff on that core’s input queue, its backlog, and sends that core an inter-processor interrupt, an IPI, to make it run the walk. Receive flow steering, RFS, picks the core where the socket’s reader last called recv, so that the walk runs beside the reader’s caches. Both are off by default and cost an IPI and a queue hop, a few microseconds.