Kernel-bypass stacks

4.9 Bypassing the kernel: what the stack gives, and when to do without it

Every packet in this chapter has passed through the kernel’s network stack, the code in the kernel that drives the card and implements Ethernet, IP, TCP and UDP for every program on the box. One choice is left: hand one program the card’s rings directly, run the protocol code inside that program’s own process, and take the kernel out of the path of its packets. That is kernel bypass, the largest single cut that can be made in the receive path and the most expensive, and whether it pays depends less on the program than on where the machine stands. This section is what the stack does that is worth keeping, three ways around it, what each takes away, what changes in a cloud, and the arithmetic that decides.

What the kernel stack gives

The stack is general on purpose, and its generality is most of what a program gets from it. It gives TCP: the handshake, fast retransmit and the retransmission timeout, flow and congestion control, and decades of fixes for what real peers and real networks do, from sequence numbers that wrap to address translators that rewrite headers and peers that advertise a zero window and go quiet. It gives routing, the table that says which interface and which next hop reach each destination, and ARP, which finds the next hop’s hardware address and keeps it fresh. It gives the firewall, netfilter, the hooks through which rules written with nftables or iptables filter, count and rewrite every packet. It gives observation: tcpdump reads packets from a tap on the kernel’s own path, and ss, nstat and the files under /proc/net read the kernel’s own structures. It shares one card among every process on the box, each finding its own packets through the socket lookup by 4-tuple, so a feed handler, an order gateway and ssh use one port without knowing of each other. And it has a driver for every card, so the same program runs unchanged on a server, in a virtual machine and on a laptop.

The price is the receive path traced earlier in the chapter, 5–20 µs of kernel with the reader asleep. Busy-polling the socket already removes the two parts that cost most and vary most, the interrupt’s delay and the wake-up, and brings the kernel’s part to 2–5 µs. What is left is the work itself, done by general-purpose code: an sk_buff built for every frame, the firewall’s hooks, the socket’s lock, a protocol implementation written for every case, and the copy out inside a system call. Bypass can take away only that remainder, bringing wire to user buffer from 3–7 µs to about 1 µs, and the price is the list in the paragraph above.

Three ways around it

All three give a program the packets in a receive ring without the kernel’s protocol walk. They differ in how much of the kernel they keep, and so in how much the program must do for itself.

DPDK, the Data Plane Development Kit, keeps none of it. The card is unbound from its kernel driver and bound instead to vfio-pci, a driver whose only job is to hand a whole device to one user process safely: it maps the card’s registers into the process, and programs the IOMMU, the unit that translates the addresses a device uses for DMA as the MMU translates a core’s, so that the card can write only into that process’s memory. The interface disappears from ip link. In the kernel driver’s place the program links a poll-mode driver, a user-space version of the card’s driver that fills the receive descriptors, reads them back when the card has used them, and never turns the card’s interrupts on. Packet buffers come from pools carved out of huge pages, 2 MB or 1 GB each, reserved at startup so that the buffers never move and a handful of TLB entries covers them all. Each receive queue gets a core of its own, running a loop that asks the driver for the next burst of packets, commonly up to 32, handles them and asks again, at 100% whether or not anything has arrived. There is no socket API. The program receives Ethernet frames and must answer ARP, check IP and run TCP itself, or link a user-space TCP built for DPDK, such as F-Stack, FreeBSD’s stack ported to it, or mTCP, from academic research. From the last bit on the wire to the frame in the program’s buffer is about 1 µs: the card’s DMA and the next pass of the loop, and nothing else. NVIDIA’s ConnectX cards are the usual exception to the unbinding: their poll-mode driver works beside the kernel’s, which keeps the interface and the traffic the program does not claim.

OpenOnload keeps the socket API and replaces the stack beneath it. It comes from a card vendor, Solarflare, now part of AMD, and has two halves. A kernel module gives a process a virtual interface on the card, a receive ring, a transmit ring and buffers of its own, mapped into the process’s memory, while the kernel’s driver keeps the rest of the card. A library, a TCP and UDP stack, runs inside the process, loaded with LD_PRELOAD, the dynamic linker’s variable that loads a named library ahead of all others, so that its socket, send, recv, epoll_wait and the rest stand in for the C library’s: an unmodified program started as onload ./gateway runs on a user-space stack. A recv on an accelerated socket polls the virtual interface’s ring in user mode, runs Onload’s TCP over whatever has arrived and copies the payload out, with no system call; EF_POLL_USEC sets how long a call spins before it sleeps. Under Onload sits ef_vi, the virtual interface’s own API with nothing on top: raw frames in and out, no TCP, the shortest path the card offers and the most work for the program. Both work only on that vendor’s cards, and Onload is the common choice at co-located trading firms, because it offers bypass to code written for sockets. A Rust program qualifies when it links glibc dynamically, as the usual build does; one linked statically, against musl or anything else, leaves LD_PRELOAD nothing to replace. Onload delivers about 1 µs from wire to user buffer with its TCP included, and ef_vi a little less.

AF_XDP skips the normal kernel protocol stack while retaining the kernel’s driver and XDP processing; it does not remove NAPI or interrupts by definition. See the AF_XDP documentation. XDP, the express data path, is a hook at the top of the driver’s receive routine: for each frame, before the kernel has allocated anything for it, the driver runs a small program written in eBPF, the kernel’s facility for running small programs that a verifier has proved safe and a compiler has turned into machine code. The program reads the frame and returns a verdict: pass it to the kernel, drop it, send it back out, or redirect it. AF_XDP is a socket type to redirect to. The process registers an area of its own memory, the UMEM, divided into frame-sized chunks, and four rings shared with the driver that pass chunk addresses back and forth: a fill ring of empty chunks for the driver to receive into, a receive ring of filled ones for the program, and a transmit ring and a completion ring for sending. Each socket is bound to one receive queue, and the XDP program redirects the chosen flow’s frames, those from the venue’s address or for the feed’s multicast group, into the socket’s ring and passes everything else to the kernel, so the rest of the box keeps working on the same port. Where the driver supports zero-copy mode the card DMAs straight into the UMEM; elsewhere, copy mode copies each frame once, which still skips the protocol walk. The frames arrive raw, Ethernet header first, so TCP is still the program’s to do. Zero-copy with a polling reader comes close to the others’ 1 µs; copy mode, or a reader woken by the interrupt, costs a few microseconds. In Rust, the aya crate writes and loads the XDP program and xsk-rs drives the socket.

What you lose

What bypass removes from the path, it removes from the box, for the bypassed traffic. Packets that never enter the kernel never pass tcpdump’s tap: with a dedicated DPDK port the application’s capture path sees them, unless it captures them itself or through dpdk-pdump, and Onload ships its own onload_tcpdump. A connection whose state lives in user space is not in ss, nstat or /proc/net/tcp. The firewall never sees the traffic, and a process that holds a card’s transmit ring can send any frame with any source address. DPDK takes the whole port on most cards, so the box needs another port for everything else; Onload and AF_XDP share. And portability goes: the code is tied to a vendor, a driver and often a kernel version, and no longer runs on a laptop or in continuous integration without a stand-in.

The kernel’s TCP is replaced by a stack that has met far fewer peers, and its bugs do not look like crashes. A stack that compares sequence numbers with a plain < instead of modulo 2^32 passes every test and stalls the first time the connection’s numbers wrap, which, since every connection starts from a random number, can be anywhere in its first 4 GiB: within 12 minutes on a 50 Mbit/s feed. A stack whose ARP entry is never refreshed goes on sending to the old router’s hardware address after a failover, into nothing. The kernel’s TCP lost bugs of this kind long ago to the traffic of millions of machines; a private stack meets them in production, with only the tools its authors built.