Polling, io_uring and runtime costs

Busy-polling

The way to skip the chain is never to sleep. The simple form makes the socket non-blocking, so that recv on an empty queue returns WouldBlock at once, and calls it in a loop on a core the thread has to itself: pinned there, and isolated so that the scheduler gives the core to nothing else.

Every empty pass is a system call, 100–300 ns, so the loop asks three to ten million times a second. What it buys is skipping the reader’s wake-up: no IPI, no C-state exit and no scheduler pass, caches warm through the quiet spell, and the data found within one pass of its arrival in the socket’s queue. It leaves the arrival processing intact. The packet still raises an interrupt, after whatever coalescing delay the card is set to, is walked by a softirq on the interrupt’s core, and crosses to the reader’s core as cache-line transfers. With coalescing off, packet in DRAM to bytes in the program’s buffer is 2–5 µs.

SO_BUSY_POLL takes the left-hand column too. It is a socket option, a number of microseconds, and it changes what recv does when the receive queue is empty. Each packet the kernel receives is tagged with the NAPI instance, one per receive queue, that polled it, and the socket remembers the last one. Under SO_BUSY_POLL, a recv that finds the queue empty calls that instance’s poll routine itself, in the thread’s own context on its own core, and checks the socket’s queue after each pass. A blocking recv keeps at it for up to the set number of microseconds and then sleeps; a non-blocking one makes one pass and returns. The packet is pulled rather than pushed: the poller finds it on the ring as soon as the DMA write lands, without waiting for the interrupt or the coalescing timer, and walks it on the reader’s own core, in its caches.

Two sysctls set the default for sockets that do not set the option: net.core.busy_read for recv and its kin, and net.core.busy_poll for select, poll and epoll_wait, which poll one receive queue at a time and so work best when the sockets they watch share one. The usual setting is 50 µs, long enough to cover the gap between packets on a busy feed and short enough that a blocking reader still sleeps through a lull. Raising a socket’s value above the sysctl needs CAP_NET_ADMIN. The driver must support busy polling through NAPI, as most server cards’ drivers have since the option arrived in kernel 3.11. The socket normally learns a NAPI association from received traffic, so that packet arrives the ordinary way. And the card’s interrupt still fires, so on a busy feed the softirq and the poller can race for the ring; SO_PREFER_BUSY_POLL, from kernel 5.11, together with the device’s interrupt-deferral settings, settles the race in the poller’s favour.

Either form costs a core. It runs at 100% whether packets come or not and never idles, so it draws full power and its hyperthread sibling runs slower; under SO_BUSY_POLL each empty pass also walks the driver’s ring.

io_uring

io_uring, added in kernel 5.1, is a different way of asking the kernel for anything. The program and the kernel share two rings in memory. The program writes requests into the submission ring (receive on this socket into this buffer, read this file at this offset), and the kernel writes results into the completion ring, each tagged with its request. One system call, io_uring_enter, submits a whole batch and can wait for completions in the same call. With SQPOLL set, a kernel thread polls the submission ring on a core of its own, so a program that also polls the completion ring, which lies in its own memory, talks to the kernel with no system call at all.

It helps where the system calls are the cost: many sockets or files with work on each, where one crossing carries a hundred requests, and file I/O, which otherwise blocks. It does not change how a packet arrives. The interrupt, the softirq, the protocol walk and the socket’s queue are as before, and a receive completes through the same data-ready hook that wakes a sleeping reader. For one hot socket, io_uring saves the spin’s 100–300 ns system call at the price of a hand-off through shared rings, lines crossing to the SQPOLL thread’s core, and a second core spinning. Recent kernels let a ring busy-poll the driver too, which brings it level with SO_BUSY_POLL and no further.

What a runtime adds

Tokio, the async runtime most Rust network code runs on, is built on mio and so on epoll. A task, an async function partway through, that awaits a read with nothing queued returns to the runtime and leaves a waker, a handle that will put it back on a run queue, with the socket’s registration. When no task is runnable, one worker thread sleeps in epoll_wait and the others sleep on a futex. When a packet arrives, the chain above wakes the first; it calls the wakers, and if a woken task goes to a worker that is asleep, waking that worker is a second wake-up. On top of epoll’s path the runtime adds a pass through its scheduler, its queues and task state in the caches, sometimes a move between cores, and timers with a resolution of one millisecond. None of that matters to a REST call, a log shipper or a control connection, where these costs are usually acceptable; on a hot path it is one more hop and one more cache footprint between the packet and the code.

What it costs

From packet in DRAM to bytes in the program’s buffer, a sleeping reader, in recv or in epoll_wait followed by recv, costs 5–20 µs with shallow C-states and 100–200 µs from a deep one; inside that, epoll_wait returning a ready event is about 0.5 µs and the recv and its copy 0.5–1 µs. A preempting thread on a shared core adds milliseconds, and a timeout up to 50 µs of slack. A spinning reader costs 2–5 µs and 100–300 ns per empty poll, and SO_BUSY_POLL takes the interrupt, the coalescing delay and the cross-core transfer out of those microseconds; either way the price is a whole core. io_uring removes system calls and nothing else, and Tokio adds a scheduler pass, and sometimes a second wake-up. Against the 1 µs budget, a spinning reader pays two to five budgets before the program has read a byte; a sleeping reader starts at five and goes to two hundred.

Worked example: the same packet, three ways

A 100-byte depth update arrives on a quiet feed, a few hundred updates a second, one every few milliseconds. Each time below runs from the last bit on the wire to the bytes in the feed thread’s buffer.

The first way is a blocking recv on an unpinned thread on a stock box: deep C-states allowed, adaptive coalescing, and irqbalance deciding where the card’s interrupt lands. Between updates the reader’s core idles for milliseconds and the governor takes it to C6. A typical update pays the card’s 1–2 µs, a short coalescing delay, since at this rate the adaptive scheme keeps it low, a few microseconds of softirq and then the C6 exit: a p50 of 100–200 µs, 100–200 budgets. When the interrupt’s core was asleep as well, or another thread holds the reader’s core, the update pays two exits or the rest of a slice: the p99 is several hundred microseconds and the max reaches milliseconds.

The second way is epoll_wait on a pinned core of a box with C-states capped at C1, as on the reference machine. The update pays the card’s 1–2 µs and the receive path’s 5–20 µs: a p50 of about 8–10 µs, eight to ten budgets, and a p99 of 20–40 µs, set by updates that find the interrupt’s core busy with other packets or a timer tick in the reader’s way.

The third way is a non-blocking recv loop with SO_BUSY_POLL at 50 µs, on an isolated core. The update pays the DMA write, a few hundred nanoseconds, and the next pass of the loop pulls it off the ring and walks it on the reader’s own core: a p50 of about 3 µs and a p99 of about 5–6 µs, three to six budgets. The tail is short because nothing in the path waits for another core.

The first two ways use the core only when an update arrives: a few hundred updates a second at 10–20 µs each is under 1% of it. The third uses all of it. At 300 updates a second and about 5 µs each, 1.5 ms of every second goes to updates and 998.5 ms to three to ten million empty polls. So 99.85% of the core’s 4 billion cycles a second buy the difference between 10 µs and 3 µs at the median, between 30 µs and 6 µs at the p99, and, against the stock box, between 3 µs and 150 µs for the first update after a lull.

Seeing it

strace -c -f -p <pid> counts a running program’s system calls by name, and each way of waiting has a signature. A blocking reader shows one recvfrom per update. An epoll reader shows one epoll_wait and two recvfrom per update, one returning the data and one returning WouldBlock, which strace counts in its errors column. A spinning reader shows millions of recvfrom a second, nearly all of them errors. strace stops the thread at every call for tens of microseconds, so its counts are right and its times are not; perf trace -s -p <pid> prints the same summary per thread, with the minimum, average and maximum time of each call, at a fraction of the overhead.

voluntary_ctxt_switches in /proc/<pid>/task/<tid>/status counts the times a thread gave up its core: one per update for a blocking or epoll reader on a quiet feed, none for a spinner. nonvoluntary_ctxt_switches beside it counts preemptions, which on an isolated core should also stay flat, and perf stat -e context-switches,cpu-migrations -t <tid> counts both over an interval, with migrations between cores, which for a pinned thread must be zero.

/proc/net/softnet_stat has no column of its own for busy polling, but its first column counts packets on the core where their protocol walk ran, so when SO_BUSY_POLL is working the count moves from the interrupt core’s row to the reader core’s row. nstat -az TcpExtBusyPollRxPackets counts the packets busy polling pulled directly and, despite its prefix, counts them for every protocol. If it stays at zero while the option is set, the driver does not support busy polling or the socket has not yet learned its ring.