From the socket queue to your code

The wake-up and the copy

The data-ready callback ends by making the sleeping reader runnable, whether it sleeps in recv itself or in epoll_wait, the call that sleeps until any of a set of sockets is ready. From there it is Chapter 2’s wake-up: an IPI to the reader’s core if the walk ran elsewhere, the core’s exit from its C-state and a pass of the scheduler, into caches that the sleep and the kernel’s work have cooled. Counted apart from the card, the interrupt and the walk, the wake-up’s own share is 2–15 µs from a shallow C-state and 100–200 µs from a deep one.

The thread then calls recv. The kernel takes the socket’s lock, copies bytes from the sk_buffs at the head of the receive queue into the program’s buffer, frees each one it empties, and, if reading opened the window a lot, sends a window update, an acknowledgement whose news is that there is room again. It releases the lock, processing anything parked on the backlog, and returns the count. For 100 bytes the copy is two cache lines and the whole call 0.5–1 µs. If the walk ran on another core, every line the copy touches was last written there, and each crosses at 50–100 ns.

A reader that never sleeps, spinning on a non-blocking recv on a core of its own or busy-polling with SO_BUSY_POLL, skips the wake-up, and the kernel’s part falls from 5–20 µs to 2–5 µs. The comparison counts from the last bit on the wire.

For a sleeping reader, insert the wake-up between queueing data and copying it. A spinning reader removes that step. These are additive example stages, not a fixed per-packet timeline.

What it costs

For the illustrative setup, from the wire to recv returning, a receive costs 6–22 µs with a sleeping reader, 24,000–88,000 cycles at 4 GHz and six to twenty-two budgets, and 3–7 µs with a spinning one; the card’s 1–2 µs is in both. Two additions set the tail, and they move with traffic in opposite directions. Coalescing adds whatever rx-usecs is at the moment, up to 100 µs on a stock box, and adaptive coalescing lengthens it as load rises. A deep C-state makes the wake-up 100–200 µs, and the core goes deeper as load falls.

The indirect cost falls on the core that takes the interrupt: 2–3 µs per lone packet for the top half, the poll and the walk, less per packet in a batch, and tens of kilobytes of kernel code and data pulled through its L1 and L2, so a hot thread on that core pays twice, in the microseconds taken and in cache misses afterwards.

The path is also a queue at three points: the last frame of a full poll waits behind 63 walks, 30–60 µs; past 300 frames, the rest waits for ksoftirqd to be scheduled; past the ring’s size, the card drops frames, and TCP’s recovery of each costs a round trip or more.

Worked example: a depth update arrives

A venue’s market-data gateway sits in the same rack as the trading box, one switch away, and sends depth updates over TCP. An update is 100 bytes of payload, so its frame is 14 + 20 + 32 + 100 = 166 bytes, 170 with the FCS: 136 ns on the wire at 10 GbE. The strategy thread reads the socket itself and has the 1 µs budget to decide what each update means.

In the first arrangement, a stock box, the strategy thread waits in epoll_wait on core 4. The card’s hash puts the connection on queue 3, whose interrupt irqbalance has sent to core 9. The update reaches memory and, if coalescing holds nothing at that moment, interrupts core 9 within 1–2 µs; core 9 runs the poll and the walk and wakes the strategy thread with an IPI to core 4. In a busy market, with updates every 20–50 µs, core 4 never gets past a shallow C-state, and recv returns 6–22 µs after the last bit arrived: of the 7–23 µs from the wire to the decision, the decision is 1 µs. In a quiet market, at 200 updates a second, core 4 sinks to its deepest state between updates and the wake-up alone is 100–200 µs, so the update that ends a lull, often the first sign of a move, is the slowest of all. If adaptive coalescing has settled on 50 µs under the morning’s load, add 50 µs to either case.

In the second arrangement the strategy thread is pinned to core 4 and calls recv on a non-blocking socket in a loop, and queue 3’s interrupt is steered to core 5, which shares core 4’s L3. Core 5 runs the poll and the walk, and the next recv on core 4 finds the bytes at most 100–300 ns later, the socket’s state and the payload crossing in a few line transfers. recv returns 3–7 µs after the last bit, in a quiet market as in a busy one, and the update has been acted on 4–8 µs after it arrived, at the price of core 4 at 100% for good. The spin removed the largest and most variable piece of the path; the card, the interrupt, the poll, the walk and the copy remain, three to seven budgets before the strategy starts.

Both arrangements also share the burst: when the market moves, 30 updates can arrive within 4 µs of wire time, one poll takes them all, and the thirtieth waits behind 29 walks, 15–30 µs, before any reader can see it.

Seeing it

/proc/interrupts has a row for each of the card’s queues, named by the driver, such as ens1f0-TxRx-3 on many Intel drivers or mlx5_comp3 on Mellanox ones. The row counts interrupts, not packets, so its rate set against the queue’s packet rate from ethtool -S <iface> is the batching that NAPI and coalescing achieve: one interrupt per packet on a quiet feed, one per dozens in a burst. ethtool -S also counts frames lost to a full ring, as rx_missed_errors or rx_out_of_buffer depending on the driver.

/proc/net/softnet_stat has a line per core of hexadecimal counters for the NET_RX softirq. The first column counts frames processed on that core; the second, frames dropped because the core’s backlog was full; the third, time squeezes, the softirq stopping at its 300-frame or 2 ms limit with work left over, so a rising third column means a receive-processing budget was exhausted, not necessarily that packets were dropped. The budgets are configurable.

perf trace -e 'net:*' prints the network tracepoints as they fire, with a timestamp and the core: net:netif_receive_skb as a packet enters the protocol code, among others; adding -e napi:napi_poll prints each poll with the frames it took against its limit of 64. bpftrace can aggregate packet lengths per core at the same tracepoint with @[cpu] = hist(args->len), after checking the tracepoint’s fields on the installed kernel.

The cores reported at that tracepoint run that part of the stack. On a 1,500-byte MTU, lengths above a normal frame can indicate GRO aggregation; jumbo frames and the tracepoint’s length definition must also be considered. Counting tracepoint:tcp:tcp_rcv_space_adjust, which fires inside a recv that consumes data, by core in the same way, with @[cpu] = count(), shows the cores the copies run on; if the sets differ, every update crosses between cores before the program sees it.

Software timing starts where software first sees a packet; NIC hardware timestamps can include earlier receive stages. ethtool -T <iface> reports capabilities and the PTP hardware clock. Enable hardware receive timestamping on the device and socket, then verify which stamps recvmsg actually returns for the target transport. Align the NIC and system clock domains with measured uncertainty, and define how a packet stamp maps to a complete application message. TCP aggregation, TLS records and WebSocket framing can prevent a simple one-stamp-per-message interpretation. Captures using a raw adapter clock preserve spacing within that domain, but their values cannot simply be subtracted from TSC ticks or Instant values.