Network latency and Nagle labs

4.11 Lab

Rules for every measurement

The rules of the earlier labs stand: predict first, release build, warm up, pin, performance governor, three runs, and p50, p99 and max, never a mean. This chapter adds four. Record the network as well as the cores: the card’s driver, its version and its PCIe address from ethtool -i <iface>, its model from lspci -s at that address, its coalescing settings from ethtool -c, the core that takes the test queue’s interrupts, which is the cell of /proc/interrupts that moved during the run, and the kernel from uname -r. For every two-machine run, record the number of switches between the boxes and the p50 of a thousand pings, taken from the per-packet times, since the summary line gives no median. Run the two-machine labs with the hot threads pinned and irqbalance stopped on both machines, and say so in the write-up: a prepared box timed against a stock one times the stock one. And count every variant’s system calls with strace -c, in a run of its own: strace stops the thread at each call for tens of microseconds, so its counts are right and its times are not. A spinning variant’s empty polls, whose number depends on how long each takes, are better counted with perf stat -e syscalls:sys_enter_recvfrom.

Lab 4.1 — the ladder (ch04-network/src/bin/ladder.rs)

The client sends a one-byte message and waits for a one-byte reply, a million times after a warm-up of 100,000, timing each round trip with rdtsc on its own side only, and prints p50, p99 and max. In the local modes it starts the echo side itself, on a second physical core that shares its L3. --threads is Chapter 3’s ping-pong, here for scale: two pinned threads and one padded AtomicU64. --shm runs the same protocol between two processes, each spinning on a padded word in a page created with memfd_create and mapped shared with mmap before a fork. --unix uses a connected pair of Unix domain sockets and --tcp-loopback a TCP connection to 127.0.0.1, both with blocking reads. --tcp <host> crosses to the second machine, where the same binary runs as --echo, with blocking reads at both ends. Adding --spin at both ends makes the sockets non-blocking and has each side loop on recv on its pinned core; adding --busy-poll instead does the same with SO_BUSY_POLL at 50 µs on both sockets. Every TCP mode sets TCP_NODELAY, so that the ladder measures the path and not Nagle. Where both machines have cards that Onload supports, the bypass rung is the unmodified --tcp run, started as onload --profile=latency ./ladder --tcp <host> against onload --profile=latency ./ladder --echo.

Reason it out. --threads moves one cache line each way at 50–80 ns a hop: 100–160 ns a round trip, 400–640 cycles at 4 GHz. --shm moves the same line, and the kernel, having built the mapping, stays out of the loop: 100–200 ns, the same within noise. --unix brings in a system call and a copy each way and a wake-up at each end, though no protocol: 5–15 µs, 20,000–60,000 cycles, 25–150 times --shm. --tcp-loopback adds TCP’s send path and protocol walk, run in software with no card: 10–30 µs, about twice --unix. --tcp <host> adds the cards, the switch and an interrupt at each end. The one-byte payload makes a 71-byte frame, 57 ns on the wire at 10 GbE, too little to see; each way costs a send of 2–5 µs, under a microsecond through the switch and 6–22 µs from the wire to recv returning in a sleeping reader, about 17–55 µs a round trip, the ladder’s 20–50 µs and twice loopback. --spin removes both wake-ups and most of the variation with them, and --busy-poll also moves the driver’s poll and the protocol walk onto the reader’s own core. The ladder puts both ends busy-polling at 8–15 µs, 32,000–60,000 cycles: expect --busy-poll in the lower half of that and --spin a few microseconds above it, since --spin still waits for the interrupt, the softirq on another core and the socket’s lines to cross. Bypass should land at 3–6 µs, 12,000–24,000 cycles, one and a half to three times the hardware floor of about 2 µs. The ratios are the result: about 1 from threads to processes, 25–150 up to a Unix socket, 2 up to loopback, 2 again to the second machine, then about 3 down by busy-polling and 2.5 more by bypass. Expect the blocking rungs’ p99 far above their p50, because the wake-up varies more than the rest of the path together, and the spinning rungs’ p99 within a few microseconds of theirs.

Pitfalls. Read rdtsc on the client only, so that both ends of every interval come from one clock, and keep the two sides off sibling hyperthreads, which share an L1 and flatter --threads and --shm. Setting SO_BUSY_POLL above net.core.busy_read, which is 0 by default, needs CAP_NET_ADMIN, so check what the setsockopt returns. Then check that busy polling happens at all: nstat -az TcpExtBusyPollRxPackets should rise by about one a round trip at each end, and on a driver without support it stays at zero and --busy-poll measures --spin. Steer the test queue’s interrupt, on both machines, to a core that shares the hot thread’s L3 for the blocking and --spin runs and onto the hot thread’s own core for --busy-poll, and turn coalescing off on both cards, or every rung but --busy-poll pays rx-usecs twice a round trip. Under Onload, onload_stackdump must list the binary’s stack while it runs, and strace -c should find almost no calls in the loop, since Onload’s recv stays in user space; a binary linked statically against musl ignores LD_PRELOAD and measures the kernel instead. Write down p50, p99 and max for every rung, the p50 in cycles and its ratio to the rung before, and the system calls per round trip: none for --threads and --shm, a send and a receive a side for the blocking modes, and for the spinning modes a send, one receive that returns data and tens to a hundred that return EAGAIN.

Lab 4.2 — Nagle (ch04-network/src/bin/nagle.rs)

A request is 48 bytes: a 4-byte header holding the length, then a 44-byte body. The echo server, run as --echo on the second machine or forked by the client with --loopback, reads until it holds a whole request and writes it back in one call; like a venue, it never answers half a message. The client sends 1,000 requests after 100 of warm-up, each as two write_all calls, header then body, waits for each echo before sending the next, and prints the total time and the p50, p99 and max per request. --nodelay sets TCP_NODELAY on the client’s socket. --one-write gathers header and body with write_vectored, handling short writes until all 48 bytes are accepted. --cork sets TCP_CORK before the header and clears it after the body. Run the default and each flag over --loopback and with --host <host>.

Reason it out. By default the header leaves at once, since nothing is in flight, and Nagle holds the body, since the header is unacknowledged. The server holds half a request and has no reply to carry an acknowledgement, so it waits out its 40 ms delayed-ACK timer. If a capture confirms that interaction for every request, each costs about 40 ms and the 1,000 about 40 s, on loopback and across the switch alike, because the timer sets the number and the path adds only tens of microseconds to it. With any of the fixes a request is one ordinary round trip, 10–30 µs on loopback and 20–50 µs across the switch, and the 1,000 take 10–50 ms, eight hundred to four thousand times less. --one-write needs no option because Nagle holds a small segment only while an earlier one is unacknowledged. In a request-and-reply protocol the previous echo has acknowledged every byte the client sent, so a request written whole finds nothing in flight and leaves at once: the stall takes two writes per message. --nodelay sends two segments about a microsecond apart, or a few more if autocorking holds the body for the header’s transmit completion. A server that wakes on the header alone reads 4 bytes and sleeps again, so expect a p50 a microsecond or two above --one-write’s and a p99 up to a wake-up above it. --cork sends the same single segment as --one-write for three more system calls, two setsockopt and a second send, a microsecond or two a request.

Pitfalls. An echo server that writes back whatever each read returns answers the header at once, its reply carries the acknowledgement, and the stall vanishes: whether Nagle bites depends on the far side as much as on the near one. A new connection acknowledges every segment while it is in quick-ack mode, so the first few requests may be fast even in the default run. Quick ACKs, autocorking or merging can keep later requests fast too; warm-up does not guarantee a 40 ms stall. Confirm two separate segments and the delayed ACK in a capture. TCP_QUICKACK set on the server after every read also removes the stall, which shows whose timer it is, and no venue will set it for anyone. A cork left set on an error path holds the next request for 200 ms. tcpdump -ttt shows the stall’s signature: a 4-byte segment from the client, an empty acknowledgement from the server about 0.040 s later, and the body just after. ss -ti on the server’s socket shows ato:40, and the server’s TcpExtDelayedACKs rises by about one a request. Write down the total and the p50, p99 and max per request for each variant on each path; the system calls per request, which Rust’s TcpStream makes as sendto, writev and recvfrom: two sendto and one recvfrom for the default and --nodelay, one writev and one recvfrom for --one-write, and two setsockopt, two sendto and one recvfrom for --cork; the segments per request from tcpdump; and the changes in TcpExtDelayedACKs and TcpExtTCPAutoCorking.