The TCP send path

4.4 Anatomy of a send: from write to the wire

Sending is the receive path run backwards: the program’s bytes are copied into the kernel, wrapped in TCP, IP and Ethernet headers, queued, handed to the card and put on the wire. The difference is that on the send side TCP may hold bytes back, for an acknowledgement, for more bytes to join them, for room in a window, and the program is not told. One of those holds, meeting a hold on the far side, makes the longest ordinary stall in this chapter. This section follows a small write to the wire, then the rule that holds it, then the options that change the rule.

The send buffer

A write or send on a TCP socket enters the kernel, takes the socket’s lock and copies the program’s bytes into the socket’s send queue: onto the last sk_buff there if it has room and has not been sent, into a new one otherwise. The bytes count against the socket’s send-buffer limit, which Linux starts at 16 KB and grows up to 4 MB by default. Then, in the same call and on the same thread, the kernel pushes as much of the queue to the transmit path as the rules allow, releases the lock and returns the byte count. If the buffer is full, write sleeps until acknowledgements free some space, or, on a non-blocking socket, copies what fits and returns a short count, or WouldBlock if nothing fits.

“Sent” therefore means three things, and the program sees only the first. To the program, write has returned and its buffer may be reused. To the wire, the first bit has left the card, about a microsecond after write returns when nothing holds the bytes, any time later when something does. To the far side, the bytes have arrived when its acknowledgement comes back, and even then they are in the far kernel, not yet in the far program. Between write returning and the bytes leaving, five things can hold them, and the return value reports none: the congestion window, cwnd, the number of unacknowledged segments the sender allows itself, which starts at 10; the receive window the far side last announced, zero if its program has stopped reading; Nagle’s algorithm; corking, by the program or by the kernel; and a queue in front of the card.

Read more: Joe Damato’s companion piece, “Monitoring and Tuning the Linux Networking Stack: Sending Data” (packagecloud, 2017), follows this path through the source; John Nagle’s RFC 896 (1984) is the short original statement of his rule, and RFC 1122 (1989), the requirements for internet hosts, is where the delayed acknowledgement is allowed and bounded.

Segments, qdisc and the ring

The push cuts the queue into segments no larger than the maximum segment size, the MSS, agreed when the connection opened: 1,448 bytes on a 1,500-byte MTU with timestamps. A 120-byte write is one segment. The kernel clones the sk_buff, keeping the original in the queue for a possible resend and sending the clone down, and writes the TCP header in front of the data, including the acknowledgement number for the other direction, since every segment acknowledges what its sender has received so far; the checksum is left to the card. IP adds its 20 bytes and takes the route cached on the socket, the firewall’s output hooks run, and the Ethernet header is written with the next hop’s MAC address from the neighbour table.

A large write is cut later. The kernel builds one sk_buff of up to 64 KB with one set of headers and lets the card cut it into about 45 segments, copying the headers onto each and fixing each one’s sequence number and checksums: TCP segmentation offload, TSO. Generic segmentation offload, GSO, makes the same cut in software just before the driver, for a card that cannot. Both exist for throughput, and a 120-byte message passes through them untouched.

Between the protocol and the driver sits the queueing discipline, the qdisc: the kernel’s queue in front of each of the card’s transmit queues, where packets wait when the card’s ring is full and where a policy picks which goes next. The old default, pfifo_fast, is first in, first out by priority band; many distributions now default to fq_codel, which keeps a queue per flow and serves the flows in turn. An empty qdisc is passed straight through, for a lock and a few hundred nanoseconds. One that a bulk flow has filled makes a small order wait, behind everything queued under pfifo_fast or behind one turn of each busy flow under fq_codel. Transmit packet steering, XPS, maps each sending core to a transmit queue of its own, so that cores do not contend for one qdisc’s lock.

The driver writes transmit descriptors, each the address and length of a piece of the frame with a flag asking the card for the checksum, into the TX ring, and rings the doorbell with the new tail; when the stack says more packets follow at once, it rings only for the last. The card fetches the descriptor across PCIe and then the frame, two DMA reads that each wait a PCIe round trip of a few hundred nanoseconds, about a microsecond together. It computes the checksum, appends the FCS and serialises the frame: 152 ns at 10 GbE for the 190-byte frame of a 120-byte order. Then it marks the descriptor done and, after its own coalescing, interrupts, and the driver frees the clone.

The original leaves the send queue when the far side’s acknowledgement arrives, by the whole receive path, which also takes a sample of the round trip and re-arms or cancels the retransmission timer. A small write that nothing holds takes 2–5 µs from write to the first bit on the wire: 1–4 µs inside the call for the system call, the copy, the segment and its headers, the qdisc, the descriptor and the doorbell, and about a microsecond after it while the card fetches.