TCP and UDP delivery

4.5 Two kinds of delivery: what TCP promises, what UDP leaves to you

IP, the layer every packet travels on, promises nothing beyond trying. A router whose outgoing queue is full drops the next packet, a card that finds a bad FCS discards the frame, two packets that take different paths can arrive in either order, and nobody tells the sender. TCP builds, in the kernels at the two ends, a stream that is complete and in order out of packets that are neither, and pays in round trips, timers and waiting. UDP adds port numbers and a checksum and leaves the rest to the program. This section is what each promise is made of and charges, why classic exchanges and crypto venues chose differently, and what one lost packet costs under each.

What TCP promises

TCP promises five things, and each is a mechanism running in both kernels.

The first is a connection. The SYN, SYN-ACK and ACK exchange does more than say hello: each side picks a random initial sequence number, the point its count of bytes starts from, and the two agree on options such as the largest segment each will take and whether timestamps and selective acknowledgements will be used. From then on each kernel holds the state the other four promises are made of.

The second is order. Every byte of the stream has a sequence number, the initial number plus the byte’s offset, and a segment’s header carries the number of its first byte. The receiver uses it to put each segment’s bytes in their place, to discard a copy of bytes it already holds, and to set aside a segment that arrives after a gap, in the out-of-order queue, until the gap is filled.

The third is reliable delivery while the connection remains viable, or a reported connection failure. Every segment a receiver sends carries an acknowledgement number, the sequence number of the next byte it expects, which says that every byte before it has arrived; because that number is cumulative, the receiver adds a selective acknowledgement, SACK, listing the ranges it holds beyond a hole. The sender keeps every byte in its send queue until it is acknowledged, and runs a timer on the oldest. When a segment arrives after a gap, the receiver acknowledges at once, repeating the same acknowledgement number: a duplicate ACK. Three duplicates tell the sender that three later segments got through and one did not, and it resends the missing one at once: fast retransmit. If the lost segment was among the last of a burst, too few later segments arrive to make three duplicates, so after about two round trips of silence the sender resends its last segment to provoke an acknowledgement: a tail-loss probe. If neither works, the timer fires and the sender resends on the retransmission timeout.

The fourth is flow control: the receive window bounds outstanding data, so a sender can burst faster than the reader temporarily, but cannot grow that backlog without limit. Every segment carries its sender’s receive window, the free space in its receive buffer, and the other side keeps no more unacknowledged bytes in flight than that, so a reader that stops reading stops the sender instead of losing data, and the backlog moves into the sender’s send queue.

The fifth is congestion control: the sender estimates available capacity and adjusts its window or pacing; it can still overrun the path and lose packets. A router has no window to announce, so the sender keeps an estimate of its own, the congestion window, and keeps no more in flight than the smaller window allows. A new connection starts at 10 segments, about 14 KB, and doubles the window every round trip as acknowledgements return: slow start, slow only next to sending everything at once. A loss is taken as a sign of a full queue somewhere, and the sender cuts the window: Linux’s default algorithm, CUBIC, cuts it to 70% after a fast retransmit and to one segment after a timeout, and grows it again from there.

Read more: W. Richard Stevens and Kevin Fall’s TCP/IP Illustrated, Volume 1 walks through each of these mechanisms with packet traces; RFC 9293 is the current specification of TCP, RFC 6298 its retransmission timer and RFC 5681 its congestion control.

What each promise charges

The connection charges a round trip before the first byte: 70 ms to a venue 35 ms away, before TLS adds its own. If the SYN itself is lost, the sender has no measured round trip to size a timer on, and waits a full second before trying again.

The acknowledgements charge traffic in the other direction. A receiver acknowledges at least every second full-sized segment, and on a stream of small segments read promptly it often acknowledges each one: on a feed of 2,000 segments a second, 1,000–2,000 frames of 70 bytes a second leaving our box. On an order connection the direction reverses, and each acknowledgement from the venue is an interrupt and a protocol walk in our kernel whose only effect is to free the order’s bytes from the send queue.

Recovery charges round trips. In the simplified fast-retransmit example, three later segments trigger duplicate ACKs and repair adds about one round trip plus their spacing. Tail-loss probes and newer loss-detection algorithms can change that timing. A retransmission timeout depends on smoothed RTT and its variation, subject to configured limits; it is not “minimum plus RTT”. A common Linux minimum is 200 ms, so a stable 70 ms path can still show an RTO near 200 ms rather than necessarily 270 ms. The current Linux controls include route, socket and sysctl overrides; RFC 6298 defines the estimator. Recovery on a market-data connection uses the venue sender’s settings, not ours.

Order charges waiting. The receiver cannot hand the program any byte beyond a hole, because the stream promised order. Every segment behind a lost one waits in the out-of-order queue, recv finds nothing and epoll reports nothing: to the program the connection looks exactly like a quiet market. When the missing segment arrives, everything behind it becomes readable at once. This is head-of-line blocking, named for a queue whose head holds up everything behind it: one lost segment delays every message the connection carries, so a connection carrying fifty instruments stalls all fifty for a segment that concerned one, and TCP has no way to skip the hole.

The stream also charges framing: TCP keeps no record of where one write ended and the next began, so the protocol on top must mark its messages and the program must rejoin those that arrive in pieces.

Congestion control charges slow starts, and not only at the opening. Linux by default treats a connection that has been quiet for longer than its retransmission timeout as if the path might have changed, and cuts its window back towards 10 segments. A burst of 500 updates of 250 bytes after a second’s lull, 125 KB or 87 full segments, then leaves from a window of 10 in four round trips, 10, 20, 40 and 17, and on a 70 ms path the last update arrives 210 ms after the first. The setting that decides it is, again, the venue’s.

What UDP charges and leaves to you

UDP’s header is 8 bytes: two ports, a length and a checksum. A datagram is the bytes of one send_to, sent as one packet when they fit and delivered whole or not at all, so one send is one receive. There is no setup, no acknowledgement, no retransmit, since the kernel forgets a datagram once the card has sent it, and no order. There is no flow or congestion control either: a full queue in the network drops what it cannot hold, and a receiving socket whose queue is full, 208 KB by default, drops new arrivals and counts them. The receive path is shorter for the same reasons: no sequence check, no acknowledgement to build, no out-of-order queue, no timer to re-arm.

What UDP does not do, the program does. Loss is the program’s to notice, which it can do only if the sender numbers its datagrams. Duplicates are the program’s to discard, rare from the network and routine when the same data is sent twice on purpose. Reordering is the program’s to undo or to tolerate, and several messages packed into one datagram need lengths of their own.

Size is the limit the program must respect. With plain IPv4 headers, a UDP datagram can carry up to 65,507 payload bytes, but one packet on a path with a 1,500-byte MTU carries at most 1,500 − 20 − 8 = 1,472. A larger datagram may be fragmented when IPv4 and socket policy allow it, or a path-MTU check may reject the send with EMSGSIZE. Where fragmentation occurs, IP cuts it into several packets, of which only the first carries the UDP header and its ports, and losing any piece loses the datagram, so a datagram cut in three is lost about three times as often as one that fits. Market-data protocols therefore keep every datagram inside one frame, packing several messages into it up to about 1,400 bytes.