Frames, headers and distance

What sits beneath

Under the socket, the path to the far program is a chain of devices, each with one job. The network interface card, the NIC, is a device on the PCIe bus, the bus that connects devices to the processor, with logic and a little memory of its own. It turns frames in DRAM into signals on the cable and signals on the cable into frames in DRAM, and it reaches DRAM itself, by direct memory access, without a core doing the copying. To send, the kernel writes a descriptor, the address and length of a frame, into a ring of descriptors in DRAM, then writes to a register on the card, the doorbell, to say new work is waiting; the card reads the descriptor and then the frame across PCIe and puts the bytes on the wire. To receive, the card writes the frame across PCIe into a buffer the kernel set aside in a receive ring, and raises an interrupt. Each crossing of PCIe costs a few hundred nanoseconds. The cable is copper for a few metres inside a rack and fibre beyond, with a transceiver at each end turning electrical signals into light and back. A switch joins the machines of one local network: it reads each frame’s destination hardware address, the MAC address the far card was built with, and sends the frame out of the port where it has seen that address before. A router joins networks: it reads the IP destination, consults its routing table, and sends the packet on towards the next network. In a modern data centre many boxes do both. At the far end, a card and a kernel do the receiving half.

Follow the send path from the program through TCP, IP, Ethernet and the NIC, then across the switch and back up the receive path. On the sending side each layer adds one envelope, the outermost last: the program writes the payload, the kernel puts a TCP header in front of it, then an IP header, then an Ethernet header, and the card appends the frame check sequence, the FCS, a 4-byte checksum over the whole frame, as the bytes leave. The switch looks only at the outermost envelope, and only at its first field. The receiving side takes the envelopes off in the opposite order: the card checks the FCS and drops it, and the kernel checks and strips Ethernet, IP and TCP in turn before the payload reaches the program. Across a switch every envelope travels unchanged. If the two machines sat on different networks, a router would stand where the switch is: it would remove the Ethernet envelope and write a new one addressed to the next hop, and change two fields of the IP header, lowering the time-to-live, a count of how many more routers the packet may cross, by one, and fixing the checksum to match, leaving TCP and the payload untouched. The sending kernel learns the next hop’s hardware address, the far machine’s own on one network or the router’s otherwise, by asking the network once with ARP, the address resolution protocol, and keeps the answer in its neighbour table, which ip neigh prints.

The envelopes, in bytes

Chapter 0 named the four envelopes. This is what each costs in bytes, what its fields decide, and where it comes off.

Take a 48-byte depth update sent over TCP. With plain headers the frame is 14 + 20 + 20 + 48 = 102 bytes. Linux sends the timestamps option on every segment by default, 12 bytes that let each side measure the round trip and recognise old segments, so the TCP header is 32 bytes, the IP packet 100 and the frame 114, or 118 with the FCS. Over UDP the same message is 14 + 20 + 8 + 48 = 90 bytes, 94 with the FCS. Over TCP the message is about 40% of what the card sends, over UDP about half. At the other extreme, Ethernet’s minimum frame is 64 bytes, FCS included, a rule made so that a sender on the old shared cables was still transmitting when news of a collision reached it, and kept after the shared cables went. A UDP payload shorter than 18 bytes is padded to reach it, and a TCP acknowledgement with no data in it is 14 + 20 + 32 + 4 = 70 bytes, just over the minimum. On the wire, each frame is also preceded by 8 bytes of preamble, a pattern the receiver locks its clock onto, and followed by an idle gap of 12 byte-times, so a minimum frame occupies 84 byte-times and a 10 GbE link carries at most about 14.9 million frames a second in each direction. At the large end, the MTU, the maximum transmission unit, is the largest IP packet a standard frame carries: 1,500 bytes, so the most data one TCP segment carries with timestamps is 1,500 − 20 − 32 = 1,448 bytes. Jumbo frames raise the MTU to 9,000 where every device on the path agrees to it.

The floor

Three things set a floor under every round trip, and none of them is software. The first is serialisation. A link puts a frame on the wire one bit after another, and the receiving card cannot hand the frame over until its last bit has arrived and its FCS has checked. 1 GbE sends a byte every 8 ns, 10 GbE every 0.8 ns, 25 GbE every 0.32 ns. The 118-byte frame above takes 944 ns at 1 GbE, about 94 ns at 10 GbE and about 38 ns at 25 GbE; a full 1,500-byte frame takes 12 µs, 1.2 µs and 0.48 µs. A full frame already leaving on the same link delays a small one by its whole serialisation time, which is how a bulk transfer sharing a port with a feed shows up in the feed’s tail.

The second is distance. Light in glass travels at about two-thirds of its speed in a vacuum, about 200,000 km/s: 5 ns a metre, 5 µs a kilometre. The 3 m cable inside a rack costs 15 ns; a 300 m run across a data centre hall costs 1.5 µs each way. Tokyo and Singapore are about 5,300 km apart on the great circle, but the cables follow coasts and sea floors and run about 7,000 km, so one way takes about 35 ms and a round trip about 70 ms. No card, kernel, language or protocol changes that number; only a server in a different building does.

The third is the boxes in between. A cut-through switch starts sending a frame out of its outgoing port as soon as it has read the destination address, while the rest of the frame is still arriving, and costs 300–500 ns port to port. A store-and-forward switch waits for the whole frame and checks it first, which adds the frame’s serialisation time again: about 94 ns for the 118-byte frame at 10 GbE, 1.2 µs for a full one. When the outgoing port is faster than the incoming one, the switch must buffer enough of the frame to avoid underrun; the simple same-speed cut-through estimate no longer applies. If the outgoing port is busy, the frame waits behind whatever is leaving. Every extra hop adds the same again.

Add them up for two machines in one rack with one switch between them. Each way, the sending card pulls the frame across PCIe, a few hundred nanoseconds; serialises it, about 100 ns; the switch forwards it, 300–500 ns; and the receiving card writes it across PCIe into DRAM, a few hundred nanoseconds more. That is about 1 µs each way and a round trip of about 2 µs before any interrupt is raised or a single instruction of either kernel or either program has run: two budgets, and the best the hardware can do. A data centre with several tiers of switches and hundreds of metres of fibre puts the floor at several microseconds. A cloud region’s availability zones (AZs), separate data centres tens of kilometres apart, put it at hundreds. The far side of a sea puts it at tens of milliseconds, and Tokyo–Singapore cannot be under 70. Every other number in this chapter is software and queueing piled on top of one of these floors.

Read more: Brian “Beej” Hall’s Beej’s Guide to Network Programming is the friendly walk through the socket calls, in C; James Kurose and Keith Ross’s Computer Networking: A Top-Down Approach is the standard textbook for everything beneath them, layer by layer; and the Linux man pages socket(7) and tcp(7) list every option the kernel offers on a socket and what each one does.