Sockets and connections

4.1 What the network is to a program

To a program, the network is a handful of system calls on a small integer. The program never sees a cable, a card or a header. It hands the kernel bytes and later asks the kernel for bytes, and everything in between is the kernel’s and the hardware’s: cutting the bytes into packets, getting each packet to the right machine, noticing what was lost and sending it again, putting what arrives back in order. This section is what the program sees, what lies beneath it, how many bytes the envelopes add, and the physical floor that no software gets below.

The socket

A socket is a kernel object that stands for one end of a conversation over the network. The program never holds the object itself. It holds a file descriptor, a small integer that indexes the process’s table of open files, the same table that holds its files and pipes. The table’s entry points to the kernel’s socket structure, a couple of kilobytes of state for a TCP socket, which lives in kernel memory where the program can neither read nor write it. Everything the program does to a socket it does by system call, naming the descriptor. read, write and close work on it as on a file, and the calls that only make sense for a network, connect, send, recv and the rest, take it as their first argument. In Rust a TcpStream, a TcpListener or a UdpSocket is a wrapper around one such descriptor, which as_raw_fd() returns, and dropping the wrapper closes it.

The arrangement is the one the kernel uses for every shared device, and for the same reason: many programs share one network card, and none may touch it directly. It has a consequence that runs through the whole chapter. The bytes a program sends are copied out of its memory into the kernel’s by the system call that sends them, and the bytes it receives are copied out of the kernel’s memory into its own by the system call that receives them. On the ordinary copied socket path, data crosses that boundary inside a syscall, with a batch potentially containing many messages, and what happens to it on the kernel’s side of the boundary is decided by the kernel.

What the program asks for

A TCP conversation has two sides, and they set it up differently. The side that waits for others, a venue’s order gateway or a feed server, creates a socket and calls bind to attach it to a local address and port, then listen to tell the kernel to accept connections there. From then on the kernel answers connection requests by itself: it completes TCP’s opening exchange with each caller and parks the finished connection on an accept queue, whose length the listen call set. accept takes one connection off that queue and returns a new socket, with a new descriptor, for it; the listening socket stays where it was, waiting for the next. The side that calls out creates a socket and calls connect with the far address. The kernel chooses a local port from its ephemeral range, 32768–60999 by default, looks up which interface and which next hop reach the far address, and sends a SYN, a segment with no data that asks to open a connection. It puts the thread to sleep until the far side’s answer, a SYN-ACK, comes back, sends an ACK, an acknowledgement, to confirm it, and returns. So connect costs one round trip, 70 ms to a venue 35 ms away, before a byte of data can move.

send, or write, copies the program’s bytes into the socket’s send queue and returns. Returning means the kernel has the bytes, not that they have left the machine, let alone arrived. Usually the kernel builds a segment and hands it to the card within the same call; sometimes it holds the bytes for reasons of its own, and those reasons are among the costs this chapter prices. The bytes stay in the send queue after they leave, because until the far side acknowledges them the kernel may have to send them again. If the queue is full, send sleeps until there is room, or, on a socket set to non-blocking, returns at once with WouldBlock. recv, or read, copies whatever the receive queue holds, up to the length of the program’s buffer, and returns the count. If the queue is empty it sleeps until something arrives, or returns WouldBlock. It can return any number of bytes from one to the buffer’s length, and, with a non-empty destination buffer, returns zero at orderly end-of-stream. Errors are separate results; TCP does not guarantee the peer stays reachable forever. It never returns a message, because TCP has no messages: only a stream of bytes, numbered from the start of the connection.

UDP is simpler. There is no connection to set up. send_to hands one datagram to the kernel, which sends it and forgets it, keeping nothing to resend, and recv_from returns exactly one datagram: the whole of it if the buffer is large enough, a truncated piece if not. The receive queue holds whole datagrams, and when it is full the kernel drops new arrivals and counts the drops.

For each socket the kernel keeps four things that matter here. The first is its state. A TCP socket is always in one of eleven states of a small state machine: LISTEN on a waiting socket, SYN-SENT while connect waits, ESTABLISHED while data flows, and a handful of closing states, of which TIME-WAIT keeps a closed connection’s identity reserved for a minute on the side that closed first, so that a stray segment of the old connection is not taken for part of a new one. The second is the send queue: bytes written and not yet acknowledged, with the sequence number of each and the timers that will resend them. The third is the receive queue: bytes that have arrived in order and not yet been read, together with the window, how much more the socket will take, which the kernel announces to the far side in every segment it sends. Segments that arrive after a gap wait apart, in an out-of-order queue, and join the receive queue only when the gap is filled, so recv never returns a byte that lies beyond a hole.

The fourth is the key that finds the socket. An established TCP connection is identified by four numbers, the local address and port and the remote address and port, together called the 4-tuple. Every arriving segment carries all four in its headers, and the kernel hashes them to find the socket in its table of established connections: a lookup of tens of nanoseconds, however many connections the box holds. A segment that matches no connection is tried against the listening sockets by local address and port alone, and if none matches, the kernel answers with a reset, a segment that tells the sender no such connection exists. A UDP socket is found by its local address and port, and by the remote pair as well if the program connected it to one peer. This lookup is the step that turns “a packet arrived” into “this socket’s reader has data”.