4 · Talking between machines Draft

Network review questions

6 min read

On this page

4.12 What to be able to say afterwards

  • On the ordinary copied socket path, write accepts bytes into the kernel; it does not confirm delivery. TCP is a byte stream, so reads and writes can be partial and message boundaries belong to the application.
  • With plain IPv4 and TCP timestamps, a 48-byte payload makes a 118-byte Ethernet frame including FCS, excluding preamble and inter-frame gap. At 10 GbE it serialises in about 94 ns; fibre adds about 5 µs per kilometre one way.
  • The illustrative round-trip ladder rises from 100–200 ns through shared memory to 5–15 µs over Unix sockets, 10–30 µs over loopback and 20–50 µs between nearby machines with sleeping readers. Polling and bypass can lower those costs; measure the target hardware.
  • Receive stages are NIC DMA, interrupt/NAPI, protocol processing, socket queue, optional reader wake-up and copy. Sleeping and coalescing add delays that vary with load and power state.
  • Nagle can interact with delayed ACKs to hold a split small message for tens of milliseconds. TCP_NODELAY removes Nagle’s hold; gathering a message into one write avoids the split-request pattern. Neither guarantees one packet or a complete write.
  • TCP recovery blocks bytes behind a gap. The RTO is an estimator subject to configured limits, not the minimum timeout plus RTT; current Linux commonly defaults to a 200 ms minimum. Sender-side counters describe this end’s transmissions.
  • UDP preserves datagrams but leaves loss, duplicate detection and ordering to the application. Redundant feeds can hide one-path loss. A stateful book must buffer successors or resynchronise after an unrecovered gap before publishing valid state again.
  • Non-blocking socket polling removes reader sleep but still makes syscalls. NAPI busy polling can pull receive work onto the reader’s core where supported. io_uring changes submission and completion handling and can also integrate busy polling; it does not inherently bypass the stack.
  • Stream framing needs validated sizes, a bounded receive buffer and retained partial messages. WebSocket fragmentation, control frames and negotiated compression add protocol work beyond finding one frame header.
  • The example’s 48-byte binary schema permits constant-offset loads; its 161-byte JSON counterpart needs scanning and conversion. The quoted 10–20 ns and 1–3 µs are predictions. Borrowed strings avoid ownership allocations when borrowing is valid; Value allocation counts depend on implementation and features.
  • TCP plus a normal full TLS handshake takes two round trips with TLS 1.3 or three with TLS 1.2. Record costs, allocation behaviour and the relative merits of user-space TLS, kTLS and NIC offload require measurements with the chosen API and backend.
  • DPDK, Onload and AF_XDP bypass different stages and retain different drivers, APIs and capture options. Their engineering costs are easiest to justify when local software delay dominates; distance and loss recovery dominate the cross-sea example.
  • A wire-to-decision measurement needs a defined packet boundary, a valid mapping to the decoded message, and calibrated NIC/system clock domains. Decode-to-emit timing measures a different interval.

4.13 Quiz

  1. A venue’s gateway and a trading box sit in one hall, 300 m of fibre apart with one cut-through switch between them, at 10 GbE. The gateway sends a 48-byte depth update over TCP with timestamps on. How many bytes reach the wire, and how long do they take to serialise? What is the approximate hardware-only time from the sending card fetching the frame to the payload reaching receiver memory? Which saves more: 25 GbE cards, or moving the box into the venue’s rack, 3 m away?
  2. A quiet feed sends a 100-byte update every 5 ms. The strategy thread waits in epoll_wait on core 4, which may enter C6; the card’s interrupt goes to core 9, kept in a shallow C-state; adaptive coalescing has settled on 30 µs. Estimate the time from the last bit on the wire to recv returning. If the thread instead spins on a non-blocking recv and coalescing is left alone, what is the time, and why does the 30 µs remain? What removes it without touching the card, and at what price?
  3. Assume the header leaves as a separate segment, no quick ACK intervenes, and the 40 ms delayed-ACK interaction repeats. An order gateway writes each 120-byte cancel-replace as a 16-byte header and a 104-byte body, in two write_all calls on a socket with Nagle on, and the venue delays its acknowledgements on Linux’s 40 ms timer. In the same rack, 15 µs one way, when does the venue hold the whole message, and how many request-and-reply exchanges fit in a second? What if the venue is 35 ms away? Name three fixes on the gateway’s side and what each costs.
  4. A venue 35 ms away sends 4,000 updates a second over WebSocket, TLS and TCP, one to a segment. One segment is lost and recovered by fast retransmit. How late is the lost update, how many updates become readable at once, and how late are they on average? How many would wait if the venue’s configured retransmission timeout for this example were 270 ms? A multicast feed at the same rate loses both copies of one packet, and its gap-fill service answers in 100 µs: what is late, and by how much?
  5. A feed thread receives 500 updates a second and spends 2 µs of its own work on each. Way A waits in epoll_wait on a pinned core with C-states capped at C1; way B spins on a non-blocking recv with SO_BUSY_POLL at 50 µs on an isolated core. Using the illustrative setup from section 4.6, estimate each way’s p50 and p99 from the wire to the buffer, and the share of its core it uses. How many empty polls a second does B make? What happens to each if the cap is lifted?
  6. In a burst a venue sends 50,000 updates a second, arriving in clumps of 20 per recv. What share of a core does parsing take at 1–3 µs a JSON message, and at 10–20 ns a binary one? How long does the last update of a clump wait behind the others in each case? With an owned struct of six String fields, at 20–50 ns an allocation and as much again a free, how much of each parse is the allocator, how many allocations a second does the burst make, and what does borrowing change?
  7. A venue 35 ms away closes a feed connection 50 ms into a burst of 20,000 updates a second, and the client notices at once. Reconnecting takes TCP, TLS, the WebSocket upgrade (one round trip) and the subscription (one round trip), then a snapshot over an HTTPS connection kept open for the purpose (one round trip). How long is the strategy blind with TLS 1.2 and with TLS 1.3, and how many updates does it miss? What does resuming the TLS 1.3 session without early data save? What would an idle pre-connected spare, and a subscribed one, have made of it?
  8. Two desks consider Onload, which would bring each receive and each send to about 1 µs. Desk A is co-located one switch from the venue’s gateway and busy-polls kernel sockets: 2–5 µs to receive, 1 µs to decide, 2–5 µs to send. Desk B trades a venue 35 ms away through kernel sockets with a sleeping reader: about 15 µs to receive and a few to send. For each, what share of the reaction is the kernel, and what would bypass save? Where should Desk B spend instead?