4 · Talking between machines Draft

Network review answers and reading

9 min read

On this page

Answers

14 + 20 + 32 + 48 = 114 bytes, 118 with the frame check: 94 ns at 0.8 ns a byte. One way, the sending card fetches the frame across PCIe in a few hundred nanoseconds and serialises it in 94 ns, the switch forwards it in 300–500 ns, 300 m of fibre takes 1.5 µs, and the receiving card writes it into DRAM in a few hundred nanoseconds more: about 2.3–3 µs, half or more of it distance. 25 GbE serialises the frame in 118 × 0.32 = 38 ns, saving 56 ns; the move to 3 m saves 297 × 5 ns = 1.49 µs, over 25 times as much. The fibre is the largest item on the path, and no software reaches it.
The card takes 1–2 µs, the coalescing hold 30 µs, the top half, poll and walk on core 9 2–3.5 µs, the IPI to core 4 with its exit from C6 and the scheduler pass 100–200 µs, epoll_wait 0.5 µs and recv 0.5–1 µs: about 135–235 µs, three-quarters or more of it the C6 exit. Spinning removes the wake-up and the epoll_wait, leaving about 33–37 µs. The 30 µs stays because a spinning reader finds data only once core 9’s softirq has queued it, and the softirq waits for the interrupt the card is holding. SO_BUSY_POLL makes each empty recv run the driver’s poll on core 4, which finds the frame on the ring as soon as the DMA write lands, interrupt or not: about 3 µs. The price is core 4 at 100%: at 200 updates a second and about 5 µs each, 1 ms of every second is work and 999 ms is 3–10 million empty polls.
The header leaves at once and arrives at 15 µs; Nagle holds the body, since the header is unacknowledged. The venue holds half a message and no reply to carry an acknowledgement, so its timer fires at 40.015 ms, the acknowledgement arrives back at 40.030 ms, and the body arrives at 40.045 ms. Every message is 40 ms late, and a strict request and reply manages about 25 exchanges a second. 35 ms away, the header arrives at 35 ms, the acknowledgement leaves at 75 ms and arrives at 110 ms, and the body arrives at 145 ms, 110 ms after a single write would have; with the reply’s 35 ms back an exchange takes 180 ms, five or six a second. One write per message, built whole or gathered with write_vectored, costs nothing and saves a system call, because in request and reply nothing is unacknowledged when the next message is written. TCP_NODELAY with the two writes kept sends two segments, 70 more bytes of envelope, a second send of 1–4 µs and perhaps a few microseconds of autocorking. TCP_CORK around the two writes sends one segment for two setsockopt calls, MSG_MORE on the header for none, and a cork left set holds the next message for 200 ms. TCP_QUICKACK on the gateway does nothing, because the delayed acknowledgements are the venue’s.
The lost update n is due at 35 ms. n + 1 to n + 3 arrive 0.25 ms apart, so the third duplicate acknowledgement leaves at 35.75 ms and reaches the venue at 70.75 ms, and the resend arrives at 105.75 ms, 70.75 ms after n was due. In that time 70.75 / 0.25 = 283 updates, n to n + 282, collect behind the hole and become readable together, their lateness falling from 70.75 ms to almost nothing, about 35 ms on average; draining them at 2–4 µs each takes 0.6–1.1 ms. Until then recv returned nothing, and the program could not tell the hole from a quiet market. With the timer, n is about 270 ms late and 270 / 0.25 = 1,080 updates wait behind it, and the venue’s collapsed congestion window spreads its backlog over several more round trips. On the multicast feed, n + 1 arrives 0.25 ms after n was due and reveals the gap at once; n’s information arrives 0.25 + 0.1 = 0.35 ms late, later packets arrive on time, but any dependent book remains invalid until the missing update and buffered successors are applied in order. The gap becomes visible on the first successor, not when the loss occurs.
A: a p50 of about 8–10 µs and a p99 of 20–40 µs, the tail from updates that find the interrupt’s core busy or a tick in the way. B: about 3 µs at p50 and 5–6 µs at p99. A’s core works 5–20 µs an update, for its share of the wake-up, the epoll_wait, the recv and the 2 µs: 2.5–10 ms a second, 1% at most. B’s core is 100% busy; about 5 µs an update is 2.5 ms of each second, and the other 997.5 ms go to 3.3–10 million empty polls. Lifting the cap hurts only A: 2 ms between updates is long enough for the governor to reach its deepest state, so the p50 becomes 100–200 µs, the p99 several hundred and the max milliseconds. B’s core never idles and pulls each packet off the ring itself, so nothing changes.
JSON: 50,000 × 1–3 µs = 50–150 ms a second, 5–15% of the core; binary: 50,000 × 10–20 ns = 0.5–1 ms, 0.05–0.1%. A clump arrives every 400 µs and the JSON parser clears it in 20–60 µs, so no backlog builds between clumps; the cost is inside each one. The twentieth update waits 19 × 1–3 µs = 19–57 µs before its own parse starts, against 190–380 ns in binary. Through completion of that parse the totals are 20–60 µs and 200–400 ns. Six allocations and six frees at 20–50 ns each are 240–600 ns of each parse, and the burst makes 300,000 allocations and as many frees a second, each a chance to hit the allocator’s slow path. Borrowing &str fields removes all of them and their 240–600 ns, and leaves the scan, still about a hundred times the binary read.
TLS 1.2: TCP 1 + TLS 2 + upgrade 1 + subscription 1 = 5 round trips, 350 ms, plus 70 ms of snapshot: 420 ms blind, 8,400 updates at 20,000 a second. TLS 1.3: 4 round trips, 280 ms, plus the snapshot: 350 ms and 7,000 updates. Resuming without early data saves the certificate and its checks, which are CPU time, and no round trip; early data carrying the upgrade request, if the venue accepted it, would save one, making 280 ms. An idle spare has already paid for TCP, TLS and the upgrade, so the subscription and the snapshot request go out together: one round trip, 70 ms and 1,400 updates. A subscribed spare has received the same updates all along; switching to it costs microseconds and loses nothing, provided both connections number their updates alike.
Desk A: 2–5 + 1 + 2–5 = 5–11 µs from the update at its card to the order on the wire, of which receive and send together are 4–10 µs, 80–91%, including card time rather than kernel CPU work alone. Bypass makes it about 1 + 1 + 1 = 3 µs, 2–8 µs earlier on every update, a reaction 1.7–3.7 times faster, and when two firms go for the same resting order the venue fills whichever arrives first: it pays. Desk B: an update spends 35 ms in fibre and the order 35 ms more, and the kernel’s 15 µs plus a few is under 0.03% of the round trip. Busy-polling would recover 10–13 µs of the 15 for a setting and a core; bypass a further 1–4 µs, under 0.01%, for new cards, a new stack and the loss of the tools, while one lost segment costs 70 ms. Desk B should spend on distance first, a server in the venue’s own region; then on keeping TCP from stalling, with warm connections, spares, Nagle off and one write per message; then on busy-polling.

Answers

Further reading