Multicast feeds and loss recovery

Multicast, and what exchanges choose

An ordinary address names one machine: unicast. The addresses from 224.0.0.0 to 239.255.255.255 name groups instead, and a packet sent to a group is delivered to every machine that has joined it: multicast. A program joins by asking its kernel, which announces to the network with IGMP, the internet group management protocol, that this interface wants this group, and switches that listen in on IGMP send each group’s frames only to the ports that asked. The sender sends each packet once and the network copies it where the paths branch, so subscriber count does not require one send per subscriber, though path lengths, queueing and loss can still give different arrival times, and the sender keeps no state for any of them. Multicast does not cross the public internet. It lives inside a venue’s own network: its co-location halls, where members rent space for their servers a few metres of cable from the venue’s, and the carrier networks that extend them.

Classic exchanges publish market data this way, and each chapter of the choice answers a need. Every member must see each change at the same moment, and over TCP a venue with 500 subscribers would send 500 copies one after another, the last hundreds of microseconds after the first. No member’s slowness may slow anyone else, and over TCP a subscriber that stopped reading would leave the venue holding its backlog; over multicast a slow subscriber loses packets, and the loss is its own. The price is UDP’s delivery, so the feed adds back as much of TCP’s machinery as market data needs. Every packet starts with a sequence number for its channel, a multicast group carrying some of the venue’s instruments, so a receiver can detect a discontinuity when a later packet arrives and distinguish it from duplicates or reordering under the feed’s rules. Many venues publish redundant A and B feeds over separate networks, and, on feeds with that design, the receiver arbitrates: it takes whichever copy of each sequence number arrives first and discards the other, so a packet is lost only if both copies are. For the rare double loss a gap-fill service, usually over TCP, resends packets by number, and for a receiver that joined late a snapshot service sends the whole order book to rebuild from.

A real arbiter holds an early packet for a few microseconds before declaring a gap, in case the other feed is about to deliver the missing one. That is head-of-line blocking again, but bounded, and the bound is the program’s choice.

Order entry at many such exchanges uses TCP, with application-level session recovery. The protocol needs to detect duplicate or missing requests across reconnects as well as order them within one connection, since a cancel that overtook its order would be rejected and the order would stand. It is one sender talking to one receiver, and the order protocols add sequence numbers of their own on top of TCP’s, so that a session can resume after a reconnect with nothing lost and nothing sent twice. Inside a co-location hall a fast retransmit is one round trip of tens of microseconds.

Crypto venues serve their customers over the public internet and from cloud regions, where multicast does not go, and some of those customers are browsers. They publish market data as the web serves pages, as a stack of three protocols: WebSocket, which begins as an HTTP request and turns the connection into a two-way stream of framed messages, carried in TLS, which encrypts the stream, carried in TCP, with the messages usually encoded as JSON. Every TCP cost above therefore lands on market data: round trips to open, slow start after a lull, and a lost segment stalling the program’s copy of every order book on the connection until the venue resends it. The updates usually carry sequence numbers all the same, used to stitch the stream onto a snapshot fetched separately. A few venues now offer a binary encoding of the same streams, which shortens the parse and leaves the transport as it was.

What it costs

Per packet, TCP’s 32-byte header against UDP’s 8 is 24 bytes more, about 19 ns at 10 GbE, plus a 70-byte acknowledgement back for every one or two segments. Per connection, it adds a round trip to open and a slow start from 10 segments, at the opening and after every lull. Per loss, it adds about one round trip for fast retransmit, about two for a tail-loss probe, and at least 200 ms for a timeout, to every message behind the hole: on a 70 ms path, 70,000 budgets for a quick recovery and 200,000 or more for a timeout.

UDP charges for loss whatever the application’s correctness rules require. With A and B feeds, a packet lost on one costs the difference between the two copies’ arrival, a few microseconds at most, paid for in advance by receiving every packet twice. A packet lost on both requires gap-fill or a new snapshot. Later datagrams can arrive on time, but a stateful book generally must buffer them until the missing sequence is recovered, or invalidate the book and rebuild it. UDP lets the application choose the recovery boundary; it does not remove update dependencies.

Worked example: one lost segment on a feed

The feed thread holds a WebSocket connection, over TLS and TCP, to a venue 35 ms away: a 70 ms round trip. The venue sends 2,000 updates a second, one every 500 µs, each in a segment of its own. At t = 0 it sends segment n, and a router somewhere on the 7,000 km drops it.

From 35 ms to 106.5 ms the program’s copy of the order book is the book as of update n − 1, and the program does not know it, while segments n + 1 to n + 142 arrive every 500 µs and wait where it cannot see them. At 106.5 ms the resend fills the hole and 143 updates become readable at once: n is 71.5 ms late, n + 1 is 71 ms late, and so on down to n + 142 at 0.5 ms, 36 ms late on average, on top of the 35 ms every update spends in the fibre. Draining them costs the feed thread 1.3–3.5 µs each for the TLS record, the WebSocket frame and the JSON, 0.2–0.5 ms for the lot. If the path loses one segment in 100,000, this happens every 50 seconds and delays 0.14% of updates: the p99 does not see it, the p99.9 is about 20 ms, and the max is 71.5 ms.

Now suppose the quick mechanisms fail: a link drops everything for a moment, tail-loss probe included, or the resend is itself lost, and the venue’s kernel falls back on its retransmission timer, assumed to be 270 ms for this scenario. That is a chosen timer value, not a consequence of the 70 ms RTT alone. Update n arrives about 270 ms after it was due, with about 540 updates behind it, waiting in our out-of-order queue or, as far as its windows hold them back, in the venue’s send queue. The timeout has also cut the venue’s congestion window to one segment, so whatever it held back leaves at the pace of slow start, over several more round trips. One such event an hour puts the feed’s max above a quarter of a second while its p99.9 does not move.

Now the same loss on a classic multicast feed at the same 2,000 updates a second. Packet 5,000 is lost on the A feed; its copy on the B feed arrives a few microseconds later, and nothing else happens. Suppose both copies are lost. Packet 5,001 arrives 500 µs after 5,000 was due, the arbiter sees the gap at once, and 5,001 and every packet after it are delivered on time. The program marks affected books invalid, buffers later updates, and asks the gap-fill service for 5,000. After recovery it applies the missing update and buffered successors in sequence, or rebuilds from a snapshot under the venue’s rules. The gap-fill takes a round trip of tens to hundreds of microseconds inside the hall, 70 ms if the service were as far away as the crypto venue. Under TCP, 143 updates were late and the program saw silence. Under multicast, later packets can arrive on time and the gap is visible as soon as a successor arrives. The stateful book still remains invalid until recovery; packet arrival and safe book publication are different boundaries.

Seeing it

ss -ti adds fields after a loss: lost:3 counts segments currently believed lost, beside retrans and rto. All three describe what this end sent, so they are the fields to watch on the order connections. On a feed, where the venue sends, they stay at zero, and the receiving side of a loss shows as rcv_ooopack, the count of segments that arrived out of order.

nstat -az TcpExtTCPLossProbes TcpRetransSegs TcpExtTCPFastRetrans prints box-wide counters for the recovery mechanisms: tail-loss probes sent, segments resent for any reason, and resends triggered by duplicate acknowledgements. TcpExtTCPTimeouts counts retransmission timeouts, the expensive case, and TcpExtTCPOFOQueue counts segments put into an out-of-order queue: the receiving side of every loss on every feed.

tcpdump shows a loss as it happens. Look for later sequence ranges arriving while the cumulative acknowledgement stays fixed and SACK ranges grow. When the missing segment lands, the cumulative ACK jumps across the now-contiguous range. This is the signature of head-of-line blocking; exact byte counts require the captured segment sizes. Wireshark labels the same lines “TCP Dup ACK” and “TCP Fast Retransmission” itself.

On a UDP feed, nstat -az UdpRcvbufErrors counts the datagrams dropped because a socket’s queue was full: the losses the program caused by falling behind, as distinct from those the network caused, which the feed sequence numbers reveal. For a TCP feed, a stale-feed deadline is still needed because an in-stream gap may remain hidden behind TCP recovery.