Sequence numbers, trade streams, and recovery channels

What a sequence number promises, and what it does not

A venue’s sequence number promises three things: every change to the book, in the order the engine made it, once. It does not promise that you received it; it lets you notice, after the fact, that you did not. A gap is an update whose number is not the one the join rule says must come next, and it means at least one change is missing from the copy. A missing change cannot be guessed, so the only cure is a resync: discard the copy, take a fresh snapshot and join the stream again, mechanics that belong with recovery later. Nor does the number survive the venue’s own restarts: a restarted feed’s numbers may begin again or jump, and a number far from the last cannot say whether a million changes were missed or the venue began counting afresh. And two streams’ numbers are unrelated: the depth stream’s U and u and the trade stream’s trade ids are separate counters, and neither orders one stream’s events against the other’s.

Trades: the same event seen twice

Venues publish trades on a separate trade stream. On Binance’s futures market it is btcusdt@aggTrade, every 100 ms, each message carrying a price p, a quantity q, its own trade ids and a flag m that says which side was the taker. When the market buy of 1.2 BTC walked three ask levels, the trade stream reported what traded, 350, 600 and 250 lots at three prices, and the depth stream reported what was left: two levels at 0 and one at 170. It is one event inside the matching engine, published twice. The two streams are not synchronised: each is batched on its own schedule and numbered by its own counter, so a trade may arrive before or after the level change it caused, and a level that lived for less than one batch interval may never appear in the depth stream at all, though a trade at its price does. For an L2 copy, the depth stream is the book and the trade stream is the history of how it changed. A copy that subtracted each trade from its level would come out right when the trade arrived first, and only because the delta that follows carries an absolute 170; when the delta arrives first, the subtraction counts the match twice, and 170 lots become −80.

Multicast with a recovery channel, and one WebSocket

Classic exchanges send binary messages over UDP multicast. UDP is the networking chapter’s envelope with no acknowledgement and no retransmission; multicast means one packet from the venue, copied by the switches to every subscriber, so a thousand subscribers cost the venue one send. Each packet carries a sequence number, and the receiver detects its own gaps. CME Group’s MDP 3.0 is the named example: book updates in Simple Binary Encoding over UDP multicast, in market-by-price and market-by-order forms, every packet sent twice, on a feed A and a feed B, so a receiver can take whichever copy arrives first. Behind them sits a recovery channel in two forms: a TCP replay service that resends a range of packets by sequence number, at most 2,000 from the last 24 hours and documented as meant for small-scale recovery, and market recovery feeds that loop through snapshots of every book, for a receiver that lost more or just started.

Crypto venues send JSON over a WebSocket: a long-lived TCP connection, wrapped in TLS, carrying framed messages, each with a short header giving its length so the reader knows where it ends. TCP delivers the connection’s bytes in order, once, so within one connection nothing is missing. Gaps happen between connections. Binance’s documentation says a connection is valid for only 24 hours and to expect a disconnect at that mark; its spot documentation adds a serverShutdown event, sent when the server is about to shut down, with the instruction to connect again. And no venue can hold unbounded memory for a reader that falls behind; closing a slow consumer’s connection is the usual answer. A new connection carries changes from now on. There is no replay: changes made while the program was away are gone, and the only way back is a new snapshot and the join rule again.

What it costs

A resync is mostly waiting. The snapshot of 1,000 levels a side, about 60 KB of JSON, takes one HTTPS round trip of 70 ms from a venue 35 ms away when the HTTPS connection is already open, and nearer 200 ms when it must first pay for TCP and TLS handshakes. Parsing it into ticks and lots takes 0.5–2 ms, 2–8 million cycles, 500–2,000 budgets. Meanwhile the venue changes its book 2,000–20,000 times a second and the stream keeps delivering, so a 70 ms fetch leaves 140–1,400 level updates in the buffer and a 200 ms one 400–4,000. Replaying them is little: 8–20 µs on a dense array, one slot per price, at 2–5 ns an update, 0.4–2 ms on a BTreeMap at 100–500 ns. What costs is the time the copy cannot be used, 70–200 ms, 70,000–200,000 budgets, and the money a decision taken on it would lose. One more number is set by the venue, not the program: Binance’s depth events come at most every 100 ms, so a change can wait up to 100 ms before it is sent at all, a hundred thousand budgets no structure can recover.

Worked example: a reconnect at 09:30:00.000

A feed handler, the part of the program that reads the stream and keeps the copy, holds BTC-USDT-PERP on core 3, joined to the futures stream at 100 ms speed; the venue is 35 ms away, and the last event applied ended at u = 1,873,465,902. At 09:30:00.000 the connection closes at the venue’s 24-hour mark, and from this instant the copy is not trusted. Reconnecting takes three round trips, for the TCP handshake, TLS and the WebSocket’s opening request, so the stream reopens at .210, and the first event arrives at .260 with U = 1,873,467,410, u = 1,873,467,905 and pu = 1,873,467,388. The changes since 1,873,465,902 are gone, and so is the old copy. The handler buffers the event and requests the snapshot on an HTTPS connection it already holds. The venue takes the snapshot at about .295, and the 60 KB answer arrives at .330 with lastUpdateId = 1,873,468,160, parsed by .332. The buffered event’s u is below lastUpdateId, so it is dropped: its changes are already in the snapshot. The next event arrives at .360 with U = 1,873,467,918, u = 1,873,468,402 and pu = 1,873,467,905. Its U is at or below lastUpdateId and its u at or above, so it is the first applied, and its pu matches the dropped event’s u: the stream lost nothing while the snapshot was in flight.

At 2,000–20,000 level updates a second, the venue made 720–7,200 changes in those 360 ms. Those before the stream reopened were never sent; the snapshot summarises them. The dropped event carried 100 ms of them, 200–2,000 level updates before repeats at one price are merged, discarded unread. The first applied event carried as many again, and replaying it on core 3 costs 0.4–10 µs on the dense array and 20 µs–1 ms on a BTreeMap. The copy was untrusted from .000 to .360: 210 ms reconnecting, 50 ms waiting for the first event, and 100 ms of resync, inside the 70–200 ms a resync costs from a venue this far away. That is 360,000 budgets of not knowing, against at most 1 ms of work.

Seeing it

Only the program sees the numbers, so only it can check the join. Log one line per depth event with the local receive time, the venue’s event time E, and U, u and pu; cheap at ten events a second, it turns “the book looked wrong at 09:30” into which number was expected and which arrived. Count gaps detected and resyncs completed; a gap counter above zero on a quiet day says something about the connection or the code. ss -ti on the WebSocket’s TCP connection shows the kernel’s view: rtt:70.2/0.4 is the smoothed round trip and its variation in milliseconds; retrans:0/7 counts segments being retransmitted now and over the connection’s life, and a lost segment can hold later bytes behind it until recovery; fast retransmit can recover before the retransmission timeout; lastrcv is the milliseconds since data last arrived, which on a stream sending every 100 ms should stay below a few hundred; and a Recv-Q that stays above zero means the kernel holds bytes the program has not read, which the venue sees as a slow consumer. A capture with tcpdump shows the segments and their timing but not the messages: the frames travel inside TLS records, encrypted. Log at the application, after TLS has decrypted them.

Read more: Binance’s “How to manage a local order book correctly” for futures is at https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/How-to-manage-a-local-order-book-correctly, and the futures depth event with pu at https://developers.binance.com/legacy-docs/derivatives/usds-margined-futures/websocket-market-streams/Diff-Book-Depth-Streams. The spot rule is a section of https://developers.binance.com/legacy-docs/binance-spot-api-docs/web-socket-streams, also in the official repository at https://github.com/binance/binance-spot-api-docs/blob/master/web-socket-streams.md with the 24-hour limit and the serverShutdown event. CME Group’s client wiki describes MDP 3.0’s feeds, A/B arbitration and recovery channels from https://cmegroupclientsite.atlassian.net/wiki/spaces/EPICSANDBOX/pages/457222936/MDP+3.0+-+Dissemination. RFC 6455, The WebSocket Protocol (Ian Fette and Alexey Melnikov, 2011), defines the framing. Martin Kleppmann’s Designing Data-Intensive Applications (O’Reilly, 2017) shows, in its chapter on replication, a new follower joining from a snapshot tied to a log position: the same join.