Joining a snapshot to a stream

The join rule

The two arrive by different paths. On a crypto venue the deltas come over a long-lived connection; the snapshot comes from a separate HTTPS request that takes a round trip, while the book keeps changing and deltas keep arriving. The program must decide which deltas it holds are already inside the snapshot, which is the first to apply on top of it, and what proves none is missing after it. The rule that decides is the join rule, and the venue writes it down. It is not to be inferred from field names: the boundary cases, an event whose ids straddle the snapshot’s number or end exactly on it, are where a copy silently goes wrong.

Binance documents the rule for its USDⓈ-M futures market, home of the running instrument, as nine steps. Open a stream to wss://fstream.binance.com/public/stream?streams=btcusdt@depth. Buffer the events you receive; for the same price, the latest received update covers the previous one. Get a depth snapshot from https://fapi.binance.com/fapi/v1/depth?symbol=BTCUSDT&limit=1000. Drop any event where u is less than lastUpdateId, the update id the snapshot returns. The first processed event should have U <= lastUpdateId AND u >= lastUpdateId. While listening to the stream, each new event’s pu should equal the previous event’s u; otherwise start again from the snapshot step. The data in each event is the absolute quantity for a price level; if the quantity is 0, remove the price level; and receiving an event that removes a price level not in your local book can happen and is normal. pu is documented as “Final update Id in last stream (ie u in last stream)”. The stream sends an event every 250 ms by default, or every 500 ms or 100 ms on request.

The spot market’s rule is close but not the same. It opens wss://stream.binance.com:9443/ws/bnbbtc@depth, buffers events, notes the U of the first, and fetches https://api.binance.com/api/v3/depth?symbol=BNBBTC&limit=5000. If the snapshot’s lastUpdateId is strictly less than that first U, it fetches the snapshot again. It discards every buffered event whose u is less than or equal to lastUpdateId; the first remaining event should have lastUpdateId within its [U;u] range. The local book is set to the snapshot, with lastUpdateId as its update id, and each event is then tested: if its u is below the book’s update id, ignore it; if its U is greater than the book’s update id plus one, events were missed, so discard the book and start again; otherwise set each listed level to its new quantity, removing it at zero, and make the event’s u the book’s update id. Spot snapshots reach at most 5,000 levels a side, so, as its note says, you will not learn quantities outside the initial snapshot unless they change.

Three differences matter. Spot checks continuity by U following the previous u by one (“Normally, U of the next event is equal to u + 1 of the previous event,” the page says); futures carries the previous event’s u in each event as pu and checks that, because on futures consecutive events’ ids need not be consecutive: the documentation’s own example event has U 157 and pu 149. The boundary differs by one: an event whose u equals lastUpdateId is discarded on spot and kept, as the first applied, on futures. And the depth differs, 5,000 levels a side against 1,000; a level beyond the snapshot is one reason a delete for a level the copy never had is normal. Here is the futures procedure as a timeline, with small made-up ids in the documented relations:

The event at 135 ms straddles the snapshot: its changes 461 to 470 are already in it. Applying the whole event is still right, because each level in it carries its absolute quantity as of u, 512, which is newer than the snapshot’s 470. Had it carried differences, it could not be applied at all.