The cost of stale data

Why a stale book costs money

A resting order, a limit order still waiting, is a promise to trade at a price with whoever comes first. If the market moves past that price and the program’s copy does not notice, the order is still there, and the trader whose copy did notice takes it. “Picked off” is the name for that fill. The loss is the distance the market moved times the size, and it arrives as a fill that looks normal: the right instrument, a price the program itself chose, a confirmation like any other. The general effect is called adverse selection: a resting order’s fills are selected against it by the traders who know more, so it is most likely to be filled exactly when filling it is a mistake. This is the tail of this chapter, and it is priced in money.

What it costs

The checks on every update are two compares, 1–2 ns, at most a fifth of a per cent of the 1 µs budget. A resync costs 70–200 ms during which the book is not trusted, nearly all of it the snapshot’s round trip, 70 ms to a venue 35 ms away; the 0.5–2 ms parse and the replay of up to 4,000 buffered deltas, 8–20 µs on the dense array, are the small part. In budgets a resync is 70,000–200,000, but that is the wrong unit: during a resync the program is not slow, it is absent. A stale book acting is priced in money; the example below prices one fill. The indirect cost is quieter: every second in Stale or Syncing is a second with the program’s orders withdrawn, earning nothing.

Worked example: a bid left at a price the market has moved through

The instrument is BTC-USDT-PERP: a tick $0.01, a lot 0.001 BTC. At 14:02:17.300 the copy’s best bid is 6,812,345 ($68,123.45), and the desk, the trading team whose program this is, has its own bid waiting one tick below, at 6,812,344, for 500 lots, 0.5 BTC. The price falls on other venues, and the traders bidding here cancel and re-post lower. The delta deleting 6,812,345 is lost; every other delete is applied.

At 14:02:17.340 the desk’s bid is 54 ticks, $0.54, above the price at which anyone else will buy, and a seller who knows that sells to it. The loss is 54 ticks × $0.01 × 500 lots × 0.001 BTC a lot: $0.54 a BTC on 0.5 BTC, $0.27. Small. At the desk’s real size of 50 BTC the same 54 ticks is $27 a fill, and a day with 200 such events is $5,400, none of it visible as an error, because every one is a confirmed trade at a price the desk chose.

Each check catches it at a different time. The gap check fires at the next delta, about 50 µs later at 20,000 updates a second; the handler goes Stale, the cancel leaves, and the exposure shrinks to the cancel’s trip to the venue. The crossed check needs an ask at or below 6,812,345, and since no ask can sit below the desk’s live bid at 6,812,344 without trading against it, only one at exactly 6,812,345 would do: maybe never. The fill check fires on the fill itself, which contradicts price priority and so proves the copy wrong, but arrives with the bill. The snapshot diff finds the extra level at the next audit, seconds or minutes later. Only the gap check is fast enough to matter.

Seeing it

Counters are kept in the handler and exported once a second: gaps, crosses, resyncs started and completed, and time in Stale and in Syncing per hour. A gap with no completed resync after it is a handler stuck in Syncing.

Staleness age is now minus the venue timestamp of the last applied update, the ts_ns field of the 48-byte message, exported as p50, p99 and max. The two are different clocks, so the age means something only after they are aligned, by measuring the offset between them and subtracting it; its floor is then the one-way trip, about 35 ms here. It catches a feed that has gone quiet, not a lost delta: in the example above the deltas after the loss kept arriving and the age stayed low.

The snapshot diff lives in a test harness that replays a recorded stream, brings the copy to a snapshot’s update number, and reports each differing level: side, price, both quantities. With one message deleted from the recording by hand, it should report exactly the level that message changed.

ss -ti on the feed’s connection shows retrans, the segments sent again, and rto, the retransmission timeout. About 200 ms is a common Linux default minimum, but route, socket, or sysctl settings can change it. TCP preserves the order of delivered bytes while the connection remains usable; it does not guarantee that the application receives every venue event. A prolonged stall or a growing application backlog can cause the venue to disconnect a slow consumer. Gaps that coincide with rto stalls point at the network path; gaps on a clean connection point at the venue or the framing.

Read more: Binance’s “How to manage a local order book correctly” is the procedure every resync follows, in the spot documentation (https://github.com/binance/binance-spot-api-docs/blob/master/web-socket-streams.md) and, with the futures pu field, the USDⓈ-M futures version (https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/How-to-manage-a-local-order-book-correctly). Larry Harris, Trading and Exchanges (Oxford University Press, 2002), explains adverse selection in the chapters “Informed Traders and Market Efficiency” and “Bid/Ask Spreads”. Oliver Zehentleitner’s dev.to post “Your Binance L2 order book can be gap-free and still be wrong” is a case only the snapshot diff catches.