Detecting a stale book

6.6 Keeping the book true: gaps, crossed books and recovery

The copy is correct for exactly as long as every delta (a numbered change to one price level) since its snapshot (the whole book the venue sent to start from) has been applied, in order, once, and nothing in the structure knows whether that is still so: an array holding a level the venue removed a second ago looks exactly like one holding a live level. Whether the copy is still a copy is a fact about the stream, kept beside the structure by the feed handler that calls apply (the call that writes one update into the book). A copy is stale when it no longer matches the venue’s book: some level holds a quantity the venue no longer shows, or a level the venue shows is missing. A copy can be stale and know it, or stale and not know it. This section is about turning the second into the first, and about what the second costs.

How a copy goes wrong

A missed delta, a change the venue published and the program never applied, is the commonest, with three usual causes. The connection drops and is reopened, and what was published in between is not resent. The venue’s server drops a slow consumer: when one of its thousands of readers falls behind, the bytes meant for it pile up in a send queue, and past a limit the server closes that connection or skips messages for it, rather than hold memory indefinitely for one slow program. Or the program’s own framing, the code that cuts the byte stream into messages, fails: TCP delivers bytes in order but does not mark where one message ends, so a parser that mishandles a message split across two reads quietly loses one.

A delta applied to the wrong snapshot. The join rule, the venue’s statement of which delta is the first to apply after a given snapshot, exists because the snapshot and the stream come by different routes and arrive at different times. Ignore it or misread it and the copy starts from a state the deltas do not continue. Re-apply a delta the snapshot already contains, and a level the venue deleted after it comes back to life; start one delta too late, and a change is missing while the numbering after the join looks perfect.

A reconnect. The new connection’s stream starts from wherever the venue is now. The old copy is simply of a past that no longer exists, and no delta will repair it, because the deltas that would have done so went to a connection that is gone. Only a fresh snapshot will, so the handler treats the reconnect itself as the signal instead of waiting for the numbering to jump.

The venue’s own bug. It happens: a venue publishes a level that was never there, or skips a number in its own sequence. The program cannot fix it, only notice, resync and count. One thing looks like a venue bug and is not: Binance’s documentation says that a delete for a level the copy does not hold “can happen and is normal”; a level deeper than the snapshot reached was never in the copy.

In every case the copy does not fail. Nothing throws and nothing crashes; best_bid keeps answering with a number, in the same 1 ns, and the number is wrong. None of this is particular to trading: a database replica following its primary’s change log has the same four failures and the same silence.

How you notice

Cheapest first is the sequence gap. The handler keeps the number the next update must carry. A lower number is a repeat of one already applied, and is dropped; a higher one means at least one update was lost, or arrived out of order, which for the copy is the same thing. This is the one check that is exact: as long as the venue numbers its stream correctly, it raises no false alarms and misses nothing, and it is a single compare. It catches missed deltas and reconnects, not levels that are wrong for other reasons.

A crossed book is a copy whose best bid is greater than or equal to its best ask. The venue’s own book can never be crossed, because its matching engine, the program that pairs buyers with sellers, trades any buy priced at or above the best ask instead of letting it wait. So a crossed copy is a wrong copy, and the cause is almost always a missed delete: the copy still holds a bid the venue removed, and a later ask lands at or below it. For a venue event containing several level changes, make the check only after the complete event has been applied. The copy may temporarily cross while those changes are being applied, even when the final book is valid. It catches only errors at the top (the best bid and best ask), and only when the other side comes close enough to touch them.

A negative or absurd quantity is a decoding fault or a venue fault, and is checked in the decode, where the field is already in a register.

The strategy’s own fills. A fill, the venue’s report that one of the program’s orders traded, at a price the copy says could not have traded proves the copy wrong. If the program’s bid at 6,812,344 is filled while the copy shows a bid at 6,812,345, price priority (the best price trades first) says the seller would have met the higher bid first, so that bid is not really there. Fills and depth updates arrive on separate streams and either can come first, so one mismatch is a hint and a run of them is treated as a gap.

The slow, sure check is a diff against a fresh snapshot: fetch one, bring the copy to the same update number, compare level by level, and count the differences. Only the snapshot’s depth can be compared, 1,000 levels in Binance’s documented request. It is the only check that finds a wrong level the top never touches, and it costs a 70 ms round trip, a 0.5–2 ms parse and a share of the venue’s limit on requests, so it is a test tool and a periodic audit, never a hot-path check.

Check sequence continuity before applying each complete feed event, using the venue’s sequence rules, and check for a crossed book after all of that event’s level changes have been applied. In the simplified one-level-per-event model these reduce to the two comparisons described above; a batched venue event must retain its event boundary. Everything else runs off the hot path.