Snapshots and deltas

6.2 How a venue tells you: snapshot and deltas

The matching engine changes its book thousands of times a second. To tell a program what the book holds, a venue can send the whole of it or send what changed. It does both, for different jobs, and the joint between the two is where a copy is made correct or quietly wrong.

A snapshot, then numbered deltas

A snapshot is the venue’s levels at one instant, usually only the best few hundred or thousand a side, stamped with the number of the last change it includes. A delta, which this chapter also calls a level update, is one change: a side, a price, and the new quantity at that price. The quantity is the level’s new total, not a difference: “6,812,353 now holds 170 lots”, never “6,812,353 lost 250”. A quantity of 0 means the level is gone. The networking chapter’s 48-byte update is exactly this, and the shared Update in the code holds its three fields and a number. Absolute quantities have a quiet virtue: applying the same delta twice leaves the same book, where applying a difference twice would subtract it twice.

The number is an update id, or sequence number as venues use the term: the matching engine numbers each change it makes to an instrument’s book, in the order it makes them, and every delta carries its number. Unlike a TCP stream’s sequence number, which counts the bytes of one connection, it counts the changes to one book, whatever connections carry them. A snapshot’s number says which changes it already contains, and so which delta is the first to apply on top of it. The 48-byte update carries one change and one number. Binance’s JSON messages, which its documentation calls events, batch instead: each carries every level that changed during an interval of 100 ms or more, and a range of update ids, from U, the first in the event, to u, the final one.

Why deltas and not snapshots

The first reason is bandwidth. A snapshot of 1,000 levels a side is 16 KB a side as 16-byte binary levels, 32 KB for both sides, and about 60 KB for both as JSON with prices and quantities written as strings, around 30 bytes a level. A delta is 48 bytes. On an instrument changing 10,000 times a second, deltas cost 480 KB a second; a fresh snapshot after every change would cost 320 MB a second in binary and 600 MB a second in JSON, nearly 5 Gbit/s. The second reason is that a matching engine produces changes, not states. Each order it handles adds to, reduces or removes a level or a few, and the delta records what it just did; a snapshot must be assembled by walking the whole book. So the deltas are the product, and a snapshot is a convenience offered on request, to start or restart a copy.