Swapping components and injecting stalls

The swaps

Each swap replaces one piece, keeps the rest, and should move one histogram by an amount an earlier chapter predicts. Run them one at a time, paced, against the default.

Each swap also breaks one of the zero counters, and predicting which is part of the exercise: the tree and the channel allocate after warm-up, and the mutex can put a thread to sleep, which counts as a context switch.

The fourth swap breaks the measurement instead of a piece. --stall-every 1000 makes the strategy thread spin for 10 ms once a second, starting one second into the run, at the top of its loop before it drains. Nothing it has popped waits during the stall, so think, the service time, barely moves: the backlog’s one large batch afterwards adds tens of microseconds, against the stall’s 10 ms. In paced mode the feed thread keeps releasing messages on schedule, about 1,000 during each stall, and they wait in the ring; when the strategy returns it drains them all, and total, the response time, shows about 1,000 messages spread from 0 to 10 ms for each stall. Over 19 measured seconds that is about 19,000 samples in 1.9 million, 1%: p99.9 rises to about 9 ms and the max to about 10 ms, and p99 sits on the edge of the slow 1%, swinging between microseconds and milliseconds from run to run. That is the coordinated-omission demonstration, run on the real loop.

In flat mode the same stall shows as a single event. The feed thread has no schedule: finding the ring full, it waits, and stamps nothing until there is room. So the 10 ms is charged once, as one slow batch, to the messages that happened to be queued when the stall began, at most the ring’s 4,096 however long the stall lasts, and to none of the thousand the market would have sent meanwhile; a harness with one message in flight, the closed loop in its pure form, records it as exactly one sample. Run on the default file, flat mode may not meet a stall at all, since it is over in a tenth to a fifth of a second: it compressed 20 seconds of market into a fraction of one. Either way the flat histograms describe the loop truthfully and the messages not at all.

A swap should change the time and nothing else. To show it, run writes a decisions file, one line per batch: the sequence number of the update that closed it, the top the book showed after it, and the quote the quoter wanted. With no fills the inventory stays at zero, so the wanted quote is a function of the top alone. Two runs need not close their batches at the same updates, since where a batch ends depends on timing, and a slower variant makes fewer, larger batches. But wherever two runs decided at the same sequence number they had applied the same updates to the same book, so their lines must match exactly. Joining the two files on sequence number and comparing the rest is the proof, and the lab does it for each swap.

What is not in the capstone

There is no network: t0 is the moment the bytes would be handed to the program, not the moment they reached the card, so the kernel’s receive path, several microseconds in the networking chapter, is not in total. There is no real gateway, so no send path, no acknowledgements and no fills, and the quoter’s skew never moves a quote. There are no risk checks, the position limits and price bounds a real order passes before it leaves. There is one instrument, id 42 of 64, where a real feed handler routes 64 instruments to 64 books. There is no recovery: the replay has no snapshot to pass to on_snapshot, so a sequence gap takes the handler out of Live for the rest of the run, stops the quoting, and is counted. And there is no persistence, no order log and no restart. Each would add its own cost to the path and its own chapter to the book; none changes how the path is measured.