7 · The number Draft
Assembling the capstone pipeline
On this page
7.7 The capstone: the book in one loop
The capstone is one program that runs the preceding chapters’ pieces together on a recorded feed and reports what the whole path costs, from a message’s arrival to the strategy’s decision. Each earlier chapter priced its piece alone: a decode of 10–20 ns, a ring push of 5–20 ns and a line crossing cores of 50–100 ns, an apply on the dense array of 2–5 ns, a quote of 5–20 ns. Added up, those are a prediction, not a measurement, because pieces that run together share caches, cores and a clock, and wait for one another. The crate is ch07-capstone/, with two binaries: gen, which writes the replay file, and run, which starts the threads, instruments them and prints the result. This section describes what the program is and what it should print; the loop at its centre is taken apart line by line after it, and the lab runs it.
The five pieces
Every piece already exists, built and measured in the chapter that owns it. The capstone writes almost no new code; it connects.
| Piece | In the capstone | Alone it costs | Chapter that owns it |
|---|---|---|---|
| Wire message and decoder | the 48-byte depth update, read from a byte slice into the 32-byte Update, nothing allocated | 10–20 ns | the networking chapter |
| SPSC ring | ring::<Stamped, 4096>() from feed to strategy, ring::<StampedOrder, 1024>() from strategy to gateway | 5–20 ns a push or pop; 50–100 ns a line crossing cores | the data-structures chapter |
| Feed handler and book | FeedHandler<ArrayBook>: ArrayBook::new(base), 65,536 levels a side, inside the handler whose on_update checks the sequence number, applies while its state is Live, and returns the new top when it moved | 2–5 ns an apply, plus a compare for the check | the book chapter |
| Quoter | Quoter::on_top, run once per batch when the top changed | 5–20 ns | the book chapter |
| Clock and histograms | now() (an lfence, then rdtsc), calibrate(), five Histogram<u64>s and a list of the worst 100 | 20–30 ns a clock read; 10–30 ns a record | this chapter |
Under the five sits the platform the OS and threads chapters prepared: each hot thread pinned to an isolated core, memory locked into RAM, and nothing on the path that sleeps, takes a lock or enters the kernel. The pieces’ own costs come to about 80–180 ns a message before any clock is read, most of it the line crossing cores. Whether the whole costs the sum of its parts, and if not, why not, is the question the capstone exists to answer.
Threads and cores
Three threads do the work. Each is pinned to its own core, each spins when it has nothing to do, and none ever sleeps:
The capstone: three cores and two rings. Select the diagram to open it at full size.
The feed thread on core 2 stands in for the networking chapter’s receive loop. It takes the next record from the replay file, waits for that record’s arrival time when the run is paced, decodes the 48 bytes into an Update, stamps it and pushes it. The strategy thread on core 3 owns the book: it drains the ring, checks and applies each update, quotes once per batch, and pushes the orders it decides on. The gateway stub on core 6 drains the order ring, stamps each order and counts it. It stands in for the thread that owns the order connection, so that the output ring has a consumer and its hop can be measured; it sends nothing.
One thread could do all of it and save the hop of 80–200 ns, and some production systems run the feed handler and the strategy on one thread for exactly that reason. Two things keep them apart here. The hop is one of the preceding chapters’ pieces, and the capstone exists to price the pieces together. And a separate feed thread keeps taking input while the strategy works through a burst, so a backlog waits in a ring the program sized and can watch, not in a kernel socket buffer it cannot see. The gateway is separate for a stronger reason: a real one spends 2–5 µs in the kernel’s send path on every write, the networking chapter’s number, and on the strategy thread those microseconds would sit in front of every later decision.
Cores 2 and 3 share an L3, which lscpu -e confirms, so a line crossing between them is served from the shared cache at 50–100 ns, not fetched from another socket at two to three times that. Core 6 is on the same socket. All three are isolated from the scheduler with isolcpus and nohz_full, with their interrupts steered elsewhere, as the OS chapter set out. Each hot thread pins itself before it allocates anything, so the pages of its book, its histograms and its buffers are first touched from the core that will use them.
The main thread is not pinned. Before the run it calls mlockall(MCL_CURRENT | MCL_FUTURE), so that every page the process has or will have stays in RAM; it maps, locks and touches the replay file, makes both rings and spawns the three threads. During the run it sleeps in join, and afterwards it collects the threads’ histograms and prints them. None of that is on the path, and an unpinned thread cannot land on an isolated core, because the scheduler places nothing there that was not pinned there. Main runs on the housekeeping cores, the ones left for the kernel and everything else, which is where it belongs.