1 · Memory and the CPU Draft

Hardware review and further reading

6 min read

On this page

1.11 What to be able to say afterwards

  • At 4 GHz an L1 hit costs about 4 cycles and a DRAM miss about 360, so a 1 µs software budget affords only about eleven dependent DRAM misses.
  • Count cycles for anything inside the core and nanoseconds only at the packet edges: the clock moves, a clock read costs 20–30 ns, and the TSC ticks at a nominal rate, not the core’s.
  • The cache moves 64-byte lines, so an operation’s cost is set by the number of distinct lines it touches, not the number of bytes it reads.
  • A 48 KB L1 is 64 sets of 12 ways; addresses 4 KB apart share a set, so a power-of-two stride can miss in a cache that is mostly empty.
  • Out-of-order execution hides L1 and L2 latency and overlaps independent misses, but it cannot hide a chain of dependent DRAM misses.
  • Layout should follow access: struct of arrays for scans over one field, array of structs for updates to one element, and hot fields split from cold ones.
  • Threads writing different variables in one cache line serialise through the coherence protocol, so each line should have one writer and few cross-core readers.
  • A predictable branch is nearly free, a 50/50 branch costs 8–10 cycles on average, and branch-free code helps only in the second case.
  • On a two-socket machine memory is placed by first touch, so a hot thread must be pinned and must initialise its own data.
  • Latency estimates and examples depend on the target; check Rust’s layout guarantees before treating a measured size as a contract.
  • Every performance claim needs a prediction first and a pinned, release-build, percentile-reported measurement second.

1.12 Quiz

  1. Your hot path performs six dependent DRAM misses per message. What does that cost when the memory is local, and when it is on the remote NUMA node? How does each compare with a 1 µs software budget?
  2. Take #[repr(C)] struct S { a: u8, b: u64, c: u8 }. What are size_of::<S>() and align_of::<S>()? What would Rust’s default layout give? How much memory does an array of one million take under each?
  3. A feed handler executes if side == Side::Bid on 5 million messages per second, with bids and asks in random order. The mispredict penalty is 18 cycles at 4 GHz. How much CPU time per second goes to mispredicting this branch? Is it the first thing to fix?
  4. Per-instrument state lives in a Vec<InstrumentState> whose elements are exactly 4,096 bytes. On every tick you read best_bid at offset 0 for 40 instruments. The L1d is 48 KB, 12-way, with 64 sets. What happens, and what are two fixes?
  5. Lab 1.1 shows about 15 ns per access at a 4 KB stride, where every access misses to DRAM. Lab 1.4 shows about 120 ns per node for the shuffled list, where every access also misses to DRAM. Explain the eightfold difference.
  6. An SPSC ring has head and tail on separate padded lines, but the producer reads the shared tail on every push. At 10 million messages per second, with a cross-core line transfer of about 70 ns, estimate the producer’s coherence cost per second. Then estimate the best batching case with a cached tail on a 1,024-slot ring, and explain why a consumer keeping up does not guarantee it.

Answers

  1. Local: 6 × 90 ns = 540 ns, over half the budget before any computation. Remote: 6 × 175 ns ≈ 1,050 ns, over budget on memory alone. The same code on the wrong socket breaks the design, which is why first-touch placement matters.
  2. With repr(C) on this x86-64 target: a at 0, 7 padding bytes, b at 8, c at 16 and 7 tail-padding bytes, so size 24 and alignment 8. A typical default Rust layout is 16 bytes, giving 24 MB versus 16 MB for a million elements; verify it with size_of, because the default layout is unspecified.
  3. Half mispredict: 5 M × 0.5 × 18 = 45 million cycles, about 11 ms per second, roughly 1% of a core, or about 2 ns per message. That is small next to one 90 ns DRAM miss, so fix the misses first. Mispredicts dominate when a branch runs many times per message, as in a byte-by-byte parser.
  4. Elements 4,096 bytes apart share address bits 6–11, so all 40 lines map to one 12-way set. Each is evicted before reuse, so every read misses L1 although the lines total 2.5 KB. Pad the struct to 4,160 bytes, or move best_bid into its own dense Vec<u64>.
  5. Stride addresses come from a loop counter, so the core keeps eight or more misses in flight: roughly 120 ns ÷ 8 ≈ 15 ns each. Each list address comes from the previous load, so one miss is in flight at a time, each paying full latency plus a TLB miss. Lab 1.1 measures throughput under memory-level parallelism; Lab 1.4 measures latency.
  6. If every read needs a transfer, the upper-bound estimate is 10 M × 70 ns = 0.7 s per second before overlap. If each refresh grants 1,024 pushes, about 9,766 refreshes cost 0.68 ms per second. The second figure is a batching scenario, not a guarantee from capacity or a consumer merely keeping up; actual occupancy and interleaving decide the saving. Slot and publication traffic remains.

Further reading