Building a quiet core

2.9 The quiet core

Each of the last six sections ended with what its mechanism does to a thread that never waits. This section turns those observations into what a process and a box must look like for such a thread, in the order the needs are met, and ties each to the mechanism it answers. A few of the answers have machinery of their own, how a tick is stopped, how interrupt lines are kept away, and that machinery is explained here, once. This is the only place in the chapter that reads as a list, and that is deliberate.

What the process does at startup

Before the first message, the process asks the kernel for everything it will ever need, in this order.

1. Each thread that matters pins itself to its own physical core with sched_setaffinity, shrinking its affinity mask to one core: the feed handler to 3, the order gateway to 4, the logger to 2, the control plane and everything else to 0–1. The core_affinity crate wraps the call. A thread does this first, because everything it touches next should land on its own NUMA node.
2. Each hot thread maps its big tables and touches every page of them, so that the frames exist and are local. The biggest tables come from a reserved huge-page pool with MAP_HUGETLB, so that the TLB covers them; the rest ask for transparent huge pages with madvise(MADV_HUGEPAGE) on the range, then verifies the backing; MADV_HUGEPAGE is a request and does not prevent later compaction or guarantee huge pages.
3. The process checks that mlockall(MCL_CURRENT | MCL_FUTURE) succeeds, keeping current and future mappings resident without MCL_ONFAULT. It needs ulimit -l unlimited or CAP_IPC_LOCK, and it costs memory: every thread’s stack and every later allocation becomes resident whether or not it is used.
4. Every collection the loop will touch is created at its final capacity (clear and truncate keep capacity; shrink_to_fit and drop do not); anything with a lifetime, an order, a pending message, a log record, lives in a pool, a Vec<T> of fixed size and a free list of indices; and the counting allocator is sampled after all startup work. From here no operation that allocates or releases storage runs inside the loop, including in the match arm that runs once in a thousand messages, and no thread of the process frees anything large, because returning mappings can cause shootdowns on cores sharing the address space. An Arc::clone does not allocate, but its shared atomic count has Chapter 3’s coherence cost.
5. The process opens /dev/cpu_dma_latency, writes a 32-bit zero and keeps the descriptor open, to request the strictest idle-exit latency while it runs; the supported idle driver and states determine the result.
6. The hot threads enter their loops and never block: one recv per batch of packets, and a spin with spin_loop on an empty queue. Threads off the hot path neither spin forever nor sleep deeply: they spin a few hundred iterations, yield a few, then park, which crossbeam::utils::Backoff encodes.

How the box is set up

The process can keep itself out of the kernel; it cannot keep the kernel off its core. That is the box’s configuration, and each line below answers one mechanism. Cores 3–7 are the isolated ones and 0–2 the housekeeping ones throughout.

isolcpus answers the load balancer: domain removes the listed cores from the scheduling domains it balances, so nothing is placed there unless pinned there, and the per-core kernel threads stay but have nothing to do. The managed_irq flag keeps kernel-managed interrupt lines (NVMe queues, and the NIC drivers that let the kernel spread their queues) off those cores wherever a queue’s mask also includes a housekeeping core; a queue whose mask holds only isolated cores still fires there. Recent kernels can do the same at runtime through cgroup v2, by writing isolated to cpuset.cpus.partition; the boot parameter is what most tuned boxes run today.

nohz_full answers the tick. On a core with exactly one runnable thread there is nothing to preempt for, so the kernel stops the tick while that is true. What it cannot stop, it replaces: RCU’s quiescent-state reports come from context tracking, which records every entry to and exit from user mode and treats user mode as one long quiescent state, at the cost of a little extra work on each system call, and the once-a-second accounting the tick used to do is run on a housekeeping core on the isolated core’s behalf. A few residual interrupts a second remain. rcu_nocbs answers the callbacks: it moves RCU’s deferred frees to rcuo threads on the housekeeping cores. Current kernels apply it to every nohz_full core automatically; tuned boxes spell both out anyway.

irqaffinity answers the interrupt steering: it sets the default steering mask for every interrupt line to the housekeeping cores. irqbalance, which would redistribute them, is stopped (systemctl disable --now irqbalance), and any line that still lands on 3–7 is re-steered by writing to /proc/irq/<n>/smp_affinity_list.

The hyperthread sibling of each isolated core shares its L1, L2, execution ports and branch predictor; a busy sibling slows the thread by 20–40% and evicts its lines. Either the sibling is isolated too and left idle, or SMT is off (nosmt, or the BIOS). /sys/devices/system/cpu/cpu3/topology/thread_siblings_list names core 3’s sibling.

The rest is sysctl and sysfs. kernel.numa_balancing=0 stops the placement sampling. transparent_hugepage/enabled and defrag at madvise confine THP and its compaction to the ranges the process asked for. swapoff -a removes swap-in; locking removes the rest. The frequency governor is performance (cpupower frequency-set -g performance, and energy_performance_preference=performance under intel_pstate), so that a core coming out of a lull does not also ramp its clock. intel_idle.max_cstate=1 on the command line is the coarse boot-time version of the C-state cap, and idle=poll an explicit polling idle loop. The latter differs from a PM QoS latency request; CPU idle documentation describes both. C-states, P-states and the SMI sources that can be turned off in the BIOS are turned off there.

Real-time priority comes last and is optional. On an isolated core SCHED_FIFO adds little, since nothing else is runnable, and protects against the one stray thread; on a shared core it starves ksoftirqd and can wedge the box, and the throttle deschedules a spinning FIFO thread for 50 ms every second unless sched_rt_runtime_us is set to -1, which is safe only once the core is isolated. The order is therefore: isolate first, then raise priority, and never raise priority on a core you share.

The table

Two caveats

Order matters: isolation before priority, population before locking, and all of it before the first benchmark, because every result taken on a stock box measures the box. And all of this is for bare metal. A virtual machine has a host underneath with its own tick, scheduler and interrupts; the guest’s settings do not bind it, and steal in /proc/stat is the host’s share of your tail. What is left after all of it, the residual interrupts, the SMIs, whatever perf you point at the core, is what Lab 2.1 measures; the kernel’s hwlat tracer (echo hwlat > /sys/kernel/tracing/current_tracer) spins in kernel mode with interrupts off and reports every gap it cannot account for, which on bare metal is almost always an SMI and is the tool for deciding whether a BIOS setting or a different server is needed.