Operating-system labs

2.10 Lab

Rules for every measurement

Chapter 1’s rules stand: predict first, release build, warm up, pin, performance governor, three runs, p50, p99 and max, never a mean. Add three things for this chapter. Record the kernel (uname -r), the command line (cat /proc/cmdline), whether the box is bare metal or a VM, and which core each run used. Run as a user who may use perf (kernel.perf_event_paranoid at -1 for tracepoints) and lock memory (ulimit -l unlimited). And take a snapshot of /proc/interrupts before and after every run; the deltas for your core are half the result.

Lab 2.1 — jitter histogram (ch02-os/src/bin/jitter.rs)

The exercise takes ordered TSC readings for ten seconds and records successive gaps into a preallocated histogram with resizing disabled. The gap includes the previous sample’s histogram work and loop overhead, not just the clock-read instruction. It prints p50, p99, p99.9, p99.99 and max in nanoseconds, and the number of gaps above 1 µs, 10 µs and 100 µs. Run it three ways: unpinned; pinned with taskset -c 3 to an ordinary core; and, if the box has one, pinned to an isolated nohz_full core. Take the /proc/interrupts column for the core before and after each.

Reason it out. A separate clock-read-only loop establishes timer overhead. In the histogram loop, p50 includes ordered reads, recording and loop work, so do not predict it from two bare rdtsc instructions. With the same instrumentation the central values should be similar across placements; rare interruptions show up in the counts and max. Everything interesting is in the counts and the max. Unpinned, expect the tick (250–1,000 a second at 1–5 µs), device interrupts, the occasional slice given to another process, and migrations, which each cost a cold cache: hundreds to thousands of gaps over 1 µs per second, p99.99 possibly in the microseconds, and a max of hundreds of microseconds to milliseconds depending on what else the box is doing. Pinned to an ordinary core, migrations go, and most other threads go with them if the box is idle; the tick stays, so expect 250–1,000 gaps a second of 1–5 µs, matching the LOC delta, and a max of tens of microseconds when a kworker runs. On an isolated nohz_full core, gaps over 1 µs should fall to single digits per second (the residual interrupts and little else) and the max to a few microseconds. Anything larger that no /proc/interrupts row accounts for is usually an SMI.

Pitfalls. Use an explicitly ordered timing sequence and compiler barriers; identical rdtsc instructions alone do not order surrounding work. Calibrate TSC frequency separately from core frequency. The TSC is invariant, so a frequency change does not alter the gaps in nanoseconds; the gaps it causes are real. In a VM the host’s scheduling appears as millisecond gaps with no counter to explain them. Ten seconds at 10 ns a sample would be a billion samples, but recording each one adds work; if the binary allocates in the loop or the histogram resizes, the lab measures the allocator. Write down p50, p99.99, max, the count above 1 µs and the LOC delta for each run.

Lab 2.2 — count the syscalls (ch02-os/src/bin/syscalls.rs)

The binary is Chapter 1’s stride walk with two extra flags: --log-pass prints one line per pass with println!, and --log-access prints every 1,000th access. Run the plain version under strace -c, then under perf trace -s, then under perf stat -e raw_syscalls:sys_enter,page-faults,context-switches,cpu-migrations. Then the two logging variants under each. Finally build it with the counting allocator and print the allocation count after each pass.

Reason it out. The plain walk should make a few dozen system calls in total: the mmap for 256 MB, a handful of writes for the output, exit, and nothing from inside the loop; page-faults should be 65,536 (256 MB over 4 KB) plus a few, all on the first pass. --log-pass adds one write per pass, trivial. --log-access adds a write every 1,000 accesses: at about 1 ns per access that is 1 µs of work and 2–5 µs of write, so expect the walk to run 3–6× slower than the plain version, more to a terminal than to a pipe, with the write count in the table to match. Under strace expect that variant to be a further 10–50× slower, because ptrace stops the process twice per call, and the plain variant to be barely affected, because it makes none: that difference is the lesson. The allocation counter should not move in any variant: println! formats straight into stdout’s buffer and allocates nothing, which shows that the cost of logging is the system call, not the heap. Replace the println! with a format! followed by write_all and the counter rises by one per line.

Pitfalls. strace -c without -f misses threads; perf trace needs root or a low perf_event_paranoid. Rust’s stdout is line-buffered even to a pipe, so every line is a write; wrapping stdout in a BufWriter changes the count and is a fair experiment of its own. Write down the syscall counts, the slowdown of each logging variant, and the slowdown strace adds to each.

Lab 2.3 — first touch (ch02-os/src/bin/first_touch.rs)

The binary maps 1 GB anonymously and times three passes: one store per 4 KB page (the first touch), the same pass again, and a full sequential write of the gigabyte. Flags: --populate adds MAP_POPULATE and times the mmap; --thp calls madvise(MADV_HUGEPAGE) on the region (transparent_hugepage/enabled must be madvise or always); --hugetlb maps with MAP_HUGETLB (512 pages must be reserved first); --mlockall calls mlockall(MCL_CURRENT | MCL_FUTURE) before the mmap. Run each under perf stat -e page-faults,dTLB-store-misses.

Reason it out. The plain first pass is 262,144 faults at 0.5–1 µs: 130–260 ms, and page-faults should report that number. The second pass stores to 262,144 resident pages; translations exceed the TLB, while the 16 MiB of touched cache lines may fit in L3, but the addresses are independent and overlap, so expect 5–10 ms, 20–30 ns a page, and the ratio between the two passes, 20–50×, is the cost of faulting. The full write is bandwidth: 1 GB at 10–20 GB/s, 50–100 ms. With --populate the population work moves into the mmap call, which should take somewhat less than the first pass did since the kernel walks the range without 262,144 traps, and the first pass should then equal the second. If the whole region is actually backed by 2 MiB pages, it needs only 512 initial mappings; each fault zeroes 2 MB, 100–200 µs, so the first pass is 50–100 ms and is now zeroing bandwidth rather than trap overhead, and the second pass’s dTLB-store-misses should fall by roughly 512×. --mlockall should move the cost into the mmap like --populate and show Locked equal to the region in smaps_rollup.

Pitfalls. With enabled at never the madvise is accepted and ignored, and AnonHugePages in /proc/<pid>/smaps_rollup is how you tell. khugepaged may collapse the plain run’s pages into huge pages behind your back, so a rerun can be faster for no visible reason. --mlockall fails on the lock limit if ulimit -l is low; check the return value. Inside a VM with nested paging a TLB miss walks two page tables and costs two to three times as much. Write down the fault count and the three pass times for every variant, and the cost per 4 KB page implied by each.

Lab 2.4 (stretch) — wake latency (ch02-os/src/bin/wake.rs)

Two threads pinned to two different physical cores pass a token back and forth a million times. --spin passes it through an AtomicBool with spin_loop; --park through thread::park and unpark, a futex each way; --sleep by polling the atomic with thread::sleep(1 µs) between polls. Each variant runs with and without --c0, which holds /dev/cpu_dma_latency at zero for the run. The binary reports round-trip p50, p99 and max.

Reason it out. Spinning moves one cache line each way, 50–100 ns a hop: a round trip of 100–200 ns. Parking costs a futex call, the reschedule interrupt to the far core, its C-state exit and a scheduler pass: 3–10 µs a hop with C-states capped, so 6–20 µs a round trip; without --c0 the far core can drop into C6 while the token is away, and the p99 should show 100–200 µs hops. Sleeping can add roughly 50–60 µs a hop when wake-ups use most of the default slack; this is a prediction to check, not a timer-slack lower bound. Then set PR_SET_TIMERSLACK to 1 in the sleeping variant and watch the hop drop to a few microseconds and stop there, because the wake-up path underneath is the same as parking’s.

Pitfalls. Hyperthread siblings share an L1, so a spin between siblings looks faster than any real cross-core hop; check thread_siblings_list. --c0 needs permission to open /dev/cpu_dma_latency. A VM offers no control over C-states at all. Write down the three round-trip p50s and p99s with and without --c0, and the ratio of each to the spin.