2 · The Linux platform Draft

Operating-system review

9 min read

On this page

2.11 What to be able to say afterwards

  • An operating system is one program that owns the hardware and runs it for all the others, and its services to a process are fairness, isolation and the illusion of a private machine; a process uses it by asking, and the ask has to raise the core’s privilege level, which is where kernel space and user space come from.
  • The ordinary paths into Linux are a syscall, an interrupt or an exception (including page faults); each runs kernel code through the thread’s own caches, and each can return into a different thread.
  • A trivial system call costs 100–300 ns and one that does work costs 1–2 µs, plus the cache lines it displaces; vDSO-supported calls can avoid that crossing.
  • The scheduler keeps one run queue per core and a promise of fairness, and the tick exists because a thread that never blocks never gives the core back; the tick costs 1–5 µs, and a preemption costs not the switch but the other thread’s slice and the cold caches afterwards.
  • The kernel has threads of its own on every core, and work it queues on a core, RCU callbacks among it, lands there whether or not the core has time.
  • A kernel interrupts rather than polls because most devices mostly have nothing to say; the top half is short and the softirq behind it is the real work, and when there is too much of it the network stack becomes an ordinary thread on the run queue.
  • The memory manager backs a page on first touch, at 0.5–1 µs, because processes ask for far more than they use; it can take frames back, it needs contiguous memory for huge pages and will compact to get it, and changing a mapping interrupts every core running the process.
  • 4 KB pages give the TLB 6–8 MB of reach and 2 MB pages give it gigabytes.
  • The allocator is a user-space cache in front of the kernel’s pages, fast in the steady state and a system call at its edges, and a large free on any thread reaches every other thread’s core.
  • A blocked thread wakes in 5–20 µs at best and 100–200 µs out of C6, and the idle governor goes deeper the longer the wait, so the wake-up is slowest for the message that ends a lull; a 1 µs sleep may last tens of microseconds because of allowed timer slack and wake-up overhead.
  • Every one of those costs is a service done well for a thousand ordinary threads, and for one thread with a microsecond to spend the answer is the same each time: ask once at startup, keep the core to yourself, and never tell the kernel you are waiting.
  • On a VM none of the configuration binds the host, and steal time is the host’s share of the tail.

2.12 Quiz

  1. Why does the kernel need a timer tick at all, and why can it be stopped on a core that is running exactly one thread? Name two things the tick used to do on that core that must then be done some other way.
  2. A colleague says “the NIC interrupt is pinned to core 0, so the network stack never runs on our hot core 3”. Using the top-half and bottom-half split, say what part of that is right, what can still put the network stack on core 3, and which counter would show it.
  3. A hot loop handles 400,000 messages a second and logs one line per message with println! to a pipe, at about 1.5 µs per write. What fraction of each second does the core spend inside write, and what happens to the queue behind the loop? What does the alternative cost per message?
  4. A core runs a pinned but not isolated hot thread; the kernel is built with HZ=250, each tick costs 3 µs, and the loop does one message per microsecond. How much time per second goes to ticks, and roughly which percentile of the per-message histogram first shows them? Repeat for HZ=1000.
  5. A 2 GB order table is allocated with Vec::with_capacity at startup and first written during the first burst of the day. How many faults does the burst pay, what do they cost in total, and what are three ways to move that cost to startup?
  6. A polling loop calls thread::sleep(Duration::from_micros(1)) between polls and a trace shows 55 µs between iterations. Explain the 55 µs. What does prctl(PR_SET_TIMERSLACK, 1) change, and what floor remains?
  7. A hot path makes random reads into a 40 MB table on a core whose L2 TLB holds 1,536 entries. With 4 KB pages, about what fraction of reads miss the TLB, and what does each miss add? What changes with 2 MB pages?
  8. The hot thread on core 3 allocates nothing and frees nothing. Once a second a helper thread on core 1 drops a 40 MB Vec. What does core 3 experience, where does it show up, and what are two fixes?

Answers

  1. The tick exists so that the scheduler gets control of a core at least HZ times a second, because a thread running in user mode never hands the core back on its own; without it a loop could hold a core forever. With exactly one runnable thread there is nothing to preempt for, so there is no decision to make and the interrupt can stop. Two things it did that must move: RCU’s grace-period detection, which relied on the tick to see the core pass through a quiescent state and now uses context tracking of user-mode entry and exit; and the once-a-second scheduler accounting, which is run on a housekeeping core on the isolated core’s behalf.
  2. RPS or RFS can steer the receive protocol work to core 3 even when the NIC interrupt is on core 0; busy polling on core 3 can also run the driver there. Check NET_RX in /proc/softirqs along with IRQ affinity and RPS/RFS settings. The right part: the top half, the handler that acknowledges the NIC, runs on core 0, and so does the NET_RX softirq that follows it on the same core, so the receive path for that NIC lands on core 0. What can still reach core 3: any other interrupt line still steered there, the reschedule interrupt when something wakes a thread on core 3, and ksoftirqd/3, which exists on core 3 and runs whatever softirq work is raised there (timers, RCU, and any device line that was not re-steered). /proc/interrupts column 3 shows the device rows; /proc/softirqs column 3 shows which softirqs ran there; nonvoluntary_ctxt_switches on the hot thread shows whether anything took the core.
  3. 400,000 × 1.5 µs = 0.6 s of every second inside write, before formatting and the stdout lock: 60% of the core. The loop’s budget per message is 2.5 µs and write alone uses 1.5 of it, so any burst above the average rate backs up and the queue drains only when the market goes quiet. A fixed-size record stored into a preallocated ring drained by a logger thread costs 5–10 ns per message on the hot path.
  4. At HZ=250, 250 ticks × 3 µs = 0.75 ms per second, 0.075% of the core; 250 of a million iterations are hit, so the ticks appear above p99.9, around p99.975, each as an iteration of about 4 µs instead of 1. At HZ=1000, 3 ms per second, 0.3%, and 1,000 iterations in a million, so the ticks appear at p99.9 exactly. Either way the p99 is clean and the max is not.
  5. 2 GB ÷ 4 KB = 524,288 faults at 0.5–1 µs: 0.25–0.5 s spread across the burst, 0.5–1 µs added to every message that touches a new page. Three ways: write one byte per page from the hot thread at startup; MAP_POPULATE on the mapping; mlockall(MCL_CURRENT | MCL_FUTURE) before the allocation. With 2 MB pages the count becomes 1,024 and each is 100–200 µs of zeroing, still at startup.
  6. nanosleep arms a high-resolution timer, and the kernel may delay its expiry by the thread’s timer slack, 50 µs by default for a normal thread, to batch wake-ups; add the nanosleep call itself, the scheduler pass and the C-state exit, and 55 µs is plausible, though not guaranteed. PR_SET_TIMERSLACK to 1 ns removes the 50 µs; the floor that remains is the wake-up path, a few microseconds with C-states capped and 100 µs or more out of C6. Timer slack is not applied to FIFO threads, but syscall, scheduling and idle-exit costs remain.
  7. 1,536 entries × 4 KB is 6 MB of reach against 40 MB, so about 85% of random reads miss the TLB, and each miss adds a page walk of 20–40 cycles when the page-table entries are cached and a few hundred cycles when they are not, before the data access itself begins. With 2 MB pages the table is 20 translations, all resident, and the TLB misses disappear; the data cache misses remain.
  8. If the helper’s buffer is a separate mapping, dropping it may call munmap and trigger TLB invalidations on cores using this address space, including core 3. That can add a gap followed by translation misses; the TLB interrupt counters and jitter histogram help identify it. Size makes that path plausible under ordinary glibc settings, but does not prove it: trace the allocation and free. Reuse the buffer (clear retains capacity), or use a preallocated pool; allocator threshold tuning does not prevent every shootdown.

Further reading