Inlining and bounds checks

5.8 The compiler’s part: layout recalled, inlining, bounds checks, uninitialised memory, and reading what it made

Chapter 1 settled the rules for laying out the hot loop’s bytes, and they stand. Every type has a size and an alignment, reported by size_of and align_of, and the compiler pads between fields to satisfy them; Rust reorders a struct’s fields to minimise that padding unless the struct is #[repr(C)], which every struct laid over wire bytes must be; fields a scan reads across many elements go in their own arrays, struct-of-arrays; Option<u64> is 16 bytes where Option<NonZeroU64> is 8; and anything two threads write goes on its own line with #[repr(align(64))]. What is new here is the compiler’s part: the instructions it chooses, each with a price and each readable in its output.

Inlining

To inline a function is to copy its body into the place that calls it, so that no call happens at run time. A call and its return cost 1–5 ns, 4–20 cycles, and the caller must save to the stack any register the callee may overwrite and reload it after, which is called spilling. The larger cost falls on the optimiser, for which a call is a wall: it cannot keep a value in a register across the call, cannot move work out of a loop that contains it, and cannot vectorise that loop, that is, turn it into instructions that each work on several elements at once. Inlined, the call costs nothing, and what the optimiser knows at the call site, a constant, a length, a range, applies inside the body.

The compiler inlines a great deal unasked: small functions and functions with one caller within a crate, whenever its cost model says it pays; generic functions always, since each is compiled afresh for its concrete types in the crate that uses it; and, since Rust 1.75, the smallest non-generic functions, those that call nothing, even across crates. What is left is a small non-generic function in another crate that does call something. Crates are compiled separately, so the caller sees the compiled function, not its body, and emits a call: a risk crate’s clamp_qty(q, max), which logs when it clamps, compiles to call dep::clamp_qty in the strategy’s crate on rustc 1.97. Marked #[inline], which ships the body with the compiled crate, it vanishes into its caller. #[inline(always)] asks more firmly and is still a request, since the Rust reference calls every form a hint the compiler may ignore; #[inline(never)] asks the reverse.

The release profile can give the optimiser every body instead:

[profile.release]
lto = "fat"          # rustc -C lto=fat
codegen-units = 1    # rustc -C codegen-units=1

Link-time optimisation, LTO, postpones the last optimisation pass until all the crates are linked, so a function from another crate inlines as if local. Codegen units are the pieces Cargo splits a crate into to compile in parallel, 16 by default for a release build, and the optimiser inlines between them only through a lighter LTO applied by default; one unit hands it the whole crate. With both set, clamp_qty inlines without the attribute, for the price of a slower build.

Inlining has a cost of its own. Each inlined copy is more machine code, and a hot path grown past the L1 instruction cache misses on its own instructions, each miss costing what a data miss to the same level does, 3.5–4 ns from L2 and 10–40 ns from L3. #[inline(always)] on a large function called from twenty places makes twenty copies.

Bounds checks

Rust promises that slice[i] never reads outside the slice, and keeps the promise at run time: every index compiles to a compare of i against the length and a conditional jump to a call of core::panicking::panic_bounds_check, which stops the thread. That pair is a bounds check. In correct code the jump is never taken, the branch predictor learns as much, and the check costs 0–1 cycle; mispredicted, on its first run or after a lull has retrained the predictor, it costs 15–20. The larger cost is what it forbids. A loop that may panic at pass 37 must have done exactly passes 0 to 36 first, with their effects in memory for whoever handles the panic, so the compiler may not unroll it, doing several passes’ work per trip round the machine-code loop; may not vectorise it; and may not keep a running total in a register, unless it can move the check out of the loop altogether, as it often can. Each check also adds two instructions and a panic block, widening the code.

The optimiser deletes any check it can prove never fails, and removing checks means handing it the proof. Iterating does: for p in slice and a.iter().zip(b) carry their own ends and have no index to check. Converting a slice to a fixed-size array once does: let hdr: &[u8; 48] = buf[..48].try_into().unwrap();, as the networking chapter wrote for its 48-byte update, checks the length once, and every constant index into hdr is then known to be in range. One assert!(p.len() >= n) before a loop over 0..n does, since it makes each check inside redundant. chunks_exact(4) does, handing out pieces of exactly four whose indexes 0 to 3 need no check. The last resort is get_unchecked(i) inside unsafe: the bare load, with an out-of-range i undefined behaviour, the condition a data race was shown to produce, in which the program means nothing at all. Use it only after measuring that the check costs something, and write the invariant beside it in a // SAFETY: comment: every InstrumentId is below the length of Positions::by_id because the Interner assigned them, 0 to N − 1, at startup.