The vDSO, TSC ticks, and multiple cores

Instant::now and the vDSO

Rust’s std::time::Instant::now() is, on Linux, a call to clock_gettime(CLOCK_MONOTONIC), and that would be a system call, an entry into the kernel and back, costing hundreds of nanoseconds. Normally it is not one. At the start of every process the kernel maps into its address space a small library of its own, the vDSO, the virtual dynamic shared object, which holds user-space versions of clock_gettime, gettimeofday, time and getcpu, and beside it a read-only data page, [vvar], that the kernel rewrites at every timer tick. The page holds the TSC reading at the last update, the time at that reading, and a multiplier and shift that turn ticks into nanoseconds. The C library’s clock_gettime jumps into the vDSO, which reads the TSC with a fence, reads the page, and returns the base time plus the ticks since the update, multiplied and shifted. To avoid reading the page while the kernel is half-way through rewriting it, the code reads a sequence number before and after and tries again if it changed, a seqlock, which lets any number of readers check a single writer without ever writing a line themselves. The whole call is 20–25 ns, about the cost of now().

That holds only when the clocksource is tsc, or another counter the vDSO can read from user space. The file /sys/devices/system/clocksource/clocksource0/current_clocksource names it. On a machine whose kernel did not trust its TSC it says hpet or acpi_pm, and on a virtual machine it may say kvm-clock or hyperv_clocksource_tsc_page, both readable in the vDSO, or, on older kernels under Xen, xen, which is not. When the vDSO cannot read the counter it makes the real system call, and every Instant::now() costs 300 ns to 1–2 µs, of which an HPET read is 0.5–1 µs by itself. Nothing in the program changes; only its clock got ten to a hundred times dearer.

Instant and raw ticks read the same counter on a machine whose clocksource is tsc. Instant adds the multiply and the sequence check, and in return gives nanoseconds that the kernel promises are monotonic on any machine, any architecture, any clocksource. Raw ticks are cheaper to store and carry no conversion inside the timed region, they come with rdtscp’s core number when wanted, and they need what Instant does not: a calibration, an invariant TSC, and counters that agree across cores. They are also x86 only; ARM processors have their own counter, read by a different instruction. The capstone uses raw ticks because it checks those conditions once at startup and then reads the clock millions of times a second.

Ticks are not cycles

A cycle is one tick of the core’s own clock, the rate at which it executes instructions, and that rate moves: 0.8–1.2 GHz when the core saves power, 3.4 GHz at base, 4.5–5.0 GHz in turbo with few cores busy, less when the package is hot. The invariant TSC does not move with it. An interval of 340 ticks on a 3.4 GHz TSC is 100 ns, always; in those 100 ns the core ran 500 cycles at 5.0 GHz, 340 at 3.4 and 120 at 1.2. A count of cycles comes from a different counter, the core’s cycle counter, one of the hardware counters that perf reads as the event cycles. Both counts are right, and they answer different questions: ticks say how long, cycles say how much work. A loop that gets 20% slower in ticks and not in cycles was slowed by its core’s frequency, not by its code.

The budget’s 4,000 cycles in 1 µs assume a core at 4 GHz, and a table of rdtsc differences labelled cycles, a common shorthand, holds ticks: on a 3.4 GHz TSC under a core running at 4.0 GHz it understates the cycles by 15%. To turn ticks into time, divide by the TSC frequency in gigahertz: 460 ticks at 3.4 GHz is 135 ns. Dividing by the core’s frequency instead gives 115 ns, a number that looks just as precise and is 15% wrong.

One clock or two

Every core has its own TSC, so a latency whose two ends are read on two cores, t0 on core 2 and t2 on core 3, subtracts readings of two counters. On one socket the cores’ counters are reset together and driven from one reference clock, so they agree to within a few ticks, about 1 ns, and being invariant they stay agreed. The kernel checks this at boot: as each core comes up, it reads its counter against another core’s in a tight loop, looking for a warp, a reading on one core earlier than a reading already taken on the other. Finding one, it prints a warning that marks the TSC unstable and moves to another clocksource, and from then on every clock_gettime in every process is a system call. Firmware can also leave the cores’ counters at different offsets; most current processors have a per-core adjustment register, IA32_TSC_ADJUST, that the kernel uses to put them back in line, and it says so in the boot log when it does.

Across sockets the counters are reset by the platform, not by one package, and usually agree; firmware bugs and hot-added processors are the exceptions, and the kernel’s boot check covers them too. A virtual machine adds a layer: the hypervisor may give each guest an offset, so its counter starts near zero, and a scale, so a guest moved to a host with a different TSC frequency sees its old rate, and a guest moved mid-run, or one whose virtual cores have different offsets, reads counters that disagree. Some hypervisors trap rdtsc altogether, so each read leaves the guest and costs a microsecond or more.

So a one-way latency between two cores is a time only when both cores read one synchronised counter, and only when each reading was taken on the core it names, which pinning, from the OS chapter, guarantees and rdtscp’s core number shows. A round trip timed on one core needs no agreement at all, since both ends read the same counter; that is how the offset between two cores is estimated, by stamping a ping-pong of one line in both directions and comparing the one-way times. A negative one-way latency is impossible on a sane clock and is the plainest sign of two clocks. Subtracted as u64, it wraps to about 1.8 × 10¹⁹ ticks, 170 years, which a histogram clamps to its top; a counter of intervals that came out backwards, which must read zero, catches it before it becomes a max of exactly 1 s.