Bypass trade-offs and the cloud

Cloud reality

In a cloud the program does not own a card. A virtual machine is a guest run by a hypervisor, the host software that divides one physical server among several, and its network card is the hypervisor’s invention. The older kind is paravirtual: the guest’s driver puts packets on rings in shared memory, and software on the host forwards them, adding its scheduling and copying to every packet. The faster kind, where offered, is SR-IOV, single-root I/O virtualisation: one physical card presents itself as many virtual functions, each a PCIe device with its own queues, and the hypervisor hands one to the guest, whose driver then works the rings with no host software on the per-packet path. The large clouds build their own cards this way, AWS’s Elastic Network Adapter (ENA) the best known; a DPDK poll-mode driver exists for ENA-class cards, and AF_XDP works where the driver supports XDP. Onload does not, because the public clouds do not offer its vendor’s cards.

What bypass cannot remove in a cloud is the cloud. Whatever card the guest sees, the host, or a processor on the card itself, still carries out the cloud’s virtual network for every packet, wrapping it in an outer header, applying the security rules and enforcing the instance’s limits, and the guest’s virtual cores are themselves scheduled by the host: some microseconds a packet, more on a busy host, that no guest setting can remove. Against them stands the round trip between availability zones, 0.3–1 ms, of which the kernel’s 5–20 µs is between 0.5% and 7%. Busy-polling recovers most of that; bypass would recover a further 1–4 µs, at most about 1%. Many crypto venues run in a public cloud region, so for a crypto desk this is the common case, and the priorities follow: the venue’s own zone first, busy-polled kernel sockets second, bypass a distant third.

What it costs

The engineering cost comes first and lasts longest. DPDK or AF_XDP for a UDP feed is a parser of three headers and a ring loop. For a TCP connection to a venue it is a TCP stack to write or adopt, test against the venue’s peculiarities and maintain for as long as the connection exists: months before the first order, and no end after it. Onload costs the least code, an LD_PRELOAD and a few environment variables, and the largest hardware commitment, the vendor’s cards in every production box and every spare. Polling costs CPU, but a core can service several queues or connections; one feed and one order connection do not inherently require two dedicated cores. DPDK buffer-pool memory depends on queue counts, ring sizes and buffer headroom; a gigabyte is a possible configuration, not a minimum. And every option gives up some of the tools above, a cost paid on the day something goes wrong.

Real parts vary by a factor of two either way; the shape does not. The send direction has the same shape, from the kernel’s 2–5 µs to about 1 µs. And every row sits on the same floor of PCIe, card and wire, about 1 µs each way: bypass removes software, never distance.

Worked example: the same program in the rack and across a sea

Bypass pays when the kernel is most of the time that differs between a program and whoever it races. Take a program co-located with a venue, in the same rack as the venue’s gateway with one switch between, reading a feed whose updates matter in single-digit microseconds. The cards, the wire and the switch cost about 1 µs each way, and every firm in the room pays the same. What differs is the program’s own tick-to-trade, from the update’s last bit at its card to the first bit of its order on the wire. With kernel sockets and a sleeping reader that is a receive of 6–22 µs from the wire, a decision inside the 1 µs budget and a send of 2–5 µs to the wire: 9–28 µs, of which all but the decision and 2–3 µs of card time is the kernel’s. Busy-polled, it is 3–7 + 1 + 2–5 = 6–13 µs, the kernel still more than half of it. With bypass on both the feed and the order connection it is about 1 + 1 + 1 = 3 µs. A competitor with bypass reaches the venue 3–10 µs ahead of the best kernel path on every update, and when both go for the same resting order, the venue fills whichever arrives first. Here the kernel is most of the budget, and bypass is the difference between competing and not.

Now take a venue 35 ms away over the internet. A depth update has spent 35 ms in fibre before the card sees it, and the kernel’s 5–20 µs on top is at most 0.06% of its journey. The whole reaction, update in and order back out, is a 70 ms round trip, and the program’s kernel share of it, 5–20 µs of receive and 2–5 µs of send, is under 0.04%. Busy-polling the kernel socket brings the receive to 2–5 µs and recovers most of that for the price of a setting and a core; bypass would recover a further 1–4 µs, under 0.01% of the round trip, at the whole engineering price above. One lost segment on the same path costs a 70 ms round trip to recover, thousands of times the kernel’s entire share, and a competitor whose server sits in the venue’s own region is 70 ms ahead on every round trip, which no stack recovers. Here the kernel is a rounding error, and the work is in where the machine sits and in keeping TCP from stalling.

Seeing it

Each kind of bypass leaves a mark where the ordinary tools can find it. dpdk-devbind.py --status, from DPDK’s tools, lists every network device in two groups: those on a DPDK-compatible driver, with drv=vfio-pci and the kernel driver they left in unused=, and those still on a kernel driver; a card in the first group is invisible to ip link and ethtool, and its counters are read with dpdk-proc-info or from the program’s own statistics. onload_stackdump lots prints each Onload stack’s counters, among them how often it was polled from user space and how often an interrupt made the kernel step in; on a stack whose threads spin, the second barely moves. For AF_XDP, ip link show dev <iface> prints xdp and a prog/xdp id when a program runs in the driver, or xdpgeneric when it runs after the kernel has built its sk_buff, which gives away most of the gain.

The traffic itself shows in the gap between two sets of counters. The card’s own, per queue in ethtool -S, count every frame whoever takes it; the kernel’s, nstat’s IpInReceives and TcpInSegs and the rows of /proc/net/softnet_stat, count only what the kernel walked. On a box running Onload or AF_XDP, the card’s counters for the bypassed queues climb while the kernel’s stand still. Finally, the ladder of Lab 4.1 has a bypass rung when the hardware allows one: on two boxes with cards Onload supports, ./ladder --tcp <host> run under onload --profile=latency at both ends measures the rung the ladder puts at 3–6 µs.