TLS placement and warm connections

Where the decrypt runs

The record processing can run in three places; 300–500 ns is an illustrative small-record estimate, to measure with the chosen cipher and library. The usual place is the program’s own thread, using a user-space library, rustls in Rust or OpenSSL through a binding, over an ordinary non-blocking TCP socket. A recv copies ciphertext into a buffer the library owns, and the library finds the complete records, decrypts each, checks its tag and hands over the plaintext, still in L1 when the parser reads it. The kernel sees only TCP.

Two details matter at this scale. Buffered TLS APIs may allocate and copy plaintext records; the behaviour depends on the exact library, version, backend and API. Check allocation counts after warm-up rather than treating a whole rustls release as having one unavoidable allocation per record. rustls also has an unbuffered API for caller-managed buffers. BufRead::fill_buf can avoid a further reader copy where supported, but does not by itself prove the record path allocation-free.

The second place is the kernel. Kernel TLS, kTLS, moves the record layer into the kernel; the handshake stays in user space, because the kernel has no code for certificates. After the handshake the program takes the session’s keys from its library (rustls offers dangerous_extract_secrets; OpenSSL 3 can set kTLS up itself) and hands them over with two setsockopt calls, TCP_ULP (the socket’s upper-layer protocol) set to "tls" and TLS_RX with the cipher, the key, the nonce material and the record sequence number. From then on recv returns plaintext. In the softirq, as segments arrive, the kernel’s stream parser reads record headers and holds each record’s packets until all of it is in, and only then does recv find data or epoll report the socket readable. The decrypt itself, in the software path, runs inside recvmsg, in the calling thread, and where it can (a large enough buffer and, for TLS 1.3, the TLS_RX_EXPECT_NO_PAD option) it goes from the kernel’s packet buffers straight into the program’s buffer, saving the copy of the ciphertext. A record that is not application data, such as a session ticket, comes back from recvmsg with a control message for the program to pass to its library. The saved copy is worth 1–2 µs for a 16 KB record and about 10 ns for a 200-byte one, and most on the send side, where sendfile encrypts from the page cache without the file entering user space, which is why file and video servers use kTLS.

The third place is the card. Some NICs decrypt as packets pass and hand the kernel plaintext marked as decrypted. A lost or reordered segment makes the card lose its place, and records fall back to software until it finds a header again, so its help disappears exactly when the path is already in trouble.

For one connection carrying small records, user-space TLS keeps record processing beside the parser and works with a user-space TCP stack. kTLS can save copies and exploit NIC offload, while retaining kernel record processing and control-message handling. Neither location is universally faster: compare end-to-end latency, allocation behaviour and CPU use for the actual record sizes and backend. The kernel TLS documentation describes the supported paths.

Keeping connections warm

Every handshake is paid again on every new connection, so a program connects once, early, and keeps the connection alive, and four mechanisms keep the cost of a new connection away from the moments that matter. The first is session resumption, which a rustls ClientConfig does by default: reconnecting through the same Arc<ClientConfig> skips the certificate work without further code.

The second is the pre-connected spare: a second connection taken through TCP, TLS and the WebSocket upgrade at startup, then subscribed to the same stream or left idle. A subscribed spare doubles the traffic and the decrypt work, and in return the program reads both and drops duplicates by sequence number, so a slow segment on one path hides behind a fast copy on the other and a failover loses nothing. An idle spare costs only its heartbeats.

The third is the heartbeat. Every NAT gateway (a box that rewrites the addresses of connections leaving a private network), firewall and load balancer (a box that spreads connections over several servers) on the path drops its entry for a connection idle longer than a timeout of seconds to many minutes, rarely documented, and the next segment either way is then discarded or answered with a reset. TCP’s keepalive, off by default and silent for two hours before its first probe, is far too slow, so WebSocket’s ping and pong frames do the job, and many venues close connections that fail to answer. A client’s frames are masked, each byte XORed with a 4-byte key it picks, so a pong costs a few nanoseconds of XOR on top of its send. Missing a required heartbeat response by its deadline triggers the application’s failure policy; a quiet market alone is not proof of a dead connection.

The fourth is how the program reconnects. When a venue restarts or drains a gateway, thousands of clients reconnect in the same second, a reconnect storm: the venue’s TLS terminators saturate, accept queues fill, and clients whose attempts time out retry together. A client that waits a random interval that grows with each failure, exponential backoff with jitter, spreads the load and on average gets through sooner.

Even a well-behaved reconnect to a venue 35 ms away costs four round trips before the first update, TCP, TLS 1.3, the WebSocket upgrade (an HTTP request answered with 101 Switching Protocols) and the subscription, and then a snapshot of the whole book over HTTPS, since the book has been stale since the old connection died.

What it costs

Once per connection to a venue 35 ms away, TLS costs 140 ms with TLS 1.3 or 210 ms with TLS 1.2 before the first application byte, plus a few hundred microseconds to a few milliseconds of CPU. Every record adds 22 bytes on the wire with TLS 1.3, 29 with TLS 1.2 and GCM: 18–23 ns at 10 GbE. A 200-byte record costs 300–500 ns to decrypt with AES-128-GCM and AES-NI, of which 100–300 ns is fixed; a 16 KB record pays 8–16 µs of cipher and the fixed part once; Measure ChaCha20-Poly1305 separately rather than assuming a fixed ratio. In the sequential reconnect example, the connection costs four round trips, 280 ms, and a snapshot, 70 ms on a connection already open or 210 ms on a fresh one: 350–490 ms, 350,000–490,000 budgets, plus however long it took to notice. Indirectly, a record holds its first message until its last segment arrives, and the decrypting thread keeps the cipher’s code and key schedule, a few kilobytes, in its caches.

Worked example: a WebSocket depth stream

A strategy thread on core 4 busy-polls a TLS 1.3 WebSocket connection to a venue 35 ms away. Each depth update, the 161-byte JSON object of the previous section’s worked example, arrives as one WebSocket text frame in one TLS record in one TCP segment. A WebSocket frame header is 2–14 bytes: a FIN bit, an opcode (1 text, 2 binary, 9 and 10 ping and pong) and a 7-bit length, where 126 means a 2-byte big-endian length follows and 127 an 8-byte one, plus a 4-byte mask key on frames from the client. So the frame is 4 + 161 = 165 bytes, the record 5 + 165 + 1 + 16 = 187, and the Ethernet frame 14 + 20 + 32 + 187 + 4 = 257.

Follow the update from the last bit on the wire. The card puts it in DRAM and raises its interrupt within 1–2 µs, and the kernel’s receive path delivers it to recv in 2–5 µs with the thread busy-polling, 5–20 µs had it been asleep in epoll_wait. rustls decrypts 166 bytes and checks the tag in 300–500 ns. The framer reads the WebSocket header in about 5 ns: FIN, opcode 1, length 126, then 161 from the next two bytes read big-endian, and no mask, because server frames are unmasked. The JSON parse takes 1–3 µs. From the last bit on the wire to a struct the strategy can read is 1–2 + 2–5 + 0.3–0.5 + 1–3 µs, about 4.3–10.5 µs, and the program’s own share after recv returns is 1.3–3.5 µs, of which TLS is 0.3–0.5 µs and the JSON the rest.

Now the same stream in the venue’s binary encoding. The 48-byte update rides in a binary frame with a 2-byte header, 50 bytes; the record is 5 + 50 + 1 + 16 = 72 bytes and the frame on the wire 142, 114 ns at 10 GbE. The decrypt covers 51 bytes, 25–50 ns of cipher plus 100–300 ns fixed, about 125–350 ns, then 5 ns of header and 10–20 ns of parse. The program’s share falls to 0.15–0.4 µs, inside the budget, and the order of costs has turned over: with JSON the parse was the largest thing the program did; with binary, TLS’s fixed per-record cost is, and the kernel’s 2–5 µs outweighs everything in user space together.

Then a reconnect comes at the wrong moment. Fifty milliseconds into a market move, with updates up from 2,000 to 20,000 a second, the venue’s load balancer drains a gateway and closes the connection. The thread learns at once, because read_tls returns 0, and reconnects with a resumed TLS 1.3 session but no early data: TCP, TLS, the upgrade and the subscription, 4 × 70 = 280 ms, then the snapshot over an HTTPS connection kept open for the purpose, 70 ms more. The strategy is blind for 350 ms in the middle of the burst, through 7,000 updates, while its resting quotes sit on a book it cannot see; had a NAT dropped the connection silently, the thread would have learned only when a ping went unanswered, seconds later. With an idle spare already through the upgrade, the subscription and the snapshot go out together and the blind spell is one round trip, 70 ms. With a subscribed spare it is a switch between sockets, microseconds, provided the venue numbers both connections’ updates alike.

Seeing it

openssl speed -evp aes-128-gcm prints the thousands of bytes a second one core encrypts, for blocks from 16 bytes to 16 KB: 1–2 million at 256 bytes on a server core with AES-NI, which is 1–2 GB/s, and far less at 16 bytes, where the fixed cost per call dominates. -bytes 200 -decrypt measures the receive side at an update’s size, and -evp chacha20-poly1305 gives a comparison for the same CPU and library backend.

perf record -g -t <tid> on the feed thread, then perf report, shows the decrypt at or near the top, a function with aesni_gcm_decrypt or aes_gcm_dec_update in its name, beside the parser and the kernel’s receive copy. With kTLS it appears under tls_sw_recvmsg, inside the receive system call.

SSLKEYLOGFILE names a file to which a TLS client writes each session’s secrets: rustls does when the config’s key_log is KeyLogFile::new() and the variable is set, and curl and the browsers honour it too. Capture with tcpdump -i <iface> -w feed.pcap host <venue> and give Wireshark the key log in its TLS preferences, and it decrypts every record and shows the WebSocket frames and JSON inside and the segments each record arrived in. The file unlocks every session it lists, so it belongs on test connections only.

ss -ti shows TCP’s half of the handshake, SYN-SENT until the SYN-ACK and ESTAB after; an ESTAB connection whose bytes_received stops at a few kilobytes, the server’s handshake flight, has not finished TLS’s half, and a lastrcv that grows past the ping interval marks a dead connection. On a kTLS socket ss -ti adds a tcp-ulp-tls entry naming the version and cipher, and /proc/net/tls_stat counts kTLS sessions and failed decrypts. grep -m1 -o -w aes /proc/cpuinfo prints aes if the core has AES-NI, and pclmulqdq is the flag GHASH needs; in a virtual machine the hypervisor decides which flags the guest sees, and a missing flag means software AES whatever the hardware can do.