Feed capture and replay lab

Lab 4.5 (stretch) — capture and replay (ch04-network/src/bin/replay.rs)

A crypto venue’s feed is WebSocket over TLS over TCP, so tcpdump -i <iface> -w feed.pcap host <venue> on its own records ciphertext, and the first half of the lab is getting the plaintext. One way keeps the wire’s timing. Run the feed client with SSLKEYLOGFILE set, which rustls honours once the config’s key_log is KeyLogFile::new(); start tcpdump before the client connects and capture 60 s; then have tshark, given the key log, decrypt the records and export each WebSocket payload with its capture time. The simpler way has the client write each frame’s payload, after the decrypt, to a file with the time its recv returned. Either way the result is a file of records, each a timestamp, a length and a payload. replay.rs --file <path> loads it into memory before timing anything and runs the program’s own parser over it. --full-speed parses the frames back to back and prints messages and megabytes a second. --paced releases each frame at its recorded offset from the first, spinning on rdtsc in between, and records each frame’s time from its scheduled release to the end of its parse, even when earlier work made the actual parse start late, in an hdrhistogram. If the venue offers a binary encoding of the same stream, capture both at once on two connections and replay each.

Reason it out. Real messages are not the 161-byte example, since depth updates often carry several levels, so read the distribution of sizes first and scale by bytes: 1–3 µs for 161 bytes is about 6–19 ns a byte in a running handler, and the full-speed loop is as warm as Lab 4.4’s, so expect that lab’s rate per byte. A minute of an ordinary day, about 120,000 messages at 2,000 a second, should replay in well under a second. At the recorded pace most frames arrive alone and the p50 is one parse. The tail comes from clumps, frames recorded together because one recv returned them: the k-th waits behind k − 1 parses, so the last of a clump of ten finishes 10–30 µs after its release. Number each frame by its place in its clump, and predict the p99 as the 99th percentile of those places times the full-speed time per message. The replay leaves out the decrypt. Count the TLS records per message in Wireshark and charge each 100–300 ns fixed plus 0.5–1 ns a byte, checked on this core with openssl speed -evp aes-128-gcm -decrypt -bytes <median record size>: for one update per record of about 200 bytes that is 300–500 ns, roughly a tenth to a third of the program’s own share, and less per update where the venue packs several into a record. The binary stream should parse a hundred times faster, 10–20 ns a message, and turn the order of costs over: TLS’s fixed cost per record becomes the largest thing the program does, and the kernel’s 2–5 µs outweighs all of user space together.

Pitfalls. A capture that misses the handshake cannot be decrypted, because the key log is matched to sessions by the random value in each ClientHello. tcpdump reports on exit the packets the kernel dropped, and one drop leaves a hole the stream cannot be reassembled past, so raise its buffer with -B and capture again. The key log unlocks every session it lists: use a test connection and delete the file afterwards. If the venue negotiated permessage-deflate, compressed messages must be reassembled and inflated before parsing; WebSocket fragmentation and interleaved control frames also need handling; time that separately, as a cost of the venue’s choosing. tcpdump stamps a packet where the kernel’s tap sees it, after the card, the interrupt and the poll, and the client stamps a frame when recv returns; either serves for spacing, but use one kind for both encodings. The paced replay must spin, since sleep can overshoot short gaps through timer slack and scheduling, and strace -c should show the file read at startup and no calls per frame. Write down the number of messages and their sizes, the full-speed rate in messages and megabytes a second, the p50, p99 and max at the recorded pace, the largest clump, the records per message and the decrypt cost per record, and the same for the binary stream.