Message framing and encodings

4.7 Bytes to messages: framing, encodings and parsing in place

A socket returns bytes, and the strategy wants values: an instrument, a side, a price, a quantity. Two jobs lie between them, both in the program’s own thread: finding where each message starts and ends, and turning its bytes into numbers. They are the one stretch of the receive path the program controls completely, and on a feed that sends JSON they are the most expensive stretch after the kernel. This section covers how messages are cut from a stream, the two families of encoding, how fields are read without being copied, and what each choice costs.

A stream is not a message

TCP numbers bytes, not writes. The venue’s kernel merges small writes into one segment and splits large ones across several, and the receiving kernel appends every payload to one receive queue, where the boundaries between writes no longer exist. A read returns whatever the queue holds, up to the length of the buffer: half a message, or two and a half.

So every protocol carried over TCP marks its own boundaries, in one of three ways. In a fixed-size framing every message has one length, or begins with a type byte whose length both sides know, and the reader counts. In a length-prefix framing each message starts with a fixed-size header that holds its length, as WebSocket frames, TLS records and most exchange binary protocols do. In delimiter framing a reserved boundary marks an end, such as a newline after each JSON object. FIX’s SOH separates fields, while BodyLength and the trailer delimit a whole message. HTTP’s blank line ends headers; a body follows its own framing rules. The first two let the reader jump straight to a message’s end; a delimiter makes it inspect every byte, and the content must escape any delimiter it contains.

A read usually ends inside a message, and that tail has to wait for the rest. The economical way to hold it never moves a complete message: one large buffer and two cursors, start, the first byte not yet handed out, and end, one past the last byte received. Each read appends at end, and each complete message is handed to the parser as a slice of the buffer where it already lies. Only when the room after end runs short is the partial tail, always less than one message, moved to the front: with a 64 KB buffer and 48-byte messages, about once every 1,300 messages. A ring buffer avoids even that copy, at the price of messages that wrap around its end, unless its pages are mapped twice, back to back in virtual memory, so that a wrapped message still reads as one slice.

On this plaintext receive-and-parse path, the kernel copy into buf is the message copy, and the parser reads each message where that copy put it. A length that cannot be right is treated as an error, not as a reason to wait: a reader that waits for 60,000 bytes of a message that does not exist waits forever, and so does every message behind it.

UDP has no partial tail. Each recv returns one datagram, whole, so the datagram is the outer frame, and the messages packed inside it carry lengths of their own.

Text and binary

JSON is text: field names and values spelled out in characters, readable in a terminal and self-describing, so a reader that has never seen the schema can still tell which value is which. Each of those properties is paid for on arrival. Every number comes as decimal digits, converted one at a time with a multiply, an add and a check. Venues often send prices as strings, "68123.45000000", to keep them out of binary floating point, so a price is scanned once as a string and converted again afterwards. Every string is scanned to its closing quote and checked for backslashes, because escapes such as \" change what the bytes mean. Fields may come in any order, with whitespace between any two tokens, so the parser cannot know where a value lies until it has walked everything before it, and it compares every key it meets against the names it expects, including the keys of fields the program will throw away.

A fixed binary layout settles the meaning of every byte in advance, in a schema compiled into both programs. The price is always the third 8-byte word, bytes 16–23, an integer count of ticks, and one load reads it. SBE, the Simple Binary Encoding, is the FIX Trading Community’s standard and the format of several exchanges’ feeds: fixed-size fields at fixed offsets, then repeating groups, each with a count. FlatBuffers starts each table with an offset to a small table of field offsets, so a field costs a short chain of loads instead of one, and fields can be added without breaking old readers. Cap’n Proto makes the wire format the same as the in-memory format: 8-byte words, with pointers stored as offsets. Protocol Buffers, often listed with these, is not one of them: its integers have variable length and its fields may come in any order, so it is decoded field by field as JSON is, if far faster. A custom fixed layout needs its own explicit schema; #[repr(C)] alone supplies neither byte order nor protocol versioning. The rest of this section uses this 48-byte teaching format, with little-endian multi-byte integers:

Price and quantity scales belong to the instrument schema, not to a floating-point guess.

What binary gives up is schema evolution and debuggability. A layout fixed in two programs cannot change in just one of them: adding or widening a field needs a version in the header and both sides moving together, which is why SBE states each block’s length, so that an old reader can skip fields it does not know, and why FlatBuffers and Cap’n Proto pay an indirection. A message read with the wrong schema does not fail. It reads wrong: a little-endian price read as big-endian becomes a different integer, and nothing in the load itself complains. Schema validation must catch impossible values. A packet capture is unreadable without a decoder someone has to keep current. Those costs are paid by people, once; JSON’s are paid by the machine, on every message.