Part I Bytes and Ownership

Book contents · Next: Part II FlatBuffers

Chapters 1–3 of FlatBuffers & FlexBuffers: From Bytes to Mutable Memory. Representations, scalar layout, alignment, arithmetic, and safe C++ byte views.

This is the supplied first edition, pinned to its September 2026 source baseline. The laboratories and LiveFlex capstone are educational implementations and design proposals. The book overview explains the evidence limits; Reaktor’s performance review documents the current framework.

In this part

Chapter 1: Two representations, two jobs

Start with an ordinary object

Imagine a record containing a person's name, a counter, and a sequence of measurements. In C++, a straightforward representation might use std::string, an integer, and std::vector<double>. The record is easy to modify. The string can grow. The vector has capacity. Its implementation can replace one allocation with another and update a pointer.

Now write the raw bytes of that C++ object to a file. Those bytes contain implementation-specific metadata and, commonly, pointers into allocations that were not included in the write. Another process cannot use those pointers. Another compiler may lay out the object differently. Even the same process cannot treat that byte dump as a portable reconstruction of the original ownership relationships.

A wire format replaces those implicit agreements with explicit ones: integer widths, byte order, object boundaries, reference rules, and a way to locate the root. FlatBuffers and FlexBuffers both allow readers to navigate a serialized buffer without first rebuilding a conventional object tree. Their accessors interpret offsets and metadata in place. That is the useful meaning of their direct-access design—not a promise that every interaction with the data costs zero work. [1,2,4]

What “zero-copy” does and does not mean

Consider reading the third measurement. A conventional parser may first allocate a vector and populate it from input bytes. A direct-access reader can calculate the element's position and load the number there. That avoids materializing the vector merely to read it.

But the input still had to arrive somewhere. A network stack may copy it. Validation may scan it. A language binding may allocate a wrapper. Converting its string to an owning std::string copies bytes. Decompressing a transport envelope produces a new buffer. “Zero-copy” should therefore always be followed by a boundary: no copy between which two representations, for which operation, under which lifetime conditions?

There are four different costs worth separating. Acquisition obtains the bytes. Verification establishes that navigation is safe and that required invariants hold. Access loads selected data. Materialization converts it into another representation. An application that reports only access time while paying the other three on every request has not measured its actual pipeline.

A schema is information, not magic

Suppose a four-byte slot contains 00 00 80 3f. Is that the integer 1,065,353,216 or the floating-point value 1.0? Bytes alone do not answer. A schema can tell the reader that the slot is a float. Alternatively, a type tag in the buffer can tell it.

FlatBuffers usually obtains this information from a schema and generated accessors. A table's vtable describes the presence and location of fields; it does not generally repeat each field's full semantic type. FlexBuffers stores enough type information in the buffer to navigate its supported value domain without an application-specific schema. Schemaless does not mean typeless. It means the structural typing information travels with the data. [1,2,4]

This difference has a cost model. Removing repeated field names and type information can make known-layout data compact and cheap to access. Carrying names and tags makes unknown-layout data discoverable. Neither choice makes application validation disappear: knowing that a value is an integer does not tell you whether it is a valid account age or an allowed state transition.

Wire compactness and mutation pull in different directions

A compact buffer tends not to reserve unused space. A growable vector does. A compact integer can occupy one byte; an in-memory integer that must accept every future signed 64-bit value is conveniently represented in eight. A wire reference can be a short relative offset; an independently movable object often benefits from a stable handle or a pointer.

These are not accidental implementation weaknesses. They are consequences of different objectives. Every unused byte removed from a wire representation is a byte that is not immediately available for growth. Every native indirection removed can make relocation harder. Every object that is shared to save memory raises a question about what mutation should do to its aliases.

The central design question is therefore not “Can I make FlexBuffers mutable?” Some mutations already fit its existing layout. The stronger question is: “Which mutations must be cheap, and how much compatibility must exist during them?” Chapters 17 and 24 make that question precise.

Three levels of compatibility

Value compatibility means two systems can represent and exchange the same logical values. Wire compatibility means a produced byte sequence is readable by an ordinary implementation of the format. Continuous wire compatibility means the live representation remains a valid ordinary buffer after every published update.

A native map backed by a hash table can have value compatibility and produce wire-compatible snapshots without being continuously wire-compatible. Conversely, an existing FlexBuffers scalar can be changed in place while retaining continuous wire compatibility, provided the new value fits and ownership is controlled.

Keep these levels distinct. A design that silently changes the definition halfway through its argument can appear to achieve impossible combinations of compactness, stable addresses, and arbitrary growth.

Checkpoint

Exercise 1. A service receives a FlexBuffer, copies it into an owned allocation, verifies it once, then reads ten fields. Which costs were avoided? Which costs still occurred? Write a benchmark plan that measures all four stages separately.

Chapter 2: Bytes, alignment, integers, and floating point

A byte position is not an address

Throughout this book, a buffer position is a non-negative integer measured from the beginning of a particular buffer. An address is a machine pointer. The distinction lets us reason about malformed inputs without first constructing invalid pointers.

For example, a reference at position 100 may contain the backward distance 40. Its target position is 60. After checking that 40 is no larger than 100, the address is base + 60. Reversing those steps—first constructing base + 100 - 4000, then asking whether it was valid—is a poor foundation for a safe C++ parser.

Let N be the buffer length, p a position, and n a requested extent. The safe range predicate is:

p <= N  and  n <= N - p

The seemingly equivalent p + n <= N can fail when the addition overflows. Subtraction after a bounded comparison is not just stylistic caution; it is a way to keep the proof and implementation aligned.

Little-endian storage

Both formats use little-endian scalar storage. The least significant byte appears first. The unsigned integer 300 is hexadecimal 0x012c, so a two-byte representation is 2c 01. A four-byte representation of the same value is 2c 01 00 00. The byte order is a wire rule, independent of the host's native endianness. [1,4]

A deliberately simple decoder is:

std::uint64_t u = 0;
for (unsigned j = 0; j < width; ++j) {
    u |= std::uint64_t{bytes[position + j]} << (8 * j);
}

Before this loop, validate that the width is one of 1, 2, 4, or 8 and that the entire range exists. Casting each byte to 64 bits before shifting avoids accidental promotion to an insufficient integer type. The largest shift is 56, not 64.

This version is easy to audit. Selected paths can later use fixed-size memcpy loads and endian conversion. Do not start by optimizing away the invariant that makes the load legal.

Signed values and sign extension

The signed integer -1 is all one bits at any supported width. At one byte it is ff; at two bytes it is ff ff. Reading ff as unsigned gives 255. Reading it as a signed one-byte two's-complement value gives -1. The decoder therefore needs the width of the stored signed value, not just its raw bits.

For a width smaller than eight bytes, first read an unsigned value u. Let b = 8 * width. If bit b - 1 is set, the signed value is u - 2^b; otherwise it is u. The companion uses this expression only where both operands fit its chosen signed intermediate. The eight-byte case is handled separately using std::bit_cast between equal-sized integer representations. [12]

This separation avoids two common bugs: shifting a 64-bit value by 64, and attempting to represent 2^64 in a 64-bit intermediate. Boundary tests must include -128, -129, 127, 128, and the corresponding transitions at 16 and 32 bits. Testing only positive values leaves half the width logic untested.

Alignment is a layout constraint, not a permission slip

A width w that is a power of two can be aligned with padding:

padding(n, w) = (w - (n & (w - 1))) & (w - 1)
aligned(n, w) = n + padding(n, w)

If the current length is 5 and the next object needs four-byte alignment, three padding bytes bring the start to position 8. The addition must still be checked for overflow.

There are two distinct alignment questions. Is an object's position within the format correctly aligned? And is the actual address used by a typed C++ load aligned for that type? A buffer whose internal positions satisfy the format can still be placed at an unusual base address. A bytewise reader can handle that placement; a reinterpret-cast reader may not.

Nor does a suitably aligned address automatically contain a live C++ object of the desired type. Object lifetime and aliasing rules remain relevant. The safe baseline is to read bytes into a real scalar object with memcpy, then interpret or endian-convert that scalar. Exact language guarantees matter more than a processor's willingness to execute an unaligned instruction. [12,13]

Floating-point payloads

The practical FlexBuffers C++ paths discussed here use binary32 and binary64 floating point. The source contains placeholders for smaller floating types, but these are not an implemented portable binary16 wire feature to use in a new writer. The teaching implementation accepts floating widths of four or eight bytes only. [1]

A floating-point representation has properties that integer intuition misses. Positive and negative zero compare equal but have different bit patterns. NaN does not compare equal to itself. A finite double may not survive conversion to float and back without a value change. Infinities are representable in both sizes.

The official width-selection rule for a double tests whether converting to float and back preserves equality. This commonly allows a value such as 1.5 to use four bytes while leaving a value such as a non-exactly-representable double at eight. NaN fails the equality test and therefore does not take the narrow path. This is a storage policy, not a general numeric equivalence policy. [1]

When an application cares about NaN payloads, signaling behavior, or exact storage width, it must define those requirements separately. Avoid fast-math compilation for tests whose contract includes signed zero, NaN, or exact round-tripping. The wire's available types and the application's equivalence relation are different layers.

Checkpoint

Exercise 2. Decode 80 as an unsigned one-byte value and as a signed one-byte value. Decode 80 00 as a signed two-byte value. Explain why copying the first byte and ignoring the stored width gives the wrong answer.

Exercise 3. A buffer has 100 bytes. Show why checking a requested extent with length <= 100 - position is preferable to adding position and length first.

Chapter 3: A safe C++ vocabulary for byte formats

Own the allocation; borrow the view

An owning buffer and a view into it should be different types. The companion uses std::vector<std::uint8_t> for owned bytes and std::span<const std::uint8_t> for a borrowed range. A Ref contains such a range plus positions and widths. Copying a Ref does not keep the allocation alive.

This is convenient and dangerous in exactly the way an ordinary string view is convenient and dangerous. Returning a reference into a local vector produces a dangling view. Growing the vector can move its storage and invalidate every view into it. Mutating its bytes can invalidate the reasoning established by earlier verification even when the allocation does not move.

A production-facing API should make the ownership boundary harder to cross accidentally. One possible design is an immutable VerifiedBuffer that owns its allocation and exposes small borrowing references. Another uses a shared immutable owner when references must outlive the calling scope. The additional ownership cost belongs at the boundary, not necessarily in every scalar accessor.

Four layers of an implementation

The lowest layer performs checked arithmetic and scalar loading. It knows about bytes and widths, but not maps or strings. The next layer implements format navigation: root lookup, indirect targets, length prefixes, and vector indexing. The third layer verifies a reachable object graph and its required semantic invariants. The fourth layer presents application-facing operations and ownership policies.

Keeping these layers distinct pays off during optimization. A verified, immutable buffer may use a faster internal accessor without repeating every structural check. Untrusted input still enters through the verifier. Application range checks remain application checks. A single giant “read anything” function tends to mix these guarantees until none of them can be stated precisely.

The companion chooses a conservative teaching path: accessors retain many bounds checks even after verification. This is not the fastest possible architecture. It makes individual operations useful while you are learning and makes arithmetic defects easier to localize.

A readable scalar load

A production little-endian load specialized for a known width can use an actual scalar object:

std::uint32_t native;
std::memcpy(&native, bytes.data() + position, sizeof native);
// Convert native from little-endian if the host is big-endian.
return native;

The range check belongs before the memcpy. This is a conceptual excerpt: the complete portable companion uses explicit byte assembly rather than leaving the endian step implicit. Compilers commonly optimize small constant-sized copies, but an implementation should inspect its generated code rather than assuming a particular instruction sequence. [16]

Do not replace the load with an unconditional eight-byte read followed by a mask unless eight readable bytes have actually been guaranteed. A one-byte field at the end of a mapped page is not permission to read seven bytes past the buffer. A padded internal allocation can support a carefully documented fast path; arbitrary caller-owned input cannot inherit that promise.

Error handling is part of the format boundary

The companion throws a small Error exception for a violated invariant. That makes examples compact and allows tests to assert rejection. A production library might instead return an error code or an expected-like result containing a byte position, an invariant category, and a path through the document.

The important distinction is between malformed input, unsupported-but-valid input, resource-limit rejection, and application validation failure. A nesting limit is not evidence that the bytes are structurally corrupt. A valid unsigned 64-bit integer can be unsupported by a consumer that accepts only signed integers. An empty string can be structurally fine but invalid for a required name.

A useful error object might carry kind, position, type, and detail. Avoid allocating a full human-readable path on every successful traversal. Build rich diagnostics on the failure path or in a separate diagnostic mode.

Arithmetic helpers deserve their own tests

The companion's add, mul, align_up, range, load_u, load_i, and load_f are short. Their shortness is a reason to test them thoroughly, not a reason to trust them implicitly. Almost every higher-level operation depends on them.

For multiplication, a == 0 || b <= SIZE_MAX / a prevents overflow. For a backward reference, verify distance <= slot before subtraction. For a length prefix, verify that the prefix fits before the target. For a forward reference, verify distance <= buffer_size - slot before addition. These patterns make the permitted arithmetic visible in the code review.

The best tests combine boundary values with malformed layout. A length that fits its integer type can still describe more elements than remain in the buffer. An offset that fits its slot can still point outside the allocation. Width validity and range validity are independent predicates.

Checkpoint

Exercise 4. Design the public lifetime contract for VerifiedBuffer::root(). Decide whether the root view may outlive the owner, whether the owner may move, and whether the owner exposes any mutable byte access. Explain the cost of your choices.

Book contents · Next: Part II FlatBuffers