How nested columnar data reconstructs boundaries and optional values

Flatten the integer leaves from a nested document and the resulting column loses their parent relationships. It also omits a missing collection, a null collection, an empty list and a null list element because none contributes an integer. Different nested documents can therefore produce the same leaf sequence. Reconstructing the document requires evidence about where each leaf belongs and which positions contain no leaf.

A format for nested columnar data can retain that evidence through boundaries, offsets and presence indicators. A reader may skip unrelated value columns while retaining the structure required by its selected result. The format contract determines which relationships those indicators preserve and how the decoder combines them with compact values.

Construct unequal columns from five nested records

Use a stipulated schema with one required row ID and an optional readings collection. The collection can be absent, present null, or a present list of optional integers. List order matters. A null element occupies one list position without contributing an integer leaf. This is a tree-shaped value model; it does not include shared object identity or cycles.

The five rows have IDs R0 through R4. R0 has no readings field. R1 has readings set to null. R2 has an empty readings list. R3 has readings [null, 7]. R4 has readings [8]. The ID column has five entries, while the compact integer-leaf column has only [7, 8]. There are three list-element positions, because R3 contributes two and R4 contributes one.

A reader cannot zip the integer column with the ID column by equal index. Leaf 7 belongs to R3, not R0. Leaf 8 belongs to R4, not R1. The different lengths are expected consequences of the nested schema, not evidence that values were lost. The problem is how to retain their parent and position relationships.

Flatten another document in which R3 contains [7] and R4 contains [null, 8], leaving the first three rows unchanged. Its integer leaves are also [7, 8]. Alternatively, R3 could contain [7, null] while R4 remains [8]. Again the leaf sequence is unchanged. Parent boundaries and element-presence positions distinguish these documents.

Retain offsets for structural reconstruction

Give each row a collection-state tag: absent, null, or present. The original five-row tag column is [absent, null, present, present, present]. Then store offsets into the flattened list-element positions. The offsets are [0, 0, 0, 0, 2, 3]. There is one more offset than rows, so row i owns the half-open element interval from offset i through, but not including, offset i + 1.

R0 owns [0,0), R1 owns [0,0), and R2 owns [0,0). Their ranges are all empty, but their tags distinguish their states. R3 owns [0,2), and R4 owns [2,3). Offsets identify the number and location of element positions even when some positions have no integer leaf.

Now store element validity [false, true, true]. Position zero is null; positions one and two contain integers. The compact leaf values remain [7, 8]. To connect valid positions to the compact column, use cumulative counts of valid elements: [0, 0, 1, 2]. A valid element at position e obtains the leaf at the number of valid elements preceding e.

Thus position one reads leaf index zero, yielding 7. Position two reads leaf index one, yielding 8. Position zero does not read the leaf column because its validity is false. These cumulative counts may be stored or derived by the decoder under the format's contract. They are separate from the row offsets because they count values rather than list positions.

The online-migration guide supplies an information-retention comparison for structural reconstruction. Its lowercase transform maps both "Alice" and "ALICE" to "alice", so the new key cannot recover the original spelling. Here the compact leaves [7,8] similarly cannot recover the original parent and null-element positions. Retaining the required extra evidence allows each reconstruction under its own model. The migration's authority protocol does not define column offsets, and this tree encoding does not preserve shared graph identity or cycles.

Reconstruct null and empty collections step by step

Decode R0 by reading its absent tag, leaving readings out of the resulting row. Decode R1 by reading its null tag, creating a readings field with null. Decode R2 as present with the empty element range [0,0), producing an empty list. The three rows contain no integer leaves, but they reconstruct to distinct states.

Decode R3 as present with positions zero and one. Position zero is invalid, so place a null element there. Position one is valid and its preceding-valid count is zero, so read integer 7. The reconstructed collection is [null, 7]. Decode R4 from position two, whose preceding-valid count is one, and obtain [8].

The phrase null and empty collections describes two different container states in this model. An empty list is present and has no element positions. A null collection is present as null rather than as a list. A list containing one null element is present with one element position and no integer leaf. An absent field is a fourth distinct condition.

The field-update guide supplies a presence-versus-value comparison for null and empty collections. Its collection discussion distinguishes assigning an empty array from removing the field. It also separates present null from absence in its profile model. Our column tags retain comparable state distinctions under this collection schema, while the offsets and validity retain list positions. Stored absence here describes the decoded row; it is not an instruction to delete a field. The peer's Keep, Set and Remove actions do not define the column decoder.

If an application intentionally identifies some states as equivalent, it can choose a smaller structural representation. That would be a declared loss of distinctions. Under the model here, storing only an empty offset range cannot distinguish absent, null, and empty, so the collection-state tags are necessary.

Validate the relationship between structural columns

A decoder must check that the structural columns agree with each other. Offsets have length six for five parent rows, begin at zero in this example, never decrease, and end at the total element-position count three. Validity has exactly three entries. The number of true entries is two, matching the compact integer column length.

The schema also constrains the tags. Absent and null collections own no list-element positions under this encoding, so their adjacent offsets must be equal. A present collection can have an empty or nonempty range. An absent tag with a nonempty range assigns positions to a state whose schema forbids list elements. Reject that mismatch before producing a row; silently discarding its positions would hide invalid structural metadata.

The leaf index derived from validity must stay within the available values. A bitmap claiming three valid positions with only two stored integers is invalid even if every row offset looks well formed. Conversely, extra unexplained leaf values indicate a disagreement between the compact column and structural state. Successful parsing of each column independently does not prove the columns fit together.

For the alternative R3 [7] and R4 [null, 8], offsets become [0, 0, 0, 0, 1, 3] and validity becomes [true, false, true]. The same compact leaves now produce different ownership and null positions. For R3 [7, null] with R4 [8], offsets stay at the original boundaries while validity becomes [true, false, true]. These cases separately test boundary information and within-list presence information.

Keep required structure during selected-column reads

If a query reads only readings and ignores another optional payload column, it still needs the parent row ordering, collection tags, row offsets, and element validity needed for readings. It can skip unrelated leaf payloads under the format's rules. It cannot discard the metadata that gives the requested column its shape.

A query returning only non-null integers may deliberately produce the flat sequence [7, 8]. That is a valid projection with different output semantics. It does not reconstruct the original rows, and it should not claim to preserve which collection was empty or which element was null. The query's required result determines how much structure must remain observable.

For a list of structures with several optional leaf fields, each selected leaf needs a mapping to shared element positions. One field may have values where another is null. Pairing their compact value indexes directly can associate values from different elements. The common structural position and each field's presence mapping supply the needed alignment.

Deeper nesting needs boundaries at each level whose relationships the result preserves. Outer row offsets identify the positions belonging to a row; they do not identify how those positions divide into inner lists. A format can retain offsets at each relevant level or use another sufficient structural encoding. Check that the combined evidence reconstructs one admitted shape under the schema, without assuming a particular vendor's metadata layout.

Test shape equivalence against the original records

A bounded round-trip test encodes the five example rows and reconstructs each field's presence, collection state, list length, element validity, and integer values. Compare those properties with the original nested values. Comparing only flattened integers would wrongly accept all three differing example documents as the same result.

Add cases with consecutive empty rows, a list containing only null elements, and a final nonempty list after several absent collections. They exercise repeated offsets and validity-to-leaf indexing. Also test intentionally inconsistent metadata and require a clear failure before producing a fabricated row shape.

These cases establish model behavior for the tested structures. They do not benchmark compression, certify a named file format, or prove a decoder correct for every nested schema. A production format needs its own offset widths, validity conventions, schema bounds, and malformed-input acceptance rules.

Build a row-boundary worksheet using the Lenovo IdeaPad 15.6-inch FHD touch laptop. The product entry specifies a 512GB PCIe SSD, 16GB DDR4 and an i5-1334U processor. Place each row's collection tag beside its two adjacent offsets. R0, R1 and R2 all have [0,0), yet decode to absent, null and empty respectively. R3 uses [0,2), and R4 uses [2,3). Reading the ranges without the tags would erase the first distinction. These worksheet results follow the declared encoding; the storage specification supplies no compression or structural-conformance result.

For a separate element-position worksheet on the ASUS Vivobook 14 FHD laptop, record validity [false,true,true] above prefix counts [0,0,1,2]. Its native processor description names Core 5 120U, and the memory specification lists 16GB DDR4. Position zero contributes null; position one reads leaf zero, 7; position two reads leaf one, 8. Change validity to [true,false,true] while retaining the original row offsets and the same leaves, and R3 becomes [7,null]. The reconstructed shape changes even though the compact integers do not.