How canonical serialization defines equality before byte encoding

A serializer can produce exactly the same bytes every time it receives one object and still produce different bytes for an equivalent object. Sorting fields, shortening numbers, or rewriting strings cannot resolve that issue until the application defines which distinctions carry meaning. Canonical output is one representation for a defined class of equivalent values, with separate classes kept distinct.

A contract for canonical serialization specifies its admitted data model and equality rule before evaluating a normalizer. It identifies which differences in source spelling may disappear and which distinctions the application needs the byte format to preserve.

Separate byte equality from semantic equality

Two source documents can differ in whitespace and field order. For example, {"b":2,"a":1} and { "a":1, "b":2 } have different bytes. Under a model treating objects as unordered maps, both parse to the same associations. A parser can ignore those source differences while retaining the value relationships.

Parsed-value equality can still differ from application equivalence. An application might treat particular letter-case variants as the same identifier, while another treats them as distinct user text. A date representation might be equivalent to another only under a specific timezone or precision contract. These choices must be stated for the permitted values rather than inferred from a generic instruction to clean up data.

For the bounded example here, admit null, booleans, exact integers, strings of Unicode scalar values, ordered arrays, and maps with unique ASCII-letter keys. Integers are equal by exact mathematical value. Strings are equal only when their scalar-value sequences match. Arrays are equal element by element in order. Maps are equal by their key/value associations without regard to source key order.

This semantic equality is type-sensitive. Integer 7 is not string "7". An absent map key is not a present key with null. Array [1,2] is not [2,1]. Strings "Ada" and "ada" remain distinct. These counterexamples define distinctions the canonical encoder must preserve.

The online-migration guide gives a concrete comparison for semantic equality. Its lowercase transform maps both "Alice" and "ALICE" to "alice", losing their original spelling. That is acceptable only when the application can give up the lost distinction or retain it separately. Our string model keeps case significant, so importing that transform would merge values it declares unequal. Migration authority and rollback require their own protocol; this comparison concerns only the transform's information loss.

Show why deterministic encoding is insufficient

Suppose serializer D traverses a map in its stored insertion order and uses no extra whitespace. For the first example it repeatedly emits {"b":2,"a":1}. For the second object's insertion order it repeatedly emits {"a":1,"b":2}. Each input gives stable output, so this implementation is deterministic under those stored-order assumptions.

It is not canonical for the stipulated map equality, because the two equivalent maps receive different bytes. Determinism fixes the output for one fully specified input state. Canonicality additionally relates outputs across every admitted input that belongs to the same equality class.

Conversely, an encoder that always emits one fixed byte string is deterministic and gives equivalent values equal output. It also gives nonequivalent values equal output. That fails the requirement to preserve separate classes. A complete canonical contract needs both directions: equivalent values share bytes, and equal canonical bytes do not identify values the model considers different.

The write-once-memory guide supplies a full-input comparison for deterministic encoding. Physical states 000 and 111 both decode to logical message 00, but their remaining transitions differ. Its encoder must inspect the physical state, not only the decoded message. Our insertion-order serializer likewise operates on a fully specified input state that includes ordering. That observation explains a deterministic input boundary; the memory code neither proves canonicality nor supplies this format's equality or byte rules.

Construct one bounded canonical format

Define an encoder C for the admitted model. It emits UTF-8 with no optional whitespace. Maps emit keys in ascending ASCII lexicographic order. Arrays preserve their element order. Integers use ordinary decimal spelling with no leading zeros, and integer zero has one spelling, 0. Booleans and null use fixed lowercase tokens.

Strings use double-quotation delimiters. Encode a quotation mark as backslash followed by quotation mark, and a backslash as two backslashes. For this toy contract, control scalars are U+0000 through U+001F and U+007F through U+009F. Encode each as backslash, lowercase u and exactly four lowercase hexadecimal digits. For example, a newline becomes the six ASCII characters \u000a, with no alternative short escape. Every other admitted scalar uses its direct UTF-8 bytes. These choices define this example; they do not assert conformance to a named canonicalization standard.

The key comparator must be that declared ASCII order in every emitter. For keys A and a, the uppercase key precedes the lowercase key under the stipulated order, and both remain distinct associations. An implementation cannot substitute an environment-dependent collation simply because its output looks sorted. Likewise, two emitters choosing different optional escape spellings would fail byte agreement even if their decoders recover equivalent strings. Conformance checks should compare exact emitted bytes against the specified rules, not merely round-trip through each emitter's own decoder.

Map values and array elements are encoded recursively using the same rules. Because keys are unique under the admitted model, sorted emission has one position for each association. Delimiters and type syntax distinguish integer 7 from string "7" and an array from a map. A conforming decoder must interpret those bytes according to the same model.

Both example maps now emit {"a":1,"b":2}. Optional source whitespace is gone, and input key order has no effect. A string written with an allowed source escape for a character and the same string written with that character directly also reaches one encoding after parsing, provided both source spellings decode to the same scalar sequence.

The model deliberately uses exact integers rather than an unspecified floating-number domain. If another format admits additional numeric kinds, it needs rules for their identity and encoding. This example does not settle rounding, arithmetic accuracy, or distinctions among special numeric states. Restricting the domain keeps those unstated choices out of the claimed guarantee.

Test normalization against distinctions that must survive

Lowercasing every string would merge "Ada" and "ada", contrary to the stated string equality. Removing a null-valued field would merge {"a":null} with an empty map, contrary to the association rule. Sorting every array would merge [1,2] with [2,1], contrary to ordered-array equality. Each transformation may be appropriate for a different model, but each is wrong for this one.

Unicode normalization is also a choice about equivalence. A precomposed letter and a sequence containing a base letter plus a combining mark can look similar while having different scalar sequences. Under the model as stated, they remain different. If the application chooses to equate them, it must define the normalization rule and accept the distinctions that rule removes.

Review normalization with the application's nonequivalent counterexamples as well as positive examples. If user-visible spelling, ordering or field presence matters, the checks must preserve those distinctions. A shorter output says nothing about whether the transformation retained the required information.

Admitted objects also need a unique-key rule before reaching this abstract map model. The example assumes that prerequisite rather than explaining a repeated-key admission policy. Once an accepted map exists, canonicalization can sort its associations. It cannot recover source occurrences that an earlier conversion discarded.

Keep canonical values separate from original source text

A canonical encoder represents the accepted abstract value. It does not preserve original indentation, optional escapes, or key ordering. Decoding the canonical bytes can reproduce an equivalent value, but it generally cannot reconstruct the original document's exact source spelling. Source fidelity is a separate storage requirement.

If an editor needs to retain source formatting, it can keep the original text or an appropriate concrete-syntax representation beside the abstract value. That record has a different purpose from canonical output. Overwriting original source with a canonical representation deliberately loses formatting information, even when the value's semantic meaning is unchanged.

The two representations should be bound to the same accepted value if both are used. Editing one while retaining an unrelated old canonical result can make comparison or storage behavior stale. Updating the abstract value requires regenerating or validating its canonical bytes under the current encoding contract.

Versioning the contract matters too. A later equality policy or ordering rule can produce different canonical bytes for previously accepted data. An application using the bytes for identity or comparison needs to identify which contract generated them. Calling both versions canonical without a shared model and format definition leaves interoperability unresolved.

Check properties across equivalent and distinct inputs

A bounded test set permutes map insertion order, changes optional source whitespace, and varies allowed escape spellings without changing parsed strings. Equivalent values should yield identical C outputs. Repeat the test recursively for maps inside arrays and arrays inside maps, because a top-level sorting rule does not establish nested canonicality.

For nonequivalent pairs, check that C preserves distinct bytes: integer versus string, null field versus absence, different array orders, and case-sensitive strings. Decode each emitted result and compare the decoded value with the original under the stipulated equality. These checks catch collapsing too much as well as preserving irrelevant differences.

Idempotence is another useful property at the parse/encode boundary. Parse canonical bytes and encode that value again; the result should be the same canonical bytes. This is necessary under the toy contract but insufficient by itself: an insertion-order serializer can pass such a check for its own output while failing across equivalent maps.

Use an equality-class worksheet on the ASUS Vivobook 14 FHD laptop to record each accepted value beside its canonical output. Its processor description names Core 5 120U, and its memory listing specifies 16GB DDR4. Keep equivalent map permutations together, then place integer 7 and string "7" in separate rows. The worksheet should compare exact bytes, not a display that may hide whitespace or escape differences. The listed host specifications establish neither equality rules nor encoder conformance.

Keep a second worksheet for recursive byte checks using the Lenovo IdeaPad 15.6-inch FHD touch laptop. The entry identifies an i5-1334U, 16GB DDR4 and a 512GB PCIe SSD. Record a map nested in an array and an array nested in a map; every recursive emitter must obey the same ASCII ordering and fixed escape convention. Also record the contract version beside saved bytes.