The Staff Engineer Schema Evolution Interview: Compatibility, Defaults, and Safe Rollouts

Your new consumer passes its tests, the schema registry accepts the change, and production traffic looks healthy. Then someone replays an older event. A field is missing, a default changes its meaning, or a JSON gateway rejects a message that the binary client accepted. A schema-evolution interview becomes interesting at exactly this boundary: several generations of software must agree about the same data.

For senior and staff engineers, a useful answer explains which combinations must work, what compatibility actually protects, and how to introduce a change without losing a recovery path. This guide focuses on event and message contracts across Avro, Protocol Buffers, and JSON Schema. The worked scenario and rehearsal rubric are illustrative preparation tools, rather than a description of any employer's assessment.

1. Define the compatibility boundary before choosing a format

Start with a concrete system: a job service publishes completion events consumed by reporting and customer notifications. Ask who owns each consumer, whether deployment order can be coordinated, and how long old events remain readable. Include offline consumers, archived exports, gateways, and any process that reads a message and writes it again. Draw those paths before proposing a registry.

Separate three questions. Will existing client code still compile? Can existing readers decode the new bytes? Will the resulting values still mean what the application expects? Google's AIP-180 distinguishes source, wire, and semantic compatibility. That vocabulary helps explain why a change can pass a serialization check yet still break a report.

Write one invariant for the scenario: every supported consumer must interpret a completion event's duration in the declared unit, including during replay and rollback. Then define the support boundary. A consumer that has been offline for months may need a supported upgrade route instead of indefinite compatibility with every future producer. Make that a deliberate product and ownership decision.

2. Make reader and writer direction explicit

Use old and new readers and writers as a small compatibility matrix. In Confluent's terminology, backward compatibility lets a new reader consume old data; forward compatibility lets an old reader consume new data; full compatibility covers both directions. State the direction aloud, because “backward compatible” alone leaves room for misunderstanding.

Illustrative compatibility checks for a message change

Writer	Reader	Question to prove
Old	Old	Does the existing contract still work?
Old	New	Can upgraded code process history and delayed messages?
New	Old	Can consumers awaiting upgrade, or rolled back, read new data?
New	New	Does the new behavior preserve the intended meaning?

Test the actual generated libraries and transformations in those cells. A schema comparison cannot show whether application code treats an unfamiliar status as success. Add representative fixtures for missing values, explicit zero, unknown enum values, and the oldest supported records. For each fixture, assert both decoding and the business result.

Do not promise arbitrary deployment order unless you have established both directions for the required versions. A coordinated upgrade can be a reasonable constraint. Explain how you would enforce it, identify remaining consumers, and block writers from emitting new content until those consumers are ready.

3. Explain Avro defaults from the reader's perspective

The Avro specification resolves data using the writer's schema and the reader's schema. A reader ignores writer fields it does not recognize. If the reader expects a field absent from the writer's schema, it uses its own field default; without a default, resolution fails. A default does not make a field optional when encoding a record.

For an illustrative change, add a nullable duration_seconds field with a reader default of null. An upgraded consumer can read an older record and interpret that absence as “the new measurement was not recorded.” It can then use the existing duration_ms value according to the application contract. Defaulting to zero would instead risk turning an unknown measurement into an apparently instantaneous job.

Now walk through deletion in the opposite direction. If the new writer stops providing duration_ms, an old reader still expecting it needs a suitable default in that old schema. Adding a default only to the new schema does not repair the already deployed reader. This example follows the specification's reader-side rule; it is not a guarantee that every additive or subtractive change is safe.

Ask how stored messages locate their writer schemas. Retain that association through archive and restore workflows. “We keep the bytes” is an incomplete recovery plan when nobody has established how the decoder will obtain the matching contract later.

4. Treat Protobuf identity, presence, and gateways separately

Field numbers and unknown fields

The proto3 language guide identifies binary fields by number and warns against reusing numbers. Reserve deleted numbers so a future field cannot accidentally inherit an old identity. Proto3 preserves unknown fields through binary parsing and serialization, but conversion to JSON or copying recognized fields into a fresh message can discard them.

Trace a new field through an old relay, not just directly from producer to consumer. If the relay reconstructs a message from known fields, the final reader may receive a valid message with the new information silently removed. Make preservation a testable requirement where relays are expected to carry data they do not yet understand.

Absence and explicit zero

Protobuf's field-presence documentation distinguishes implicit presence from explicit presence. An implicit scalar cannot distinguish an unset value from its default through the generated API. Proto3 optional scalars track presence. In the duration example, that difference matters when zero is a legitimate measurement and absence means the producer did not supply one.

Specify which layer applies a fallback and whether it preserves the fact that the original value was absent. Test a round trip through every relevant client language. An application that replaces absent values before storing them may erase information needed to diagnose a later migration.

Binary compatibility does not cover ProtoJSON

The ProtoJSON documentation explains that names appear in JSON and unknown fields are rejected by default, although parsers may offer an ignore option. Therefore, a field addition acceptable in binary can fail at an older JSON consumer. Record the encoding and parser settings on each edge of the diagram. Test the gateway path with the new field before enabling writers.

5. Make JSON validation and defaulting separate decisions

JSON Schema's object guidance makes properties optional unless they are listed in required. Additional properties are allowed by default; additionalProperties: false can reject them. A new optional field therefore does not automatically work with an old consumer whose validator accepts only its original field set.

Check both sides of the proposed change. Making a new reader require a property can reject old messages that lack it. Adding a property to new messages can upset a restrictive old reader. Decide whether the boundary should allow unknown fields, reject them, or translate through a versioned adapter. Each choice needs an owner and a stated reason.

There is another trap: JSON Schema's annotation reference describes default as metadata, not an instruction to fill missing values during validation. If an application needs a fallback, implement and test that policy explicitly. Do not assume an Avro reader default, a Protobuf scalar default, and a JSON Schema annotation perform the same operation.

Keep raw input and normalized interpretation conceptually distinct. In a practice design, a validator checks whether a message is acceptable, then a named adapter produces an internal representation. That separation lets you explain where old contracts remain supported and where the application begins relying on a new invariant.

6. Include retained history in the compatibility policy

Confluent's compatibility documentation distinguishes checks against the latest schema from transitive checks against all prior registered versions. Its default BACKWARD setting is non-transitive. If a new consumer must read a long event history, describe the historical coverage required instead of assuming that acceptance against yesterday's schema proves replay safety.

Create a retained fixture corpus containing each supported writer generation and the encoding paths that still exist. Exercise the current reader against it before release. Include restore jobs, delayed deliveries, archived files, and messages waiting in an error queue. Those paths belong in your migration inventory even if they do not appear in normal request traffic.

Separate durable records from external request objects when their evolution needs differ. The Protobuf best-practices guide recommends distinct messages for RPC APIs and storage, accepting a translation layer so storage changes do not directly reshape client contracts. In an interview, explain the maintenance cost of that adapter as well as the independence it buys.

7. Walk through one change and its rollback

Use the duration example to make the sequence reviewable. These steps are an illustrative engineering proposal; the exact order depends on the format and compatibility policy. For a backward-only policy, Confluent advises upgrading consumers first, because old readers are not assured access to new data.

AWS's rollback-safety article illustrates a preparation release that reads both formats while writing the old format, followed by activation of new writes. Rolling back activation returns to the prepared reader. Returning to the original reader can be unsafe after new-format data exists. Use that distinction to name your rollback target precisely.

First, establish meaning. Keep duration_ms in milliseconds. Add duration_seconds as a separate field with documented absence behavior and precision. Never silently reinterpret the old name. As an illustrative calculation, 1,500 milliseconds equals 1.5 seconds; reading the original number as seconds produces a result one thousand times larger while preserving a perfectly valid numeric type.

Next, prepare readers and adapters. Readers accept both generations and prefer the new field only when present. Define what happens if both fields disagree: quarantine, report an error, or retain the established value according to the product's requirements. Initially compare interpretations without changing customer-visible behavior. Require a passing historical fixture run and visibility into which consumer versions remain.

Then, enable new writes gradually. Derive both representations from one authoritative measurement in the same message. Compare values after unit conversion using a declared rounding tolerance. Monitor decoding errors, fallback usage, conflicting values, and consumer progress. Choose stop conditions before the rollout so a quiet error dashboard cannot substitute for semantic verification.

Finally, prove rollback and retirement. Run the intended rollback reader against messages already emitted by the new writer. Stopping new writes does not remove those messages from queues or archives. Keep the old representation until the supported consumers and recovery paths no longer require it. Record who authorizes retirement and which retained data still needs an adapter.

8. Rehearse with a rubric that tests your evidence

Use this illustrative practice rubric to review your own explanation. It is not a company hiring scorecard, and it does not predict an interview outcome. For each dimension, mark your answer “asserted,” “explained with an example,” or “supported by a concrete check.” Your goal is to make missing evidence easy to identify.

• Boundary: Name the consumers, encodings, retained history, and versions you promise to support.
• Direction: Walk through an old writer with a new reader and a new writer with an old reader.
• Meaning: Distinguish absent, null, zero, unknown, and valid values for one actual field.
• Delivery: Explain the deployment gate, rollback fixture, stop condition, and retirement owner.

Practice one follow-up at a time: an old relay strips a field; a consumer returns after an extended outage; a registry approves a change but a report changes units; or the team needs to roll back after new messages have been archived. Answer by updating the matrix and the release evidence. A clear account of these transitions makes the design useful beyond its steady-state diagram.