The Staff Engineer Replication Interview: Quorums, Stale Reads, and Safe Failover

A customer changes a delivery address, refreshes the page, and sees the old address. Your API returned success. The database has replicas. Neither fact explains whether the next read should include that write. A replication interview becomes interesting when you can trace exactly what success promised and which machine can honor that promise.

For senior and staff engineers, prepare an answer that separates durability, visibility, ordering, and availability. This guide uses PostgreSQL, DynamoDB, and replicated-log research to make those boundaries concrete. The design choices and rehearsal rubric below are illustrative preparation tools, not a description of any employer’s confidential assessment.

1. Give each operation a consistency contract

Start with a workflow, rather than “we need three replicas.” Imagine an order service with an address editor, order-history page, and inventory reservation. Ask what happens if the address editor briefly shows old data, whether a customer must see their own update immediately, and whether two regions may both reserve the last item. These operations may deserve different read paths.

Write down the acknowledged-write boundary: does success mean durable storage on the primary, durable storage on another machine, or application on a serving replica? Then specify the failure being tolerated. A process crash, storage loss, and loss of an entire region impose different requirements. Name where the copies live; a replica count alone does not describe their failure domains.

Use product documentation to challenge assumptions. DynamoDB’s read-consistency documentation distinguishes a successfully persisted write from an eventually consistent read that may omit it. Strong reads are available on tables and local secondary indexes, while global secondary indexes and streams remain eventually consistent. A strong base-table read therefore does not make every downstream view fresh.

State whose history must be preserved

Distinguish “this user sees their own write” from “every subsequent reader sees the latest completed write.” Also ask whether a user may move backward after switching replicas. Turn the requirement into a short history: update address to version 42, receive success, then read version 42 or a later version. That history gives the interviewer a concrete counterexample to test.

2. Follow a write through receipt, flush, and replay

Draw the stages of a replicated write separately. A standby can receive log bytes before those bytes are durable, and durable log records can wait before being replayed into query-visible state. PostgreSQL’s standby documentation explains that streaming replication is asynchronous by default. Unreplicated transactions can be lost if the primary fails catastrophically.

The PostgreSQL 18 commit settings make the stages explicit. With synchronous standbys configured, synchronous_commit=on waits for their durable WAL flush. remote_write waits for writes to the standby operating system, offering weaker protection against an OS crash. remote_apply additionally waits for replay and query visibility on the acknowledging synchronous standbys. Without configured synchronous standbys, these labels do not establish a remote guarantee.

Translate that mechanism into a routing decision. If replica A acknowledges replay and replica B is asynchronous, sending the next request to B still needs a freshness check. An acknowledgment from one replica is evidence about that replica, not permission to treat every read endpoint as interchangeable. In a mock interview, point to the exact eligible destination on your diagram.

Explain what happens when an acknowledgment is slow. Propose a bounded request deadline and an explicit error or pending outcome. If an operator relaxes synchronous requirements to restore write availability, record that as a changed durability contract. Silently taking the slower replica out of the required set can change the failure protection the application thought it purchased.

3. Explain what quorum arithmetic proves

Illustrative quorum calculation: with a fixed replica set of three nodes, a completed write stored on two and a read consulting two must overlap on at least one node. This is the useful intuition behind R + W > N. It says the sets intersect. It does not specify how the reader identifies a valid version or handles a concurrent, partially completed write.

The original Dynamo paper describes configurable read and write participation, object versioning, and sloppy quorums. During failures, substitute healthy nodes can temporarily hold data through hinted handoff. If your arithmetic assumes fixed membership while your implementation chooses substitute nodes, you have skipped an assumption needed by the intersection argument. Keep this historical design distinct from today’s DynamoDB service.

When the requirement is an agreed sequence of commands, explain a protocol rather than stopping at “majority.” The Raft paper connects leader election, log replication, and safety rules, and describes overlapping majorities for membership changes. Those rules determine which history survives leadership changes. Simply writing to two independent databases does not supply the same protocol.

A useful follow-up is to remove or add a replica. Ask whether old and new configurations can each accept writes independently. Another is to let a timed-out write reach only one participant. Explain how your chosen system resolves those cases before claiming linearizability. You do not need to recreate a consensus implementation; you need to identify the guarantees you rely on.

4. Make replica routing preserve the read promise

A simple starting proposal is to send consistency-sensitive reads to the current authoritative writer and move tolerant reads to replicas. For read-your-writes on a replica, design an explicit barrier: capture a replication position after commit, carry its identity with the session, and wait until the selected replica has applied through it. Bound the wait; then use an eligible authoritative path or return a freshness error.

PostgreSQL exposes recovery information functions, including pg_last_wal_replay_lsn(), to observe replay progress. Treat a position-based routing barrier as an application design proposal, not a built-in end-to-end guarantee. Acquire the position after the transaction commits, verify that it covers that commit, and scope it to the correct database history. After failover, validate lineage rather than comparing unrelated positions.

Start the read with a snapshot that can see the replayed change. PostgreSQL’s Repeatable Read documentation explains that successive queries keep the same transaction snapshot. Our design implication: an older snapshot can defeat the intended experience even after replay catches up. Carry the session requirement through retries and connection changes. “Stick to this replica for a second” is a time heuristic; it does not prove a write arrived during an unusual replication delay.

A managed alternative can expose this contract directly. Azure Cosmos DB’s consistency documentation describes session consistency, including read-your-writes and monotonic reads within the session, with session tokens scoped to partitions. Its token-management guide explains propagating that state between application instances. If your design depends on that model, show how tokens travel with requests. Do not turn a partition-scoped mechanism into a database-wide ordering claim.

5. Choose how conflicting regional writes behave

“Active-active” leaves an important question unanswered: what happens when two regions accept incompatible updates? In DynamoDB global tables’ MREC mode, replication is asynchronous, conflicts use per-item last-writer-wins resolution, and conditional writes evaluate against the regional item. A condition checked independently in two regions is not a global serialization point.

For the hypothetical last inventory item, propose an authoritative reservation owner or a coordination mechanism with an appropriate conditional-update guarantee. For independently editable preferences, ask whether convergence by replacement is acceptable. Describe the user-visible loss if two edits conflict. Conflict resolution is a product decision as well as a storage mechanism.

The same AWS documentation now describes multi-Region strong consistency, or MRSC: writes synchronously replicate to another region before success, and strongly consistent reads return the latest item version across replicas. It also warns that regional round-trip latency affects performance. Name the exact mode and operation you need, then measure the intended topology. A service name alone does not identify the consistency contract.

Expose the availability consequence in the API design. If the system cannot safely coordinate the inventory operation during a partition, define the unavailable or pending response. You may keep other functions usable under a weaker read policy. State which operations remain supported so that “regional failover” does not accidentally promise every workflow under every failure.

6. Require evidence before promoting a replica

Promotion changes authority; moving traffic changes routing. Keep those steps distinct. PostgreSQL’s failover documentation explicitly requires preventing the former primary from continuing as primary after recovery, to avoid two writers and data loss. It also says PostgreSQL itself does not supply the external failure-detection and notification system.

For an illustrative promotion procedure, verify the candidate’s identity and durable progress, determine whether acknowledged writes are covered, and establish that the former writer cannot accept writes. Then promote, update application routing, and test both a new write and its required read path. Record uncertainty if the failed primary’s final state is unavailable; do not rename an unknown loss window “zero data loss.”

Include the return journey. Rebuild or reconcile the former primary under the new authority before admitting it as a replica. Keep it out of writable routing until that process finishes. A health endpoint reporting “database accepts connections” cannot establish that a node has the right history or the right role.

7. Measure progress and rehearse failures

Monitor where replication work accumulates. PostgreSQL’s replication statistics distinguish sent, written, flushed, and replayed positions. The documentation also cautions that lag intervals describe recent processing delays rather than predicting catch-up time. Pair progress with commit latency and application-level stale-read checks.

Illustrative capacity calculation: suppose a replica has 600 MiB of unapplied log, can replay 80 MiB/s, and receives 50 MiB/s of new log. Under constant rates, spare replay capacity is 30 MiB/s, suggesting 20 seconds to drain the backlog. This arithmetic is a practice estimate, not a benchmark or recovery guarantee. If incoming work reaches replay capacity, the backlog no longer shrinks.

Test at least four histories in your preparation: the primary dies after acknowledgment; a replica receives bytes but pauses replay; two regions lose contact; and an old primary returns after promotion. For each, name the allowed response, observable evidence, and recovery action. Add a routing test where consecutive reads hit different replicas, because a healthy replica fleet can still violate a session promise.

An illustrative practice rubric

Review your explanation using four questions: Can you identify the exact acknowledgment boundary? Can you show a read history your routing preserves? Can you explain which writer remains authoritative during a partition? Can you demonstrate the claim with a failure experiment? Mark an answer incomplete whenever it relies only on a topology label. This is a self-review aid, not an employer’s scoring system.

In your next mock interview, keep the address-update example visible from the first requirement through the promotion test. Every replica should have a reason to exist, every read should have an eligibility rule, and every success response should correspond to evidence you can name.