The Distributed Systems Interview: Retries, Idempotency, and Exactly Once

Your payment service commits a charge, but its response disappears. The client times out and sends the request again. Saying “add retries” leaves the central interview question unanswered: how do you recover without charging the customer twice?

The stakes are measurable. Amazon’s retry analysis illustrates how three attempts at each of five layers can create 243 database attempts. In a separate AWS contention simulation, adding jitter cut calls by more than half with 100 competing clients. These are engineering examples, not evidence that a particular answer guarantees an offer.

This guide turns those failure modes into a practice framework for senior and staff engineers. You will defend an operation’s identity, transaction boundaries, retry budget, and recovery path. Unlike a broad system-design walkthrough, the exercise stays with one difficult question: what happens when a caller cannot tell whether work completed?

1. Start With the Business Invariant

Before drawing a queue, define the outcome that must remain true. For this hypothetical checkout service: one confirmed purchase intent may create at most one successful charge; every accepted intent must eventually reach a visible terminal state or an accountable exception queue. “Exactly once” is too vague until you name the effect and its boundary.

Ask whether the prompt concerns message delivery, database mutation, or an external action. Those guarantees differ. Kafka’s delivery-semantics documentation explicitly separates publishing durability from consumption guarantees and explains that external destinations require cooperation. A broker cannot, by itself, make an unrelated payment API participate in its transaction.

Write a small operation record: tenant, purchase-intent ID, request fingerprint, state, provider reference, creation time, and last reconciliation time. Define states such as accepted, processing, succeeded, failed, and outcome-unknown. The unknown state is useful because a transport error describes what the caller observed, not necessarily what the remote system did.

Amazon’s idempotent-API guidance uses precisely this ambiguity to motivate explicit caller intent. In your interview, make the invariant visible before naming infrastructure. Then ask the interviewer whether delayed confirmation is acceptable; the answer determines whether you can favor safety while reconciling an uncertain charge.

2. Spend One End-to-End Deadline

A timeout limits waiting; it does not reverse a committed write. Separate the user-facing deadline from individual connection and request limits. As a practice assumption, give checkout 1,000 milliseconds: reserve 150 for local processing and response delivery, leaving 850 for dependency attempts and backoff. Two 350-millisecond attempts plus a 100-millisecond delay fit; a third identical attempt does not.

These numbers are illustrative, not production defaults. Amazon describes choosing a latency percentile from an acceptable false-timeout rate: 0.1% maps to p99.9, with qualifications for network latency and connection setup. Cite the published timeout method, then explain which measurements you would collect for this service.

Carry the remaining budget through downstream calls. gRPC’s deadline guidance explains propagation and the deduction of elapsed time. It also makes the application responsible for stopping work spawned for a cancelled RPC. Mentioning that responsibility prevents “we set a timeout” from becoming a claim that all backend work automatically stops.

For an accepted durable workflow, cancellation needs a separate business contract. A closed browser may stop waiting while the purchase continues. Return an operation identifier that the customer can query, and show pending status instead of encouraging another purchase. Explain which work may stop safely and which work must continue to preserve the recorded intent.

Practice explaining a late result to a product partner. “We could not confirm the charge yet” is different from “Your payment failed.” The first leaves room to reconcile the existing operation; the second may invite another purchase. Define a status endpoint and a customer-visible pending state before optimizing the retry loop. A useful interview answer connects the network ambiguity to the interface, support workflow, and eventual resolution. Otherwise, a technically careful backend can still create duplicate customer intent through confusing product behavior.

3. Calculate Retry Amplification Before Adding Backoff

Draw the full call chain, including SDK and proxy behavior. If five layers each allow three total attempts, the worst-case leaf count is 3 × 3 × 3 × 3 × 3 = 243. “Three retries” would mean four attempts, so label your arithmetic precisely. Amazon’s five-layer example makes the multiplication concrete.

Choose an explicit retry owner for each dependency boundary and account for retries already performed elsewhere. gRPC retry configuration exposes retryable status codes, attempt limits, backoff, throttling, and server pushback. An application wrapper should not silently multiply the library’s configured behavior.

Per-request limits also need an aggregate limit. Google’s overload chapter describes a three-attempt request budget plus a client retry-ratio budget of 10%. Treat those as a documented design example, not universal settings. For your proposal, specify the denominator, measurement window, and behavior when the budget is exhausted.

Distinguish a transient unavailable response from invalid input and an uncertain mutation. Invalid input needs correction. Overload needs admission control. An uncertain mutation needs the same operation identity and possibly reconciliation. Your explanation should show how the error contract changes caller behavior, including when the service explicitly tells clients to stop retrying.

4. Use Jitter to Spread Work, Then Limit Total Work

Exponential backoff without randomness can synchronize clients into repeated waves. In the AWS simulation, jitter reduced contention and completion time relative to unjittered backoff. The reported reduction of more than half the calls was for 100 contending clients under that simulation’s conditions, not a promised improvement for every workload.

A practice policy might sample a delay uniformly between zero and a capped exponential limit. State the cap, maximum attempts, and remaining-deadline check. Before sleeping, verify that enough budget remains for a useful attempt. After sleeping, check again because scheduling delays and cancellation can invalidate the earlier decision.

Randomness cannot create capacity. Google’s cascading-failure analysis explains how retries can keep a backend overloaded after the initial problem. Pair jitter with bounded concurrency, limited queues, and rejection before expensive processing. Prioritize completing already accepted purchases over repeatedly admitting new work you cannot finish.

For a staff-level discussion, identify who owns the policy across teams. A shared client library, explicit retry metadata, and compatibility tests make the contract enforceable. Present these as your proposed operating choices. If one service changes its retry defaults, someone must evaluate the effect on downstream load before that change reaches every caller.

5. Make Idempotency an Atomic State Transition

An idempotency key identifies one intended operation across attempts. It should survive a network retry but change when the customer intentionally buys again. Scope it to the tenant and operation type. Do not infer intent solely from identical payloads: two legitimate purchases can have identical amounts and products.

Store a request fingerprint with the key and reject reuse with incompatible parameters. Stripe’s API contract provides a concrete example: it compares incoming parameters and replays the stored status and body for a previously executed request, including stored 500 responses. A key does not mean every replay produces success.

The dangerous implementation checks a cache, performs the mutation, and then writes the key. A crash between the last two steps leaves a successful mutation with no duplicate protection. Amazon’s atomicity requirement explains why recording the token and relevant mutations must succeed or fail together.

Within one relational database, propose a non-null unique key over tenant, operation type, and intent ID, plus a transaction covering the business update and saved result. PostgreSQL’s constraint documentation supports enforcing uniqueness in storage. Explain what a concurrent loser returns: the committed result, a pending operation reference, or a documented conflict. “Check then insert” without database enforcement does not settle the race.

If your isolation choice can abort transactions, retry the whole database transaction as described in PostgreSQL’s isolation guidance. Keep irreversible external calls outside that retried transaction body.

6. Close the Database-to-Queue Gap With an Outbox

Now make the interviewer’s failure precise: checkout commits its operation record, then crashes before publishing the work message. Reversing the order is also unsafe because a message can describe a database change that never commits. This is the dual-write problem addressed by the transactional outbox pattern.

Write the operation and an outbox event in the same local transaction. A separate relay publishes committed events. Give each event a durable identifier and preserve the ordering required by the business. The relay can crash after publication but before marking delivery, so the consumer must tolerate receiving the same event again.

For a consumer updating its own database, record the processed event ID in the same transaction as the effect. AWS’s outbox considerations explicitly warn about duplicate messages and recommend idempotent consumers. The outbox protects the database-to-publication handoff; it does not erase all downstream failure windows.

For an external charge, use a stable provider operation key and persist the provider reference. If the provider response is lost, reconcile that same operation rather than minting a fresh identity. Where multiple services require compensating actions, AWS’s saga guidance describes orchestration across local transactions. A refund is a new business action with its own failure modes, not a magical rollback of history.

Draw the payment provider outside the local transaction box. Then narrate three outcomes: a definite rejection, a confirmed success, and no authoritative answer. For the third, retain the original provider key and look up the operation using the provider’s supported mechanism. If the provider offers neither idempotent submission nor authoritative lookup, say that your proposed guarantee is unavailable with that dependency. Escalate the product tradeoff instead of inventing certainty. You may need a different provider, a narrower promise, or manual review before another attempt.

7. Ask Where “Exactly Once” Begins and Ends

Use vendor guarantees accurately. SQS standard queues can deliver a message more than once. SQS FIFO deduplication prevents duplicate sends within a five-minute deduplication interval. Neither statement proves that an external charge and consumer acknowledgment commit atomically.

Trace a worker that charges successfully and crashes before deleting the queue message. SQS visibility-timeout behavior allows an undeleted message to become available again. A longer timeout can reduce premature concurrent processing, but it cannot remove the crash window. The payment operation still needs duplicate protection.

Kafka offers a stronger coordinated boundary for reading, processing, and writing Kafka data using transactions and appropriate consumer settings. Its design documentation also states that guarantees for external destinations require cooperation. Describe the actual transaction participants before claiming end-to-end exactly-once behavior.

Similarly, Google Pub/Sub’s exactly-once documentation scopes its guarantee regionally for subscribers. The interview skill is reading and defending the boundary, not dismissing every guarantee as marketing. State what the chosen platform eliminates, what the application must enforce, and which failures remain detectable through reconciliation.

8. Design Retention, Replay, and Human Recovery Together

Idempotency protection expires if you delete its evidence. Stripe permits key pruning after keys are at least 24 hours old; a reused pruned key creates a new request. Do not confuse that API retention contract with your own obligation to recognize an old purchase intent.

Replay windows may be longer. Stripe’s webhook guidance describes automatic live-mode retries for up to three days, manual Dashboard resends for up to 15 days, and CLI resends for up to 30 days. These are different mechanisms. They demonstrate why “keep deduplication records for a day” cannot be copied blindly between producers, API clients, and consumers.

As an illustrative sizing exercise, 100 accepted operations per second produces 8.64 million records daily. At 200 bytes per record, that is about 1.73 GB per day before indexes, replication, and database overhead. A 30-day window is roughly 51.84 GB of raw records. Show your assumptions, then choose retention from the actual replay contract and storage model.

Separate bulky response retention from durable business identity where necessary. Give operators a reconciliation view with operation ID, observed provider state, attempted actions, and accountable owner. A replay tool should reuse the original identity, limit throughput, and record its decisions. Any deliberate new charge requires an explicit new intent; support staff should never have to guess whether clicking retry creates one.

9. Test Every Gap Between an Effect and Its Acknowledgment

Turn the diagram into adversarial tests. Inject failure before the local commit, after commit but before response, after outbox publication but before relay acknowledgment, and after the provider effect but before worker acknowledgment. In each case, assert the business invariant and the eventual recovery state, not merely an HTTP status.

Also send concurrent requests with one key, reuse a key with different parameters, deliver an old event after a newer one, and replay after the normal retention window. Stripe warns that webhook events can be duplicated and delivered out of order. That makes event ordering and duplicate suppression separate test dimensions.

Measure attempts per intent, completed business effects, unresolved-operation age, duplicate suppression, and recovery throughput. During overload testing, compare useful completions against total attempts. Google’s cascading-failure guidance explains why extra activity can coexist with worsening service health.

Include a recovery drill with humans. Ask who pauses replay, who confirms provider state, and who communicates pending status to customers. These questions expose operational gaps that a happy-path load test misses. Keep identifiers in logs access-controlled, and avoid turning full payment payloads into debugging breadcrumbs.

Make the test oracle independent of the code path you are testing. For example, count successful provider charges for an intent and compare them with your local terminal records. A counter incremented only by the worker cannot reveal an unrecorded external success. Run the same crash point repeatedly with varied timing and concurrent consumers. After recovery, require both safety and progress: no duplicate charge, and no accepted operation abandoned indefinitely. If a test only proves that duplicates were suppressed, it can miss a system that suppresses all useful work.

10. Rehearse a 45-Minute Failure-Centered Interview

Use this suggested practice schedule, not an assumed employer rubric. Spend five minutes defining the invariant and acceptable pending behavior. Use ten minutes for operation identity, storage, and the external-provider boundary. Spend the next ten walking through the lost-response and duplicate-consumer cases.

Use another ten minutes to calculate retry amplification, allocate deadlines, and constrain overload. Spend the final ten on retention, reconciliation, tests, and ownership. Have a partner interrupt twice: “The provider succeeded but you lost the response,” and “Support replays this operation next month.” Explain what changes without abandoning the original invariant.

Score the recording on observable evidence: you named the side effect, located the atomic boundary, calculated attempt limits, handled unknown outcomes, and described a safe replay path. This is a self-assessment tool, not a validated hiring score. Pair it with our system-design tradeoff guide and API interview guide when you need broader practice.

End with a short explanation someone else could implement: the same intent keeps the same identity; local state changes commit together; external effects have their own duplicate protection; retries consume bounded capacity; uncertain outcomes remain visible until reconciled. That is a defensible design, with clear assumptions an interviewer can challenge.