The Staff Engineer Distributed Transactions Interview: Sagas, Reservations, and Compensation

The checkout request timed out after inventory was reserved. Payment might have succeeded. A cancellation arrived while the coordinator was restarting. What should the customer see now?

That is the useful center of a distributed-transactions interview. Naming a saga does not answer which effects happened, which promises still hold, or who repairs an unresolved order. For senior and staff engineers, preparation should connect transaction boundaries to a precise business state machine. This guide develops an illustrative order workflow; its policies and practice rubric are teaching examples, not a description of any employer’s assessment.

1. Choose the smallest transaction boundary that works

Start by asking what must change together. If an order and its inventory allocation can live under one transactional owner, explain that simpler option before distributing them. Separate teams or deployment units do not automatically justify separate consistency boundaries. Ask whether independent ownership, external providers, or long waits actually require the split.

Database transactions have concrete scope. For example, DynamoDB’s transaction documentation describes atomic writes across tables within one account and Region. It also warns that downstream stream records can be interleaved: transaction atomicity at the writer does not imply an atomic stream view. A diagram labeled “transaction” needs an explicit boundary.

Two-phase commit is a conditional choice

When participants support a common transaction protocol, two-phase commit deserves a reasoned discussion. PostgreSQL’s PREPARE TRANSACTION reference describes prepared state surviving the session and being stored on disk. It warns that prepared transactions retain locks and can obstruct vacuum if left unresolved. Your answer therefore needs a coordinator recovery plan and a limit on how long work can remain prepared.

For this practice scenario, assume inventory and orders have independent databases and payment is an external API without a shared prepare protocol. Choose local transactions plus a durable workflow. State the tradeoff: intermediate states become visible, and recovery is a business operation. Do not promise a global database rollback across these participants.

2. Write invariants before drawing the saga

AWS defines a saga as a sequence of local transactions with forward continuation or compensation after failure. Use that vocabulary to organize the answer, then supply the rules the pattern itself cannot choose.

For our illustrative store, propose three invariants: confirmed inventory allocations never exceed sellable stock; an order can enter fulfillment only after its required allocation and payment evidence exist; cancellation cannot silently erase an already completed external effect. Ask the interviewer whether these are the intended product rules. A backorder business or a pay-after-delivery business would require different ones.

Represent the order with explicit states such as pending, reserved, payment-unknown, confirmed, compensating, canceled, and needs-review. Keep participant evidence alongside the state: reservation ID, provider operation ID, command identity, result, and timestamps. “Failed” alone is too vague to distinguish a declined payment from an unanswered request.

Give the irreversible boundary a name

The Azure saga guide distinguishes compensable steps, a pivot, and retryable steps after that pivot. Apply this as a design question: at what point does this workflow commit to moving forward? In our example, handing goods to a carrier changes the recovery policy. A later problem may require a return or refund rather than pretending shipment never occurred.

Draw that boundary on the state machine. Require an authorized transition before fulfillment, and reject a stale cancellation command that assumes the order is still merely reserved. Make the customer-facing cancellation response describe the durable decision, including a pending resolution when the system cannot yet establish the outcome.

3. Make progress durable and commands repeatable

For this example, choose orchestration because a single workflow owner makes the order’s next action and recovery policy easy to explain. Choreography can suit a small event-driven flow; AWS’s orchestration guidance also highlights the coordinator’s availability and observability responsibilities. Centralized workflow logic still needs durable state and recoverable execution.

Persist an intended step before dispatch, use a stable command ID across retries, and record the result before advancing. A replacement coordinator should derive its next action from stored evidence. It must not infer that an unrecorded reply means the participant did nothing. Guard state transitions with an expected workflow version so concurrent workers cannot independently advance different branches.

Close the local state-to-message gap

The transactional outbox pattern addresses the gap between committing a database update and publishing its event. Write the local state and outgoing command together, then relay the command. The relay can publish duplicates, so consumers still need idempotent handling.

In our proposed inventory service, put the command’s deduplication record and its reservation mutation in the same local transaction. A repeated reserve command returns the established reservation result. The coordinator’s outbox does not make that participant update atomic by itself; show both local boundaries. Retain command records for the actual replay horizon, including delayed messages and operator recovery.

Keep this section distinct from transport retries. The interview question is whether the business operation can be resumed after a crash at every transition. Walk through a crash after the participant commits but before the reply arrives, then show why the same command produces the same business result on recovery.

4. Treat an unanswered request as an unknown outcome

Suppose payment completes but its response is lost. Immediately refunding or issuing a new payment with a new identity can compound the uncertainty. Our workflow enters payment-unknown, preserves the original operation identity, and resolves it using the provider’s supported lookup, retry, or reconciliation mechanism. It withholds fulfillment until the configured evidence requirement is met.

Provider semantics matter. Stripe’s idempotency reference says API v1 saves the first executed request’s response, including server errors, and compares parameters on reuse. It permits pruning keys after at least 24 hours, after which reuse can create a new request. A provider’s idempotency window is not your order’s lifetime guarantee.

For this hypothetical integration, persist the provider object identifier when available and keep internal business deduplication longer than short-lived provider keys. When a response is ambiguous, define the exact evidence that permits either forward progress or compensation. A timeout budget can trigger escalation; it cannot prove that the remote action never happened.

Register recovery before the uncertain effect

Temporal’s compensating-actions explanation demonstrates why registering compensation only after an activity returns can miss an effect that occurred before timeout. It recommends accounting for actions that may or may not have completed. In our durable state machine, record the recovery obligation before dispatch and make the recovery handler inspect or safely address the corresponding operation.

This does not mean blindly issuing an inverse command. If reserve is still in flight while release arrives, the participant needs an ordering or cancellation protocol. For example, an operation record can remember that a specific reservation command was canceled and prevent a late attempt from creating it afterward. Explain that rule explicitly; “idempotent release” alone does not settle this race.

5. Protect inventory with reservations and conditional transitions

Sagas do not provide isolation across participants. The Azure guide’s discussion of anomalies includes lost updates and semantic locks. For the practice store, reserve stock inside the inventory owner’s local transaction, checking available quantity and recording the allocation together. Two coordinators reading the same earlier stock count must not both spend it.

Give each reservation an identity, owner, state, version, and logical expiration. Confirmation should conditionally transition the correct unexpired reservation; release should affect only that reservation’s allocation. A stale release must not increment stock twice or free stock belonging to a replacement reservation. These are proposed application rules to defend with concrete interleavings.

Expiration is a decision, not a cleanup notification

Do not use background deletion as the clock that decides validity. DynamoDB TTL deletes expired items asynchronously, typically within days. A checkout reservation can be logically invalid while its record still exists. Check the deadline in the transaction that consumes or renews the reservation.

Consider the hardest race: payment succeeds just as inventory expires and is sold elsewhere. Our example policy routes the order to resolution and refunds the successful payment if stock cannot be reacquired; it does not mark the order confirmed using an expired allocation. Another product might permit a backorder with customer consent. State the chosen policy and the owner responsible for resolving it.

6. Design compensation as a second workflow

Microsoft’s compensation guidance explains that recovery must account for concurrent changes and may not restore the original state or execute in exact reverse order. It also describes resumable, idempotent compensation and manual intervention when automatic recovery cannot finish. Treat these as explicit requirements, not an exception hidden in a catch block.

In our example, cancellation before fulfillment schedules release of the identified allocation and, if applicable, a refund tied to the original payment. Persist each obligation separately. An order remains compensating while either obligation is unresolved. A refund timeout gets its own unknown-outcome handling; repeatedly creating new refund identities would undermine the recovery design.

Define needs-review as an operational state with an owner, evidence bundle, and permitted actions. The bundle should include workflow version, participant references, outstanding obligations, and the reason automation stopped. An operator’s repair should issue an audited command with a stable identity rather than editing a status column to make the dashboard green.

Monitor age in nonterminal states, unresolved external outcomes, and outstanding compensation obligations. These are suggested signals for this design, not universal service-level targets. Ask the product owner which delays require customer communication and which thresholds should page an engineer. A low HTTP error rate cannot establish that old orders reached a valid final state.

7. Rehearse the failure timeline

Use the following illustrative practice rubric to evaluate your explanation. It is not a validated hiring scorecard. First, can you state the invariant and its transaction owner? Second, can you recover after each uncertain external call? Third, can you explain cancellation racing with completion? Finally, can another engineer operate the unresolved cases using the evidence you persist?

Run a mock interview with four interruptions: the coordinator dies after sending reserve; the payment reply disappears; the customer cancels during confirmation; compensation times out after taking effect. For each interruption, name the durable state, the next allowed command, its identity, and the evidence required to declare completion. If the answer changes between retries, find the missing transition guard.

Finish by explaining what would simplify the system: moving an invariant back under one transactional owner, narrowing a provider integration, or eliminating an unnecessary intermediate state. The strongest practice answer is one whose promises survive the failure timeline you just drew.