The Staff Engineer Disaster Recovery Interview: Prove Your System Can Restore

GitLab’s January 2017 database outage lasted about 18 hours after production data was accidentally deleted. Its postmortem revealed that expected database dumps were missing and backup failure emails had been rejected: the recovery problem existed before anyone needed the backups. GitLab’s account makes a sharp opening question for your next system design interview: what evidence proves this system can come back?

For a staff engineer disaster recovery interview, prepare an answer that connects a business promise to a tested restoration path. Draw the recovered state, identify the prerequisites, and explain how you would demonstrate that customers can use their data safely. A second region is a location; a recovery plan is an executable argument.

This guide focuses on restoring service and trustworthy state. For coordinating an active outage, use our incident response interview guide. For modeling demand, use the capacity planning guide. Here, your central question is whether recovery actually meets its promise when the primary environment is unavailable or untrustworthy.

Define the recovery contract before drawing regions

Start by asking which customer operation must recover, what disaster is in scope, and which data loss is acceptable. AWS defines recovery time objective, or RTO, as the maximum acceptable delay from interruption to restoration, and recovery point objective, or RPO, as the acceptable gap since the last recovery point. These are business objectives in the AWS recovery framework, rather than performance claims established by naming a storage product.

Write an explicit contract: “Authenticated customers can submit new orders, retrieve prior orders, and receive an unambiguous result.” Then decide whether reporting, recommendations, and historical search belong in that initial recovery. Ask who can authorize degraded operation and lost-write reconciliation. AWS recommends objectives for each workload; apply that idea to the business workflows in your scenario.

Give the clock a start and a finish

For a hypothetical order service, assume a 60-minute RTO and a five-minute RPO. These are invented exercise requirements, not industry benchmarks. Start the recovery clock at the customer-impacting interruption. Stop it when the agreed order workflow works against validated state. Keep detection, decision-making, and verification inside the budget; otherwise you can report a fast database restore while customers remain unable to place orders.

Ask whether acknowledged orders may be lost at all. If the answer is no, revise the contract and architecture together instead of quietly reinterpreting the five-minute target. AWS’s objective-setting guidance connects the targets to business impact. Your interview answer should make that conversation visible.

Specify the evidence the business would accept. A status page saying “restored” is different from a customer retrieving the correct order and submitting a new one. In your contract, identify the verification account, the expected response, the permitted degradation, and the person who accepts the result. Ask whether the recovered service can operate while fulfillment or payment dependencies remain unavailable. AWS places workload recovery inside business continuity; use that boundary to avoid restoring a technically healthy but unusable workflow.

Separate infrastructure failure from data corruption

Use two distinct scenarios: a primary site becomes unreachable, and an authorized process deletes valid records. Then ask which recovery mechanisms address each. A current replica may help with lost infrastructure while faithfully containing an unwanted deletion. For PostgreSQL, write-ahead log replay can reproduce changes or stop at a chosen point; the PITR documentation explains both behaviors.

For the deletion scenario, propose recovering a known-good point into an isolated environment. Determine the damaging transaction’s boundary, preserve evidence, and identify legitimate activity after that boundary. Explain how you will reconcile that activity before replacing production state. Merely recovering yesterday’s data leaves the business consequences unanswered.

Be precise about the backup chain. PostgreSQL continuous recovery requires a base backup and a continuous sequence of archived WAL; logical pg_dump output cannot substitute for that physical recovery chain. Cite the PostgreSQL recovery requirements, then name what you would verify: the selected base backup, archive continuity, recovery target, and actual replay result.

Explain how you select trustworthy state

Suppose the interviewer says an incorrect migration deleted order items and the alert arrived later. Ask for the migration history and evidence of the first damaging operation. Propose testing a recovery target before that boundary in isolation, then checking the affected relationships. State what you do if the candidate restore point still contains corruption: reject it, preserve its validation result, and evaluate an earlier point. PostgreSQL supports stopping replay at a selected point in its continuous-archiving recovery model. Your contribution is the method for determining whether that point is acceptable.

Do not silently equate the latest archived timestamp with the latest trustworthy state. In this exercise, report both the technical archive boundary and the chosen business recovery boundary. List which legitimate orders require reconstruction and where authoritative evidence might exist. If you lack that evidence, state the uncertainty and ask who can accept it.

For malicious deletion, include protected copies. AWS Backup Vault Lock distinguishes removable governance locks from compliance locks that become immutable after their grace period while recovery points remain. Explain the selected protection, retention, and restore identity. Immutability protects retention; your drill must still establish that a retained copy can be decrypted and restored.

Choose a strategy by its unfinished work

Use the strategy name to summarize work that remains at disaster time. AWS describes four recovery patterns with different operational commitments. Avoid borrowing illustrative recovery times as guarantees for your design. You have to measure your own path.

Strategy	What is ready	What your answer must prove
Backup and restore	Recoverable data copies	Infrastructure deployment, restoration, and verification fit the budget.
Pilot light	Core resources and replicated data	Missing services and dependencies can be activated.
Warm standby	A functional environment at reduced scale	It can expand and take the agreed workload.
Multi-site active/active	Traffic served in multiple regions	Surviving sites preserve required behavior after evacuation.

The table summarizes AWS’s strategy definitions. For the exercise, compare a restore-based design with a warm standby by listing unfinished steps and recurring costs. State your recommendation conditionally: “I would accept restore-based recovery only after a representative drill demonstrates the full workflow within our objective.”

When asked why you did not choose active/active, explain which requirement would justify it and what additional decisions you would need to make. Cover write ownership and how the application resolves conflicting or uncertain results. A staff-level argument gives the organization a decision boundary it can revisit.

Calculate the complete recovery critical path

Turn the architecture into an ordered execution plan. Which steps can run together, and which depend on a restored database or available identity? Google’s data integrity chapter calls for testing the complete recovery process, including resource availability, elapsed time, monitoring, and external dependencies. A restore throughput estimate covers only part of that process.

Use arithmetic to expose assumptions

Hypothetical calculation: restoring 600 GB at a sustained 200 MB per second takes 3,000 seconds, or 50 minutes, using decimal units. Assume another ten minutes for provisioning, 15 for replay, and ten for validation. If sequential, those steps total 85 minutes before traffic is admitted. This invented example fails the 60-minute objective even before detection and approval; it is a calculation, not a measured cloud result.

Now make the answer useful. Could provisioned infrastructure remove the first stage? Could a newer base backup reduce replay? Could the initial workflow exclude rebuilding search? Write down the changed assumption and the measurement needed. Do not claim overlap unless the dependency graph permits it.

Track the chosen recovery point separately from the time restoration finishes. In another hypothetical example, a disaster at 12:00 with recoverable state only through 11:53 leaves a seven-minute gap, missing the five-minute objective. A successful backup job is insufficient evidence of either target; AWS recommends recovery tests that validate both RTO and RPO.

Make the recovery path survive the disaster

Cloudflare’s November 2023 control-plane and analytics postmortem describes services that depended on components available only in the failed PDX-04 facility. The company had tested parts of its high-availability setup but had not tested removing that entire facility. Cloudflare’s primary account provides a concrete reason to challenge the boundaries of a drill.

Draw a separate recovery dependency graph. Include identity, key access, backup catalogs, artifact registries, configuration, deployment tooling, routing, and verification. For each dependency, ask: “Can the recovery operator use this while the primary site is unreachable?” If you cannot answer, mark an unresolved prerequisite instead of assuming independence.

Also distinguish existing resources from administrative APIs needed to create or modify them. AWS recommends minimizing control-plane operations during recovery. Explain which resources and routing mechanisms you would prepare in advance, and which operations your chosen restore strategy still requires.

The practical tradeoff is evidence versus expense. You may deliberately keep a cheaper recovery path that depends on provisioning in a healthy region. State that assumption and test it. If the prompt requires surviving a broader control-plane impairment, redesign that path. AWS’s guidance supports reducing these dependencies; it does not prove that every pre-created service meets your recovery contract.

Validate business state before opening writes

Before reopening writes, explicitly fence the former primary and any stale writers. Demonstrate how the recovered environment becomes the sole write authority even if routing updates only partially. Treat ambiguous ownership as a stop condition for the recovery exercise.

Define a release gate for the recovered environment. Check that data is present, readable, consistent with its recovery point, and usable by the application. AWS explicitly identifies restoring without querying data as an anti-pattern. A green infrastructure health check should trigger deeper verification.

For the order exercise, propose these application-specific checks:

• Retrieve known orders and compare them with records captured at the recovery point.
• Check that order items and payment references belong to valid orders.
• Locate acknowledged requests beyond the restored boundary for reconciliation.
• Verify that retrying a known request preserves its intended result.
• Run an isolated customer workflow while blocking real payment and notification side effects.

These are proposed test requirements for the fictional service. Tailor them to the actual data model and external system contracts. Ask how you will distinguish an unapplied operation from one already completed by a payment provider. Restoring your local database alone does not answer that question.

Keep restoration and validation outcomes separate in the evidence. AWS Backup supports event-driven validation after a restore job completes and a separate validation result. Use that distinction in your answer: resource created, data checked, workflow verified, then traffic admitted.

Google recommends a shared validation framework with product teams responsible for business-specific checks in its data integrity guidance. In your proposed organization, assign platform ownership for restoration evidence and product ownership for order correctness. Name who accepts unresolved reconciliation before writes resume.

Build a drill that can disprove the plan

Define success before execution: the target recovery point, customer workflow, elapsed-time limit, correctness checks, and permitted degradation. AWS recommends regularly exercising recovery paths and warns that rarely used paths can contain incorrect assumptions in its disaster recovery testing guidance. Your drill should create evidence strong enough to reject your design.

Propose a staged program: isolated backup restoration, dependency-removal exercises, and a controlled failover rehearsal with explicit stop conditions. Restrict credentials and outbound effects in isolated tests. For a production exercise, specify scope, authority, customer-impact limits, and a recovery path for the exercise itself. Decide what each stage proves and what remains untested.

Use the real runbook with an operator who did not write it. Capture timestamps, backup identifiers, unexpected permissions, manual repairs, validation results, and the exact service version. If someone privately fixes a broken prerequisite, record the failure rather than awarding the original plan a pass.

For a fictional failed drill, imagine restoration finishes within the budget but the application cannot decrypt customer records. Report a failed customer-workflow gate, an unresolved key-access dependency, and the elapsed time already consumed. Assign a repair owner and repeat the affected path before claiming readiness. Keep the original result. A later successful run can demonstrate the repaired design without rewriting the evidence of what failed.

Give every drill a short, reviewable record: scenario, scope, environment version, recovery point, stage timings, data checks, manual interventions, and conclusion. Attach the logs that support each claim. Link each defect to the recovery step it blocks. This is a proposed evidence format, not a universal standard; adapt it to the workload while retaining enough detail for another engineer to reproduce the test.

Repeat after meaningful changes. AWS’s configuration-drift guidance emphasizes parity between primary and recovery environments. Treat new dependencies, schema changes, and deployment changes as reasons to review the recovery contract. GitLab’s backup-version and notification failures also justify testing that failures reach someone who can act.

Treat failback as a second recovery

After failover, ask where authoritative writes now live. AWS’s failback guidance explains that the recovery environment contains current data and the former primary must be resynchronized. Returning traffic to an old primary without that work contradicts the restored-state contract.

Propose fencing the old writer, establishing the new authority, rebuilding replication in the required direction, validating convergence, and scheduling a controlled return. Keep the old environment available for investigation according to the incident’s needs. Explain what happens if routing changes only partially or an old application instance reconnects.

Use an explicit decision point after writes reopen: a return to an earlier data copy needs reconciliation of newly accepted operations. Test this phase with the same workflow checks as failover. Consider remaining in the recovery region if returning would add risk without business value; identify the approval and preparation needed to make that a durable operating decision.

Rehearse a staff-level answer with evidence

For your next mock interview, ask a partner to present this fictional prompt: “A region is unavailable. Restore the order workflow within 60 minutes, with at most five minutes of lost state.” Then have them reveal one complication: the newest backup is incomplete, deployment credentials depend on the failed region, or deletion has reached the standby.

Practice a concise opening: “I’ll confirm the customer workflow and loss tolerance, identify trustworthy state, map the recovery prerequisites, and measure the full path to validated service. I’ll name the write authority and the reconciliation gate before accepting traffic.” Expand each clause with a design choice and a test.

Bring an evidence sheet with the contract, failure scenarios, recovery graph, critical-path estimate, validation gates, last drill results, and open assumptions. Use real experience where you have it, and label hypothetical designs clearly. Ask your practice partner to challenge the weakest assumption instead of rewarding a polished diagram.

Review the answer against observable decisions

After the rehearsal, check whether you made these decisions explicit: the business acceptance gate; the scope of the disaster; the source of trustworthy data; the complete recovery timeline; the dependencies that remain available; the write authority; and the conditions for returning to the former primary. For each missing decision, rehearse the specific sentence that would resolve it. This editorial checklist is a practice aid, not a claimed employer scoring rubric.

Also review how you handled uncertainty. Replace “the backup should work” with “the last representative restore passed these checks, and this change still needs a drill.” Replace an unexplained time estimate with its stage assumptions. If you have never operated the proposed design, say what you would test first. That gives your interviewer a concrete engineering judgment to question.

Close your answer by naming the next measurement that would change the recommendation. AWS calls for tested recovery paths; translate that into an engineering commitment someone can execute and inspect. The strongest ending is a recoverable customer workflow with a known boundary of proof.