The Staff Engineer Background Job Interview: Durable Work, Retries, and Recovery

The queue does not tell you whether the work happened

A report worker writes the finished file and crashes before acknowledging its message. Another worker takes the message and starts again. The difficult question is not whether your queue can scale; it is what the second worker can prove about the first worker’s effect.

SQS standard queues explicitly permit duplicate delivery. Kubernetes also warns that a program in a Job can sometimes start more than once even with a single completion configured. Use those documented behaviors to motivate an execution contract that tolerates repeated attempts. Avoid promising exactly-once business outcomes merely because a scheduling layer offers deduplication.

Begin with one workflow: a user requests a large export, receives a job identifier and checks status later. Clarify the maximum duration, acceptable delay, output retention and whether canceling should delete an already produced artifact. Separate an accepted request from an executing attempt and from a completed business result.

Define invariants before drawing workers: accepted work survives a process crash; one logical export identity maps to one current result; retries cannot create unauthorized outputs; cancellation and completion have explicit precedence. The examples in this guide are interview design proposals, not vendor benchmarks or claims about a specific hiring loop.

Close the gap between application state and enqueue

Suppose the API inserts a report request, then publishes to a queue. If the process dies between those operations, the request exists but no worker knows to handle it. Reversing the order can produce a queue message for a request that never committed. Explain the split-write problem before proposing more workers.

A transactional outbox is one possible answer: insert the request and its publication intent in the same database transaction. A relay sends the intent and records progress. Because a relay can crash after sending, the worker still needs idempotency. Alternatively, use a database-backed queue whose eligibility state commits with the request. State the tradeoff you are choosing.

PostgreSQL documents SKIP LOCKED as useful for avoiding contention in queue-like tables, while warning that it provides an inconsistent view unsuitable for general-purpose queries. If you choose that mechanism, describe a short claim transaction, an index on eligible work and a bounded batch. Do not hold a database transaction open during a lengthy export.

Record logical job identity separately from attempt identity. The API returns the former; every worker execution gets the latter. A unique request key can prevent duplicate submissions when the client retries after an ambiguous response. Bind that key to the authenticated owner and original parameters so it cannot accidentally reuse someone else’s job or change the requested operation.

Recover ownership without trusting a paused worker

SQS visibility timeout hides a received message for an interval; if processing does not finish, the message can become visible again. Think of that as temporary delivery ownership, not proof that a previous worker has stopped executing. A long pause or network partition can outlive the interval.

For your illustrative queue, store a claim token and lease expiry. A worker periodically renews its lease, and its completion update checks the expected token. A later owner receives a new token. Explain which output operation also needs protection against stale owners: rejecting a stale status update is insufficient if that worker can overwrite the result file.

Use immutable output names tied to an attempt or content hash, followed by a conditional update that publishes the winning artifact reference. Keep unauthorized and abandoned outputs out of the user-visible namespace. Have a cleanup policy for orphaned artifacts and avoid deleting an artifact still referenced by a completed job.

Size leases around expected operation behavior and recovery needs, not an arbitrary large timeout. Ask whether the operation can checkpoint or heartbeat. If the worker loses ownership, it should stop launching further effects where possible. Be explicit that cancellation or a lost lease cannot undo an effect the external system has already accepted.

Make retries a bounded recovery policy

Classify errors into transient dependency failure, invalid input, permission failure and internal defects. Retry a dependency timeout with a budget; do not repeatedly parse the same invalid document forever. The AWS Builders’ Library explains how timeouts, backoff and jitter help control retry load, and why side effects require care when a response is uncertain.

Set an attempt limit, an elapsed-time limit and an expiry for work that no longer has value. Choose the limits with the business owner and identify what happens when they are reached. A report requested before a meeting can become useless afterward even if it eventually succeeds. Persist the next eligible time so a restart does not reset the budget.

SQS dead-letter queues provide a place for messages that repeatedly fail under the configured redrive policy. A dead-letter queue needs an owner and an inspection process. It is not successful completion. Store an actionable error category, protect payload access and specify how a corrected job is replayed without resetting its business identity accidentally.

Separate scheduling fairness from failure recovery. A customer with many expensive jobs should not starve small jobs from other customers. Consider per-tenant concurrency limits, workload classes and admission control before increasing global worker count. Monitor wait time by class; a healthy average can hide a queue whose interactive work never reaches the front.

Choose a workflow engine when state transitions justify it

A single export and a multi-day approval flow have different coordination needs. Temporal models durable Workflow Executions, and its Event History records execution events. Workflow definitions have determinism requirements because histories can be replayed. Those capabilities can support recovery of orchestration state; they do not automatically make every external activity idempotent.

If you propose a workflow engine, explain the division: workflow code decides what should happen next, while activities perform effects and may retry. Give the payment or email activity a stable operation identity. Define the outcome when an activity succeeded externally but its completion was not recorded. Your answer should retain the same ambiguity reasoning as a simpler queue.

Consider compatibility when deploying new workflow code. Existing executions may resume with older histories. Describe your versioning or migration approach and test it with a representative in-progress execution. Avoid assuming that replacing a container is equivalent to safely changing a durable workflow definition.

Cancellation also needs semantics. Can it prevent a not-yet-started activity, signal a running one or only suppress publication of its result? Make the UI show those distinctions. A user should not see “canceled” while an irreversible effect is still being silently applied. For an interview, explaining this boundary is more valuable than listing every feature of an orchestration product.

Build an operating plan that makes stuck work visible

Track queue wait, execution time, lease expiry, oldest unresolved job, terminal failure and accepted-to-result time. Link logs and traces through the logical job and attempt identifiers. Keep high-cardinality identifiers in logs rather than turning each job into a metric label. Define which signal pages an operator and which belongs in a routine review.

Test a specific sequence: enqueue without relay completion, worker crash before effect, worker crash after effect, expired lease with a paused worker, cancel during execution and replay after a defect fix. For each case, explain the persistent state and user-visible status. These are a practice checklist, not evidence of any particular production system’s reliability.

Practice a concise architecture argument: “I will commit request and queue intent together, claim work with temporary ownership, make the effect idempotent, publish completion conditionally and surface exhausted recovery.” Then let the interviewer change one assumption, such as a job lasting days or a result becoming irreversible.

Durable background work is a state-management problem with a scheduler attached. Your design becomes defensible when the application can explain what it accepted, what effect occurred and what recovery remains possible. Rehearse that explanation before spending the rest of the interview on worker autoscaling or queue throughput calculations.