The Staff Engineer Batch Processing Interview: Backfills, Data Quality, and Safe Publication

Your daily revenue report is green, but yesterday's totals changed after a retry. Then a correction removes every row from one partition, and the dashboard keeps showing the old numbers. A batch-processing design becomes interesting when you have to explain exactly which result deserves to be published.

Use this practice prompt: design a daily merchant revenue pipeline, then repair ninety days after discovering a calculation bug. The problem is bounded computation over named inputs, with explicit rules for replacing derived results. This guide offers an authored preparation framework for senior and staff engineers; it does not describe an employer's rubric or predict interview outcomes.

1. Name the business date and the input cutoff

Start with the report's meaning. Propose that the November 1 report covers transactions whose business timestamps fall within that date in the merchant's agreed timezone. Specify whether refunds adjust their original transaction date or appear on the refund date. Then define an arrival cutoff, such as 06:00 the following morning. These are illustrative product choices to negotiate, not universal accounting rules.

Keep three clocks visible: the business interval being reported, the latest input arrival accepted into this revision, and the time the computation executes. Airflow's Dag Run documentation distinguishes the data interval from the logical date and actual execution time. A scheduled daily run normally starts after its interval ends; its logical date identifies the interval's start.

A cutoff is a policy, not proof of completeness. Ask how each upstream source declares its delivery complete and what happens when that declaration is missing. Propose a publication record containing business date, cutoff, input revision, output revision, and status. Consumers should be able to distinguish a provisional report from an approved one without interpreting a scheduler timestamp.

Express the business interval with an inclusive start and exclusive end, and pass its timezone and boundaries as explicit parameters. Keep the cutoff separate from that interval: a transaction can belong to the business date while arriving after its first publication deadline. In your example, decide whether that transaction waits for a correction or prevents initial publication. That decision determines which completeness checks and revision labels you need.

2. Make a rerun reproduce a named computation

Define a computation by its input snapshots, transformation artifact, configuration, and business interval. Record the code commit and executable artifact digest, dependency versions, reference-data versions, and any rounding or currency-conversion policy. A date partition alone is insufficient if somebody can mutate its contents. Your proposed manifest must identify what was actually read, not simply where a worker looked.

Airflow's task best practices recommend reading and writing specific partitions, avoiding “latest” inputs, and producing the same outcome on retries. They also caution against using the current time for critical computation. Apply that guidance by pinning inputs at run creation and passing the manifest through the workflow.

Separate retry from correction

A retry repeats revision A because execution failed. A correction creates revision B because the accepted inputs or calculation changed. Give each attempt an identifier for diagnostics, but use the computation identity to govern publication. Retrying a finished attempt must not append the same merchant totals twice. Define idempotency as the same externally visible result after repetition, then explain whether you achieve it through replacement, keyed writes, or a guarded commit.

Draw a small control record: planned, running, staged, validated, published, or rejected. These are proposed application states, not built-in Airflow states. Store the evidence needed to resume between them. If a worker dies after publication, recovery should inspect the published revision before attempting another change.

3. Require evidence of quality and completeness

Split validation into separate questions. Did every expected input arrive? Does the transformation preserve the intended business meaning? Is the result recent enough for its users? dbt data tests support assertions such as uniqueness, non-null values, accepted values, and relationships. Their queries identify failing records; zero failing rows makes a test pass.

That last detail matters: an empty relation can contain no nulls or duplicates while still representing a failed extraction. Add an explicit completeness gate against the input manifest, expected merchant population, and source control totals. For the practice pipeline, reconcile accepted transactions, excluded records, refunds, and output totals using a stated policy. Choose tolerances with the report owner rather than inventing an unexplained percentage.

Timeliness needs its own check. dbt source freshness evaluates configured freshness information and can use warning and error thresholds. A recent maximum load timestamp does not establish that every expected merchant arrived; a structurally valid table does not establish that it is current. Treat freshness and completeness as separate publication conditions.

Represent approved empty output explicitly: the source completed, all relevant records were evaluated, and the business result contains zero rows. Preserve the previous published revision when extraction or validation fails. Also wire failure propagation deliberately: Airflow's Dag Run status documentation warns that a successful leaf task using an all_done trigger rule can make the run succeed despite an earlier task failure. A green orchestration status is insufficient publication evidence.

4. Stage the result before making it authoritative

Keep computation separate from reader visibility. Write an isolated candidate revision, validate that exact revision, and expose it through a controlled publication step. Readers should resolve one approved revision for their query. A worker writing its last output file should not automatically declare the entire business result ready.

Apache Iceberg's reliability documentation provides a concrete table-level mechanism: snapshots track table files, and commits atomically replace the reference to the current table metadata. Readers use a consistent snapshot. Iceberg also describes optimistic concurrency, with conflicting writers validating their assumptions before retrying a commit.

Explain the scope of the guarantee you need. One atomic table update is not evidence that your merchant report, monthly aggregate, and exported statement all became visible together. Propose a release manifest naming the approved snapshot of each required output, and have participating consumers resolve that manifest. State which consumers obey this protocol and what remains independently published.

Make the manifest immutable and publish its identifier through one atomic, conditional update to the authoritative release record. Require the previously approved revision to match the publisher's expected base. If it differs, stop and resolve the conflict. This is an application protocol to implement with a transactional or conditional-write primitive; the manifest file alone provides no atomic publication guarantee.

Iceberg's audit-branch workflow illustrates writing separately, validating the branch, and then advancing the main branch. Use it as an implementation option, subject to the selected catalog and engine's supported operations. Define what happens if production advances during validation: revalidate against the new base or reject the stale candidate. Keep the approved manifest tied to the revision that passed its gates.

5. Propagate corrections through business dependencies

Now introduce a late refund or corrected exchange rate. Ask whether the product promises an “as known at cutoff” historical report, a restated report using corrected knowledge, or both. Preserve that distinction in revision metadata. Silently changing the input definition while retaining the same report identity makes later reconciliation ambiguous.

Track affected outputs by business dependency, not only by task execution order. In this example, correcting November 1 may require rebuilding November's merchant aggregate. If another model uses a rolling seven-day window, identify every window containing the corrected date. Mark those outputs stale until their replacements are validated. This dependency policy is part of the proposed design; scheduling a rerun does not itself prove the downstream result was repaired.

Handle a correction that removes everything

Choose replacement semantics carefully. Iceberg's Spark write documentation says dynamic overwrite replaces partitions that have rows in the query result. Consequently, an empty corrected result needs explicit handling: do not assume that producing no rows clears a previously populated partition. Use a supported operation whose declared scope includes the intended deletion, and validate both removed and retained data.

Give consumers a revision transition to acknowledge. Record which exports or aggregates consumed the old revision and which have adopted the replacement. For externally delivered statements, propose a correction workflow with an owner; rewriting a warehouse table is not evidence that an already delivered artifact changed. Retain rollback evidence according to an agreed retention and access policy.

6. Budget backfills alongside today's production work

Specify the date range and whether the operation creates missing runs, retries failures, or deliberately recomputes completed dates. Airflow's backfill controls distinguish those reprocessing behaviors. They also provide a backfill concurrency limit applied independently of the Dag's normal max_active_runs setting. Explain both limits instead of assuming one protects all workloads.

Limit the scarce resource as well as run count. Airflow pools bound parallel tasks, and a heavy task can consume multiple pool slots. Propose separate reserved capacity for daily publication and a bounded backfill allocation, with additional limits on source reads, warehouse concurrency, and destination writes. Stop admitting new repair work when the current daily deadline is at risk.

Use an illustrative estimate: ninety daily partitions at twenty minutes each represent 1,800 partition-minutes. Four equally effective slots give a capacity lower bound of 450 minutes, or 7.5 hours. With indivisible partitions, 23 waves actually need 460 minutes before overhead. These calculations assume independent, uniform work without contention or retries; they are not benchmarks. Sample a representative date and a known heavy date before making a delivery estimate.

Explain what you would inspect when one partition dominates. Spark's SQL tuning guidance describes runtime adaptive query execution, shuffle partition coalescing, and handling skewed joins. Inspect the physical plan and task distribution before adding workers. Track bytes scanned, spill, failed attempts, oldest unfinished partition, and validated publications so fast computation cannot hide a stalled release.

7. Carry data permissions into repair operations

Define who may read source history, create candidates, approve publication, and retrieve rejected rows. Historical repair should use the same explicit tenant and data-access boundaries as routine processing. For an AWS implementation, IAM best practices recommend temporary workload credentials through roles and permissions limited to the required actions and resources.

Apply those principles to staging tables, branch access, debug samples, and exports. Give validation enough access to check invariants without copying customer payloads into broadly readable logs. Airflow's masking documentation warns that sensitive values passed through side channels may not be masked when printed downstream. Automatic secret masking is therefore insufficient as your proposed policy for protected business records.

Use synthetic or appropriately redacted examples in interview preparation. Describe access checks with identifiers and aggregate evidence. For the production design, assign ownership for candidate cleanup and retained historical revisions, including the permission to inspect them. A failed backfill should leave a diagnosable record without creating an ungoverned second dataset.

8. Rehearse the decision at each publication boundary

Open your mock answer with a compact contract: “I will compute a named business interval from pinned inputs, validate an isolated candidate, and publish an approved revision. Repairs create explicit replacements, and downstream consumers acknowledge the change. Backfills have a capacity budget that protects the next daily deadline.” Then draw only the components needed to uphold that contract.

Use this authored practice rubric: can you identify the authoritative revision after a retry, explain what makes zero rows valid, prevent a stale candidate from replacing a newer result, and name the downstream work caused by a correction? These are rehearsal questions, not a hiring scorecard. Ask a practice partner to interrupt after staging, after validation, and immediately after commit.

For each interruption, state the evidence you would inspect, the operation you can safely repeat, and what readers see meanwhile. Finish by choosing the person responsible for approving a restatement and the signal that proves its consumers adopted it. The strongest result of this exercise is a design you can explain one revision at a time, including the unfinished work a green dashboard might conceal.