The Staff Engineer Build Systems Interview: Cache Keys, Hermeticity, and Provenance

Imagine two CI workers build the same commit and produce different binary hashes. Both report success. Before you discuss faster builds, decide which artifact your release process can trust.

That scenario makes a useful senior or staff engineer interview exercise because it connects dependency modeling, distributed execution, performance, and supply chain trust. Start with a precise promise: the system reuses an output only when its recorded inputs describe the work requested, and consumers can verify how a published artifact was produced. Then explain how you would test that promise.

This guide develops that answer using build-system documentation and software supply chain specifications. The design scenario, calculations, and practice rubric are illustrative preparation tools, not a company's interview criteria.

1. Make the input graph a correctness contract

Ask what the system builds, which platforms it targets, and which outputs must remain stable. Separate a developer's incremental compilation from a release artifact and an integration test that contacts a changing service. Propose explicit correctness boundaries for each rather than applying one caching policy to all three.

Bazel describes an action as a build step with inputs, output names, a command line, and environment variables. Required inputs and expected outputs are declared. Use that model to draw a graph: a schema generator produces source files, compilation consumes them, and packaging consumes the compiled output. Each dependency becomes an edge the scheduler and cache can reason about.

A commit hash is only part of the identity

For your proposed system, describe a schematic action key as a hash of the action definition, input digests, toolchain identity, relevant environment, and platform configuration. This is a design sketch, not Bazel's exact serialization. A source revision alone leaves unanswered which compiler, flags, dependencies, and generated files participated.

Consider a generator that reads a configuration file absent from its declared inputs. A configuration change should invalidate generation and downstream compilation. If the graph omits that edge, the cache can return an outdated result while reporting success. Bazel's sandboxing documentation identifies undeclared inputs as a cause of incorrect incremental builds and warns that incorrect shared entries affect other developers.

In the interview, walk one edit through the graph. A documentation-only change should not require recompiling an unrelated library in your design. Changing that library's source should invalidate its action and the downstream actions that consume its changed output. An overly broad key sacrifices reuse; an incomplete key sacrifices correctness. Explain how you would compare the affected-action set against expected dependencies.

Also decide when inputs are frozen. Use an immutable checkout or a captured input snapshot for the proposed CI pipeline. Preserve that snapshot's identity in diagnostics so a later retry can request the same work. A mutable branch name can be convenient at submission, but resolve it to a specific revision before defining the build.

2. Verify hermeticity and deterministic output separately

Bazel's hermeticity guide emphasizes isolation from host changes and specific versions of tools and dependencies. A compiler discovered through a worker's local installation weakens that boundary. For your design, supply the toolchain through declared inputs or a controlled, identified execution environment instead of assuming every worker has equivalent software.

Be precise about reproducibility. The Reproducible Builds definition requires another party, given the same source, build environment, and instructions, to recreate the specified artifacts byte for byte. Define those artifacts explicitly. An executable might be in scope while diagnostic logs and invocation identifiers remain outside it.

Design an independent rebuild check that executes the work again without accepting the original cached result. Compare artifact digests across clean workers under the declared conditions. Reading the same object from a shared cache twice demonstrates retrieval consistency; it provides no evidence that two executions produce identical bytes. Retain mismatches for investigation instead of silently selecting whichever output arrived first.

Give time-dependent metadata a policy

The SOURCE_DATE_EPOCH convention lets supporting tools consume a stable timestamp expressed as seconds since the Unix epoch, typically derived from source modification time. Use it where appropriate, and check that the actual tools honor it. Setting the variable does not establish that every timestamp or other source of variation has disappeared.

Container caching introduces a separate question: what causes execution again? Docker documents that a cached RUN instruction is not invalidated simply because time passes. Changing a build secret's contents also does not invalidate its cache entry, while file modification time alone is excluded from the checksum used for copied files. Explain which changing inputs your recipe intentionally models and which freshness checks occur outside ordinary cache lookup.

Finally, verify the isolation actually enforced. Bazel's sandboxing guide distinguishes strategies and notes that sandboxing does not hide the entire host environment. Specify the filesystem and network restrictions your execution service needs, then test them. A container image label or a successful local sandbox run is insufficient evidence for every claimed isolation property.

3. Treat shared caches as a trust boundary

The Remote Execution API separates an action cache from a content-addressable store. Keep their responsibilities distinct in your design: one answers which result belongs to an action; the other retrieves bytes by their digest. A matching content digest checks byte identity, but it cannot establish that the bytes are the correct output for the requested action.

Define who can publish that mapping. Bazel cautions about remote-cache writers and suggests that only CI might receive write access. A defensible practice design gives controlled CI workers permission to populate a trusted namespace, lets developer builds read according to policy, and places untrusted pull-request execution in a separate namespace. Explain the authentication and authorization checks behind that separation.

Protect reads as well as writes. Build outputs and captured logs can contain private project information; Bazel's remote cache stores action stdout and stderr. Include access control and retention in the cache proposal, and keep credentials out of action output. A globally readable cache would undermine the privacy assumptions of a private repository.

Measure useful work saved, not just hits

Illustrative calculation: suppose 1,000 actions contain 900 tiny actions taking 0.02 seconds each and 100 expensive actions taking 5 seconds each. Total execution work is 518 seconds. Caching all 900 tiny actions gives a 90% action-count hit rate but removes only 18 seconds, about 3.5% of execution work, even before lookup and transfer overhead. These invented values illustrate a measurement pitfall; they are not benchmark results.

Track avoided execution time, bytes transferred, lookup latency, and the longest dependency chain alongside hit rate. The calculation measures aggregate execution work, not elapsed build time: parallelism and the critical path change the latter. An interview answer becomes more credible when it distinguishes those units before proposing more workers.

For a suspected poisoned result, stop further writes to the affected trust domain, preserve the action key and diagnostic evidence, and rebuild through a trusted path. Describe how your service invalidates the affected mapping or replaces its cache namespace. Do not promise precise incident cleanup unless the proposed cache actually maintains the required ownership and dependency records.

4. Scale execution while keeping failures explainable

Remote execution distributes build and test actions across machines; remote caching reuses previous results. They can work together, but they solve different parts of build latency. In your proposed execution service, dispatch ready actions only after their inputs are available, bound queued work, and cancel actions no longer needed after a build is canceled.

Bazel distinguishes host, execution, and target platforms. The machine running the build coordinator, the worker running a generator, and the platform receiving the final binary may differ. Check tool compatibility with the execution platform and compiled-output compatibility with the target platform. “Everything runs in Linux” is too imprecise for a cross-compilation design.

The same documentation describes implicit dependencies carried in compiler state: one local action can succeed because a previous action left useful information behind, while separately executed remote actions fail. Model the necessary dependency explicitly. Scheduling the two actions on the same worker merely preserves the accidental coupling and makes future failures harder to explain.

Use a diagnostic matrix for flaky behavior

When an interviewer introduces intermittent failures, propose a repeatable investigation. Record the action key, worker environment identity, output digests, exit status, and infrastructure error category. Then choose experiments that distinguish these possibilities:

• Same declared identity, different artifact bytes: investigate an omitted input or nondeterministic execution. Repeat clean execution while varying one suspected factor.
• Unexpectedly different keys: compare action descriptions and configuration to find why equivalent requested work is receiving different identities.
• Alternating test outcomes: preserve each attempt and its environment. Check test data, shared state, timing assumptions, and external services before widening retries.
• Worker loss or transfer failure: classify the attempt as an execution-service failure and apply the stated retry budget, rather than presenting it as a reproducible compiler result.

This matrix is a proposed debugging method, not a published failure taxonomy. Explain how retries remain bounded and visible. A passing retry should retain the earlier failure in your build record, so the team can distinguish recovered infrastructure problems from unresolved flaky tests. State separately which outcomes your system permits in a shared cache.

5. Verify provenance against an explicit policy

The SLSA v1.2 build track describes increasing trust in artifact provenance and verification against consumer expectations. Its build provenance format connects artifact subjects to a build definition and run details, including builder identity. Use this to ask which builder, source revision, workflow, and parameters your consumer accepts.

Keep the cache inside that threat model. SLSA's provenance model treats communication with build caches and the control plane as implied by the builder identity, rather than recording every interaction directly. Therefore, signing a final artifact does not eliminate the need to secure cached inputs and results. Your accepted builder must include the cache behavior on which its claims depend.

GitHub artifact attestations provide signed provenance claims that include workflow and repository context, commit SHA, and triggering event. GitHub also states that attestations must be verified to provide a security benefit and do not guarantee an artifact is secure. Verification needs a policy, not merely a successful signature check.

For the illustrative pipeline, require an attestation matching the artifact digest, an approved signer and builder, the expected repository, and permitted build instructions. Reject an unexpected identity even if its signature is valid. Record the decision and its policy version. Treat independent reproducibility checks and vulnerability assessment as separate evidence with their own acceptance rules.

Practice the complete argument

Rehearse this prompt: “Design a shared build service for a monorepo that produces Linux release binaries and supports untrusted pull requests.” Follow one schema change from input capture through generation, compilation, cache lookup, execution, and artifact verification. Introduce a compiler update, a bad cache writer, and an intermittent test failure as follow-ups.

Author-created practice rubric: give yourself one point for each capability you can explain concretely: complete input modeling; independent output comparison; cache-writer authorization; bounded, observable failure handling; and consumer-side provenance policy. The five-point exercise helps locate gaps in your preparation; it predicts no company's hiring decision.

End your answer with a decision and the evidence that could overturn it. For example, start with trusted CI cache writes and a small remote-execution pilot, then expand only after clean rebuild comparisons and timing measurements support the change. The strongest rehearsal follows one artifact all the way from its declared inputs to the consumer's trust decision.