The Staff Engineer Sandboxing Interview: Untrusted Code, Isolation, and Resource Limits

An interviewer asks you to design a service that compiles and runs code submitted by strangers. A queue and a pool of containers are an easy starting diagram. The harder questions are what that code can reach, how much damage it can cause without escaping, and what evidence lets the platform safely accept the next submission.

For senior and staff engineers, this is an exercise in defining trust boundaries. A working program is only one outcome; a runaway compiler, a missing worker, and a failed cleanup also need explicit semantics. This guide uses a hypothetical coding-practice platform to connect isolation mechanisms to operational decisions. The proposed architecture, numerical examples, and practice rubric are illustrative preparation tools, not a description of a particular employer's interview process.

1. Define the execution contract before choosing a runtime

Start by separating the person who submits a program from the authority that executes it. Assume submitted source, dependencies, filenames, and output are hostile. Ask which languages and operating-system features must work, whether submissions require network access, and whether two customers may share a physical host. Distinguish ordinary programming mistakes from a deliberate attempt to read another customer's data or exhaust capacity.

For the practice platform, propose a narrow contract: a submission references a supported language image and bounded input; the platform returns an exit classification, capped output, and resource measurements. Compilation and execution both happen inside an isolation boundary. The control plane stores ownership and scheduling state, while a separate worker fleet handles untrusted computation. Neither the application database nor its credentials belongs in the execution environment.

Write invariants that the rest of the answer must defend: one submission cannot read another's workspace; execution cannot obtain platform administration credentials; output cannot grow indefinitely; a worker cannot declare reusable capacity until cleanup is confirmed. These are proposed product requirements. They give you specific failure cases to discuss instead of an untestable promise that the service is secure.

2. Explain what each isolation layer actually protects

Linux namespaces provide separate views of resources such as processes, mounts, and networking. They are useful building blocks, but naming a container does not establish a separate kernel boundary. The kernel's seccomp documentation explicitly describes syscall filtering as an attack-surface reduction mechanism rather than a complete sandbox. Explain how reduced privileges, filesystem isolation, and resource enforcement work together.

Two possible stronger boundaries illustrate the design choice. gVisor's security model interposes its Sentry implementation between application system calls and the host. It also leaves resource-exhaustion defenses to host mechanisms and network-policy enforcement to surrounding controls. The 2020 Firecracker paper describes a specialized virtual machine monitor for lightweight serverless virtualization. A microVM supplies a guest-kernel boundary, with a different compatibility and operating-cost profile.

Choose based on the workload and threat model, then state what you would measure: compiler compatibility, syscall-heavy workloads, startup latency, memory overhead, and recovery behavior. Avoid an absolute ranking of security from product names. Firecracker's production host guidance includes host hardening and the jailer; adopting the VMM alone does not reproduce an entire production security posture. Patching the host, runtime, guest, and language images remains part of the design.

3. Bound CPU, memory, processes, and elapsed time separately

A program does not need to escape to harm its neighbors. Give compilation and execution separate budgets, including CPU, memory, process count, scratch storage, output, and wall time. The Linux cgroup v2 interface supplies distinct controls: cpu.max limits CPU bandwidth, memory.max sets a memory ceiling with cgroup OOM handling, and pids.max constrains task creation. CPU throttling does not implement an elapsed-time deadline. A blocked or sleeping job can use little CPU while occupying a worker.

For an illustrative policy, allow a short compile phase and a separately timed execution phase, with an external supervisor enforcing each deadline. Specify how the result distinguishes compilation failure, runtime failure, memory exhaustion, elapsed-time expiry, and platform failure. Where evidence is insufficient, report an unknown infrastructure outcome rather than inventing a user-code verdict. Preserve enough diagnostics for support without returning host paths or privileged environment details.

Resource limits also need room for the isolation machinery. Reserve host capacity for the supervisor, runtime, logging, and operating system before promising slots to users. Protect the result collector and scheduler from the same workloads they police. A timeout handler that becomes unavailable under memory pressure is a weak foundation for the execution contract.

4. Restrict network access and ambient credentials

For code that only reads standard input and writes standard output, propose no external network access. If package installation is necessary, move it into a separately controlled preparation stage with approved sources and bounded downloads. Arbitrary package installation is itself code execution. Do not give that stage broader access simply because it runs before the user's main program.

When Kubernetes is the orchestrator, its NetworkPolicy documentation makes two useful limitations explicit: enforcement requires a supporting network plugin, and allowed traffic is the union of applicable policies. A policy object alone does not prove denied egress. Establish isolation before starting work, account for DNS and reachable infrastructure, and test the effective network boundary. Do not equate a domain allowlist with complete application-level traffic inspection.

Credentials are another network path. Kubernetes documents how to disable automatic service-account token mounting with automountServiceAccountToken: false. Give execution environments no unnecessary Kubernetes or cloud identity, and block access to infrastructure credential endpoints through independently enforced controls. The trusted worker agent may need narrowly scoped authority to fetch an input and upload a result; the submitted process should not inherit that authority.

The Restricted Pod Security Standard provides concrete hardening requirements around non-root execution, privilege escalation, capabilities, and seccomp. Treat those controls as a baseline to enforce and inspect. The orchestration policy complements the chosen sandbox boundary; it does not answer every question about hostile workloads sharing hardware.

5. Treat inputs, outputs, and artifacts as hostile data

Pin supported execution images by immutable digest so a language version is part of the submission record. Kubernetes' image documentation distinguishes mutable tags from digests that identify image content. Pair that identity with the sandbox policy version and resource profile. Reproducing a failure requires knowing what ran, not just that the user selected Python.

Validate filenames, archive expansion, total input bytes, and permitted destination paths before materializing a workspace. Keep trusted runtime files read-only where possible and give each attempt a fresh writable area. If you export files, copy only the permitted artifacts through a trusted boundary; do not let user-controlled paths select arbitrary host files. Apply quotas to extracted size and scratch use as well as uploaded size.

Output is also a capacity and security boundary. Cap stdout and stderr bytes while continuing to drain or terminate the producer according to a documented policy. Otherwise a full pipe can change execution behavior and confuse timeout diagnosis. Render output as text and handle terminal control sequences deliberately in logs. An HTML preview, if required, deserves its own browser isolation design. A safe server process does not automatically make every returned byte safe to display.

For illustration, 500 concurrent jobs each emitting 2 MiB produce nearly 1 GiB of output before transport and indexing overhead. Those assumed values show why a per-job cap alone is insufficient: also bound collector memory, upload concurrency, retained data, and aggregate tenant usage. Decide whether exceeding the output cap truncates results or terminates execution, and expose that decision clearly to the user.

6. Make cleanup and uncertain outcomes first-class states

Model an attempt through queued, preparing, running, collecting, and cleaning states, followed by a terminal result. Give it a unique attempt identifier, a lease, and an owning worker. If a worker disappears, the scheduler must not assume the old execution stopped. A replacement attempt can overlap it. Prevent stale attempts from overwriting the accepted result by checking current ownership when publishing results.

In this hypothetical platform, duplicate execution is tolerable only because jobs have no external side effects and remain independently bounded. If you add user network calls or persistent writes, revisit that assumption. Retry policy follows the execution contract. It cannot be justified solely by the queue's delivery guarantee.

Cleanup must account for descendants and runtime-owned resources. The cgroup documentation describes subtree population reporting and cgroup.kill; stopping one process is not proof that the whole workload is gone. Confirm teardown of the sandbox, workspace, mounts, and network attachments before returning capacity. Quarantine a worker when cleanup cannot be verified, and use a trusted reconciler to find orphaned attempts.

Warm capacity can reduce startup delay, but distinguish an unused, prepared environment from a previously used writable workspace. Reuse requires an explicit reset argument. If restoring snapshots, review Firecracker's snapshot guidance, including security considerations, rather than assuming a restore creates fresh identity or fresh secrets. Keep tenant inputs out of shared base state.

7. Connect admission control to the isolation promise

Use the example to show how capacity reasoning affects correctness. With an illustrative arrival rate of 20 jobs per second and mean active time of three seconds, the average active population is roughly 60 jobs in steady state. That calculation assumes stable traffic and excludes queue time. It is not a safe worker count: bursts, long compilations, runtime overhead, reserved headroom, and multiple resource bottlenecks still need measurement.

Admit work against tenant quotas and available resource profiles rather than letting every accepted request immediately create a sandbox. Bound queued submissions and communicate overload explicitly. Separate short interactive runs from expensive compilation workloads when their service objectives differ. Track queue delay, preparation time, execution time, output volume, failed cleanup, orphaned attempts, and host pressure. A fast median run can coexist with a fleet that is slowly losing reusable capacity.

8. Rehearse the boundary failures, not just the happy path

Use this illustrative practice rubric: for every protection, name its trusted enforcer, observable evidence, and failure response. Ask a partner to introduce one change at a time: a program creates many descendants, stdout never stops, the worker loses its lease, a network policy is ineffective, or workspace removal fails. Explain which component contains the problem, what result the user receives, and whether capacity remains eligible for reuse.

Then change a product requirement. Add outbound requests, customer-provided images, or persistent workspaces. Identify which original assumptions no longer hold before drawing more components. This is where the exercise becomes useful senior-level preparation: the architecture is a set of conditional promises, and you can explain how those promises change when the trust model changes.

End your answer by tracing one submission from admission to confirmed teardown. Tie the selected isolation boundary to explicit access restrictions, bounded resources, trustworthy result publication, and measured recovery. That coherent explanation is more useful preparation than memorizing a runtime's feature list.