The Staff Engineer Resource Scheduling Interview: Placement, Fairness, and Preemption

Your cluster reports idle CPU, yet an urgent workload cannot start. A large tenant keeps winning the queue, and evicting a small job still does not create a usable machine. A resource-scheduling interview becomes interesting when you explain why those outcomes can all occur in the same system.

For senior and staff engineers, this is a chance to connect algorithms with product policy. You must decide whose work runs, where it fits, what may be interrupted, and which promise the platform can defend. This guide uses primary research and project documentation to build a rehearsal scenario. Its calculations, proposed policies, and practice rubric are illustrative, not an employer's assessment criteria.

1. Define the promise before choosing a scheduler

Use one concrete prompt: design a shared compute platform for several engineering teams running API services and batch analysis. Ask whether tasks are independent, whether some must start together, which resources they require, and whether interruption loses valuable work. Establish the difference between a task needing one machine and a distributed workload needing coordinated capacity.

The 2015 Borg paper describes combining admission control, task packing, overcommitment, and performance isolation. Its lesson for this exercise is that utilization comes from interacting policies. A priority queue alone cannot express the entire resource-management contract. Treat Borg as a historical design reference, not a claim about today's Google implementation.

Propose explicit states: submitted, queued, admitted, assigned, running, and completed, with cancellation and failure paths. Define admitted as permission to consume a quota budget; assigned means a placement was committed. Neither means the application can serve traffic. Kubernetes makes a related distinction between PodScheduled and Ready conditions. Keep your platform's states separate from Kubernetes API field names.

Then negotiate a measurable target. For this hypothetical platform, interactive services might need reserved headroom, while batch tasks accept variable queueing delay. Ask whether the objective is time to assignment, time to readiness, deadline completion, or throughput. Record what happens when capacity is unavailable: wait with a reason, reject, offer a different resource class, or request explicit permission to relax a preference.

2. Model capacity as a vector on each machine

Represent a task using CPU, memory, accelerators, storage requirements, and placement constraints. Distinguish requested resources from observed consumption. The Kubernetes resource-management documentation explains that scheduling uses requests, while runtime limits govern consumption. On Linux, CPU limits involve throttling; memory-limit enforcement can involve reactive out-of-memory termination. These are different operational consequences, not interchangeable safety margins.

Illustrative calculation: aggregate capacity can be unusable

Assume two nodes each offer 8 CPU units and 16 GiB of allocatable memory. Existing reservations on each node consume 6 CPU units and 4 GiB. Across the cluster, 4 CPU units and 24 GiB remain unreserved. A new task requesting 4 CPU units and 8 GiB still fits nowhere: each individual node has only 2 CPU units available. Ignore all other constraints for this calculation.

That example exposes fragmentation without invented production statistics. Ask which action could make a feasible placement: a larger node, finishing another task, or moving interruptible work. Reducing the task's request changes its resource contract and needs evidence. Low observed CPU does not automatically authorize promising away capacity already requested by other workloads.

Classify constraints before optimizing. Kubernetes node affinity distinguishes required rules from preferred rules. In your design, hardware compatibility and an isolation requirement belong in feasibility checks; a preference for nearby data may belong in scoring. Explain who may edit these constraints. A scheduler should not silently discard a required isolation rule merely to shorten the queue.

3. Separate feasibility, scoring, and commitment

Draw the placement path as a sequence: select eligible work, filter nodes, score feasible candidates, reserve resources, commit the assignment, and observe execution. The Kubernetes scheduler documentation describes filtering followed by scoring and binding. An attractive score cannot make an infeasible node valid. This separation also gives you intelligible failure reasons when no candidate survives.

Choose a scoring objective that follows the workload. Packing batch tasks tightly may leave whole nodes available for other uses; distributing service replicas may reduce shared failure exposure. Topology spread constraints make placement across failure domains explicit. Explain whether spread is mandatory or preferred, and what your platform does when one zone lacks capacity. Name the availability and cost tradeoff instead of hiding it inside a weighted score.

Defend the gap between choosing and binding

Two scheduling decisions can observe the same free capacity before either finishes. In a custom design, propose an authoritative assignment record with a task identifier, attempt generation, selected node, reservation, and status. Use a conditional state transition to establish ownership. Keep this proposal distinct from an assertion that a particular Kubernetes plugin implements your entire durability model.

The Kubernetes scheduling framework provides Reserve before binding and Unreserve when reservation or later phases fail; Unreserve implementations must be idempotent. Use that concrete lifecycle to discuss cleanup. In your proposed system, reconcile an uncertain bind outcome against authoritative state before releasing its reservation or issuing another assignment. Otherwise a timeout could create duplicate execution or misleading capacity accounting.

4. Make fairness a policy you can calculate

Equal task counts are a poor starting point when one tenant submits tiny CPU tasks and another submits memory-heavy analysis. First choose the fairness unit: team, project, or customer. Then define entitlements, borrowing rules, priority classes, and limits on outstanding submissions. Explain which allocations count against each tenant while work is admitted but has not started.

The original Dominant Resource Fairness paper defines a user's dominant share as its largest share of any resource and develops allocation policies around that quantity. It offers a useful way to reason about several resources together. It does not mean that every real cluster, with discrete tasks and placement constraints, can immediately reach an identical share for every tenant.

Illustrative calculation: in a pool with 16 CPU units and 64 GiB, tenant A holds 4 CPU units and 8 GiB. Its dominant share is max(4/16, 8/64), or 25%. Tenant B holds 2 CPU units and 24 GiB, giving max(2/16, 24/64), or 37.5%. Under an equal-weight policy inspired by DRF, A would have the lower dominant share. A's next task must still fit; this comparison alone does not determine a legal placement.

Use Kueue's ClusterQueue documentation to make queue policy concrete: quotas and cohort borrowing are explicit, and StrictFIFO can block newer work behind an older workload that cannot be admitted. BestEffortFIFO allows newer work that fits the available quota to proceed. That choice trades head-of-line blocking against the risk that large work waits while smaller work repeatedly fits.

For your hypothetical platform, propose a bounded bypass policy and an escalation path for aging large tasks. Track oldest eligible wait by tenant and resource class. Specify whether fairness considers current allocation, historical consumption, or both. Kueue's fair-sharing overview distinguishes admission fairness using historical queue usage from preemption-based sharing. Name the mechanism you intend to use; the word “fair” is not a configuration.

5. Price interruption before enabling preemption

Priority answers which work deserves earlier consideration. Preemption answers whether running work may lose its allocation. The Kubernetes priority and preemption documentation supports non-preempting priority: a workload can receive higher queue priority without removing running Pods. Its PodDisruptionBudget protection during scheduler preemption is best effort, not an absolute guarantee against disruption.

For the interview scenario, classify workloads as non-interruptible, restartable, or checkpointable, and ask who validates that classification. Estimate the lost work, restart delay, and downstream effects before selecting victims. If a task has no durable checkpoint, eviction does not magically preserve its progress. Application recovery is part of the proposed service contract and needs its own evidence.

Return to the fragmentation example. Removing a memory-heavy task from a different node may free resources without creating the CPU shape the urgent task needs. Evaluate the resulting feasible node before interrupting anything. Include termination delay in the start-time estimate. Prevent repeated eviction cycles with an explicit policy for victim eligibility, interruption budgets, and admission cooldowns; present those as design choices you would test.

6. Explain pending work and prove useful progress

Report why work is waiting: quota exhausted, incompatible hardware, insufficient capacity on any eligible node, placement constraints, or an assignment that has not become ready. Preserve the task's requirements and the relevant decision version so an operator can reconstruct the explanation. A single “Pending” counter cannot show which policy or capacity change would help.

Kubernetes node-autoscaling documentation describes using Pod scheduling constraints and autoscaler configuration to determine provisioned nodes. It also distinguishes requested resources from actual runtime usage. In your answer, make autoscaling conditional on a compatible node option existing. Adding generic machines will not satisfy an impossible hardware request or a contradictory placement rule.

Propose separate measurements for submission-to-admission, admission-to-assignment, and assignment-to-readiness. Also track failed placements, reservation cleanup, interrupted work, and queue age across tenants. Pair a utilization improvement with service readiness and interruption measurements. For this practice system, success means useful work progresses within the agreed policy; keeping every machine numerically busy is only one signal.

Illustrative practice rubric

Rehearse with four follow-ups: a large task never fits, one team floods submissions, a scheduler crashes during binding, and an urgent service arrives during a batch run. Score each answer from zero to two on contract clarity, feasible placement, tenant policy, and recovery evidence. Zero means missing, one means named, and two means explained with a concrete failure case. This is a self-assessment exercise, not a validated hiring score.

Revise whichever explanation cannot survive a changed assumption. If tasks must start together, revisit admission and coordinated reservation. If interruption loses hours of work, revisit headroom and victim policy. Bring the conversation back to one task throughout its lifecycle. That makes your reasoning visible: who authorized its resource use, why this node was valid, and what evidence proves the work actually started.