The Staff Engineer Load-Balancing Interview: Health, Connections, and Safe Draining

Imagine three backends with one HTTP/2 connection each. One connection carries 100 concurrent RPCs; the others carry 10 each. Connection counts look perfectly balanced, yet one server owns most of the active work. Now a rolling deployment removes that server. Where do its existing calls go?

Those numbers are an illustrative interview exercise, not a production benchmark. They expose a useful starting point for senior and staff engineers: define the work being distributed before choosing a balancing algorithm. A complete answer connects selection, health, and termination to a user-visible contract. This guide uses protocol specifications and implementation documentation; its scenarios and practice rubric are preparation tools, not an employer's assessment criteria.

Define the work you are distributing

Start by asking whether the service handles short requests, unary RPCs, long-running streams, or a mixture. Clarify whether calls have similar costs, whether a session must remain on one backend, and where TLS terminates. Then identify who discovers instances and who chooses the destination. A box labeled “load balancer” leaves all those decisions unresolved.

Connections and requests are different units

RFC 9113 defines HTTP/2 streams that multiplex concurrent exchanges over a TCP connection. Equal connection counts therefore do not establish equal concurrent request counts. The protocol also does not eliminate TCP head-of-line blocking. Explain which boundary your proposed balancer can inspect rather than assuming that multiplexing automatically distributes work.

The gRPC project's load-balancing architecture explanation distinguishes a transport-level proxy that forwards connections from an application-level proxy that understands HTTP/2 and can distribute streams across backends. Its 2017 discussion remains useful for that architectural distinction; it is not a current performance benchmark. In your design, mark both the client-to-proxy connection and the proxy-to-backend connections, then trace one RPC through them.

Put the selector in a real place

Client-side balancing requires a concrete discovery and policy story. The gRPC policy guide describes a resolver providing addresses and a policy maintaining backend connections and selecting one for an RPC. It documents pick_first as the default and round_robin as rotating RPCs among connected backends. Available policies vary by language, so verify the actual runtime and service configuration.

For an interview exercise, you might choose an HTTP/2-aware proxy when centralized routing is an explicit requirement. Alternatively, propose client-side selection when the client fleet can reliably distribute discovery and policy updates. State who owns configuration changes, how stale destinations are handled, and how you would test the choice. A proxy-versus-client preference becomes useful only when its operational responsibilities are visible.

Choose a policy against the workload

Round robin is a reasonable proposed baseline when backend capacity and request cost are similar. Defend that assumption with a test plan. If some requests hold a backend for much longer, compare distribution by active work and completion latency rather than counting only arrivals. An even arrival rate can coexist with an uneven backlog in the exercise.

Be precise about “least requests.” For equally weighted hosts, Envoy's documented implementation samples a configurable number of hosts, two by default, and picks the sampled host with fewer active requests. It does not scan the entire fleet. With unequal weights, Envoy instead uses weighted round robin with effective weights adjusted by active requests and a configurable bias.

Active request count is still a proxy for load. In an illustrative mixed workload, a cheap metadata lookup and a costly export each count as one request. Your proposed evaluation should include request classes, queueing, CPU pressure, and target capacity. If those classes interfere, separate capacity pools or admission rules are options to compare, not conclusions established by the algorithm's name.

Feature combinations also matter. AWS documents that ALB sticky sessions bypass the selected routing algorithm after initial target selection, WebSocket messages remain with their selected target, and least outstanding requests cannot be combined with slow start. These are ALB-specific constraints. Use them to demonstrate the habit of checking how affinity, connection lifetime, and warm-up interact in the actual product you propose.

Make health an eligibility decision

A health check should answer a specific question. Can the process make progress? Has it finished starting? Can it serve this traffic? Kubernetes distinguishes startup, liveness, and readiness probes: startup checks can delay the others, liveness failure drives restart behavior, and readiness controls traffic eligibility. Its documentation warns that poorly designed liveness checks can cause cascading failures under load.

In a practice design, ask what the readiness endpoint actually exercises. A handler returning success without touching a critical dependency offers different evidence from a bounded representative operation. Conversely, making every backend's liveness depend on a shared database could make a database incident trigger fleet-wide restarts. Specify the intended recovery action for each signal and test a correlated dependency failure.

Combine probes with observations from real traffic

Envoy outlier detection uses observed errors, timeouts, and resets as passive health signals. Ejection is configuration-dependent and bounded; panic behavior can still use ejected hosts. A successful active check can also uneject a host even while application traffic is failing. Describe the interaction between these mechanisms instead of promising that a failed target can never receive another request.

Then answer the uncomfortable follow-up: what happens when every target is unhealthy? ALB documents a fail-open behavior when all registered targets fail health checks across enabled Availability Zones: it routes to those targets despite their health status. That is a documented product behavior, not a universal rule for balancers. In your proposed system, decide whether remaining capacity, explicit rejection, or a degraded response best satisfies the service contract. Explain how clients would observe that decision.

Drain existing work within a real deadline

Removing an endpoint from selection and finishing work already assigned to it are separate responsibilities. Draw the termination sequence as carefully as the happy-path request. Identify the component that stops new assignments, the protocol that informs peers, the application behavior for existing calls, and the deadline after which the process must exit.

Stop new work without inventing stream migration

HTTP/2 provides a useful mechanism: GOAWAY lets an endpoint stop accepting new streams while existing streams can finish. A peer receiving it must not open more streams on that connection. GOAWAY does not move an active stream to another backend. Your design still needs a completion or reconnect contract for that stream.

The gRPC graceful-shutdown guide similarly describes stopping new RPC acceptance while allowing existing calls to finish. It recommends a separate forceful-shutdown timer to prevent waiting indefinitely; forceful shutdown fails in-flight RPCs. Verify the language-specific API rather than assuming every graceful-stop method has a built-in timeout.

For the exercise, propose a maximum completion window for bounded unary calls and a documented reconnect path for indefinite streams. If a client resumes using a cursor or sequence number, explain who persists progress and how repeated messages are handled. If the application cannot resume, say that the stream will be interrupted at the deadline. “Graceful” describes a shutdown attempt; your application contract determines what the user experiences.

Fit every step into the same shutdown budget

Kubernetes documents concurrent activity during Pod termination: local shutdown and control-plane endpoint updates proceed together. A preStop hook runs before the termination signal and consumes the termination grace period. Terminating endpoints are marked ready=false, while serving can convey readiness for draining. These mechanisms do not create an immediate, globally observed sequencing barrier.

Consequently, treat a fixed sleep as a mitigation to validate, not proof that every router has observed withdrawal. In a proposed test, record when termination begins, when each routing layer stops assigning new calls, and when the last existing call completes. Account for propagation, hooks, application shutdown, and any proxy or sidecar in the same end-to-end budget. Name what happens if that budget runs out.

A managed balancer adds another deadline. ALB's deregistration documentation says it stops new requests to deregistering targets and defaults to a 300-second delay; premature target connection closure can cause 5xx responses. Coordinate that setting with the application's lifetime. A long balancer delay cannot preserve a call after its backend has already exited.

Prove the decisions with observable evidence

Choose observations that could disprove your assumptions. Equal connections were misleading in the opening exercise, so measure both connections and active requests. Distinguish arrivals from pending work and successful completions. For a proposed per-backend view, include active streams, request classes, response latency, and rejected assignments. Compare the distribution during steady traffic, scaling, target failure, and rollout.

Envoy's cluster statistics provide concrete examples: active connections, active requests, pending requests, connection failures, circuit-breaker overflows, active ejections, and panic-routing signals. These documented statistics are generally cluster-scoped; they do not automatically supply the per-backend breakdown your experiment requires. Final-response counters and retry-response counters are also separate, so specify which outcome your dashboard reports.

For shutdown, propose two distinct measurements: new calls assigned after withdrawal starts and previously accepted calls interrupted before completion. The first tests routing convergence; the second tests the drain contract. Also report how long draining takes and how often forced termination occurs. A green readiness chart alone cannot demonstrate that active streams survived a deployment.

Keep the result tied to the workload. A policy that improves completion latency for short RPCs might still leave one long stream carrying most of the bytes. Ask whether that is acceptable under the stated contract before demanding uniformity. Your goal is evidence that the service's requirements hold, with enough detail to explain the exceptions.

Rehearse the follow-ups that expose missing contracts

Use the following as an illustrative practice rubric, not a hiring scorecard. Give yourself five minutes to state the workload and selection boundary, ten minutes to compare two policies, and ten minutes to trace health failure and termination. Finish with five minutes of questions from a practice partner. Those timings are a rehearsal suggestion, not a recommended interview schedule.

1. One client carries most RPCs. Show whether its calls can reach multiple targets and which connection or policy decision constrains that distribution. Identify the measurement that would reveal concentration.
2. Probes succeed while requests fail. Explain what the probe omits, which traffic signal detects the discrepancy, and how recovery avoids immediately reintroducing a broken target.
3. Every backend fails a shared check. State the actual balancer behavior, your intended degraded response, and the client-visible consequence. Do not hide the decision behind “automatic failover.”
4. A stream outlives the drain deadline. Walk through withdrawal, peer notification, application progress, and forced termination. Explain precisely what reconnecting can recover and what it cannot.

After each answer, ask your partner to identify one assumption you left implicit. Rewrite that assumption as a requirement or an experiment. A useful opening might be: “I will first choose whether we balance connections or RPCs, then define endpoint eligibility and a bounded drain contract. I will validate request concentration and interrupted calls under rollout.” Adapt it to the actual prompt rather than memorizing it.

Follow one request from discovery through selection to completion, then follow the backend through failure and exit. That habit gives you a concrete way to explain tradeoffs and identify missing guarantees. Practice until each component has a purpose, each deadline has an owner, and each claimed behavior has evidence you would collect.