The Staff Engineer Rate-Limiting Interview: Quotas, Fairness, and Overload

A client can have thousands of requests left in its hourly quota and still be throttled. GitHub documents a 5,000-request hourly limit for ordinary authenticated users alongside a secondary limit of 100 concurrent requests shared across REST and GraphQL. That distinction is the opening to a much better rate-limiter design.

For a senior or staff interview, prepare to explain what you are protecting, who receives scarce capacity, and what happens when the limiter itself fails. Stripe separately documents rate and concurrency controls, illustrating why a single requests-per-second counter cannot express the entire policy. This guide offers a practice framework, not a claim about any employer’s hiring rubric.

1. Define the contract before choosing an algorithm

Start with a hypothetical multi-tenant API offering interactive reads and expensive report exports. Ask whether the requirement is a commercial allowance, protection from abuse, a dependency budget, or a guarantee that healthy customers retain service. These objectives can require different enforcement points.

Write down the identity, unit, scope, time period, and enforcement strength. A tenant’s monthly export allowance differs from a regional burst limit. An authenticated organization identifier is useful for account policy; an IP address may be a coarse pre-authentication signal. Explain how multiple keys belonging to one organization share its allowance, so creating another key does not create another entitlement.

Keep a hard spending cap separate from a best-effort throttle. AWS warns that API Gateway usage-plan quotas can be exceeded and should not be relied on to control costs. If your hypothetical product promises a strict paid-operation allowance, propose authoritative reservations before the expensive operation starts and a defined settlement process afterward.

Describe admission as a sequence: inexpensive ingress checks, identity validation, tenant policy, then resource admission near the worker. Reject large inputs before parsing or processing them unnecessarily. OWASP’s resource-consumption guidance includes payload sizes, execution timeouts, batching, and third-party spending limits. State which layer owns each control instead of implying that the gateway handles every resource.

2. Count the resource you need to protect

A cached lookup and a full export should not automatically consume equal capacity. Request counts are easy to explain, but choose a cost unit when workload size varies materially. For the hypothetical API, you might charge one admission point for a lookup and twenty for an export; those numbers are proposed weights requiring measurement.

Shopify’s GraphQL Admin API uses calculated query cost, checks requested cost before execution, and refunds the difference between requested and actual cost afterward. Use that documented pattern to explain estimation and reconciliation. Your own implementation still needs an operation identifier and idempotent settlement so a duplicated completion message cannot refund twice.

Distinguish customer accounting from capacity protection. A failed export can consume CPU even if your product chooses not to bill for it. Define whether rejected attempts, cache hits, retries, and canceled work count in each budget. For external services, reserve capacity against the provider’s relevant account or resource scope, rather than only the caller’s tenant.

Google’s overload chapter explains why requests per second can be a poor capacity measure and discusses resource consumption such as CPU. In your proposal, collect cost estimates by operation class and compare them with actual work. Revise weights when that comparison changes; do not present a point system as permanently accurate.

3. Choose among five limiter algorithms

Explain the observable behavior before the data structure. These five choices answer different questions:

Algorithm	Behavior	Tradeoff to explain
Fixed window	Count requests in each aligned period.	Simple state; adjacent windows permit a boundary burst.
Sliding log	Keep timestamps inside a moving interval.	Precise interval accounting; state grows with retained events.
Sliding counter	Weight the previous period and add the current count.	Compact state; approximate timing distribution.
Token bucket	Refill up to a capacity; spend tokens on admission.	Separate sustained throughput from permitted bursts.
Leaky bucket	Drain accumulated work or debt at a configured rate.	Specify whether excess requests wait or are rejected.

Use examples to expose the tradeoff

Redis documents counter-based rate-limiter patterns. For a hypothetical fixed limit of 100 requests per minute, 100 arrivals immediately before a minute boundary and 100 immediately afterward can pass the two separate counters. That arithmetic explains why an aligned window does not enforce 100 requests in every rolling minute.

For a sliding log, propose unique request identifiers scored by trusted timestamps. Redis provides score-range deletion and score-range counting. Prune, count, and conditionally insert atomically; define the interval endpoints so events exactly on the boundary have consistent treatment.

Cloudflare describes a sliding-counter approximation using the previous window’s weighted count. Its approximation assumes a distribution within that previous period; discuss whether the resulting uncertainty fits the requirement.

API Gateway documents token-bucket rate and burst settings. Prefer that model when you want to articulate both explicitly. For leaky buckets, specify scheduling semantics: NGINX documents delaying excessive requests, rejecting beyond a burst threshold, and an optional nodelay setting. An algorithm name alone does not tell clients whether they will wait.

4. Calculate bursts, concurrency, and state

Use explicit hypothetical assumptions: a bucket refills at 200 points per second, holds 400 points, and charges one point for the ordinary request. Starting full, the bucket admits at most 400 + 200T points over an interval of T seconds, assuming correct single-authority accounting. In ten seconds, the envelope is 2,400 points. It permits an initial burst rather than evenly spacing every request.

Now assume average service time is 0.2 seconds at a steady 200 requests per second. The corresponding average in-flight work is approximately 200 × 0.2 = 40 requests under a stable workload. If service time rises to two seconds at that arrival rate, the corresponding demand becomes 400. The rate policy has not changed, but resource occupancy has.

Propose a separate concurrency ceiling and a bounded waiting policy. Envoy documents separate limits for pending requests and outstanding requests. Size these against the worker pool and dependency capacity. Explain how timeouts and cancellation release permits, and why a slow dependency should trigger fewer admissions before every worker becomes occupied.

Estimate limiter state too. In another hypothetical configuration, one million active identities at 128 bytes of application state require 128 MB before datastore, indexing, replication, and allocator overhead. A sliding log retaining 100 events per identity changes that calculation substantially. Use expiry for inactive identities and bound who can create state. Request a load test to validate both memory and decision latency.

5. Allocate local and global budgets

Draw the path: client, gateway, local admission, tenant budget service, application, dependency. Envoy’s HTTP local rate limiter applies per process by default. Therefore, ten independent processes each configured for 100 requests per second do not establish a shared 100-request allowance; their combined configured refill is 1,000.

A central service can make tenant decisions against shared state, but every synchronous check adds a dependency to admission. Envoy describes combining a coarse local token bucket with fine-grained global limiting to protect the global limiter from large bursts. Explain both budgets and their ordering.

Make budget distribution explicit

For the hypothetical design, compare per-request coordination with short-lived allocations to regions or gateways. Allocations reduce coordination frequency, but can strand unused capacity and complicate recovery. Track issued, consumed, expired, and remaining allocations. Specify whether a gateway may continue spending a valid allocation during a partition.

If the global allowance is strict, the authority must count all outstanding allocations before issuing more. Lost allocation state must not authorize duplicate spending. If the allowance is approximate, state the tolerated excess and bound it using allocation size, node count, and validity period. Autoscaling should redistribute a fixed aggregate budget rather than accidentally multiply it.

Make that accounting concrete. In a hypothetical allowance of 1,000 operations, two regions each receiving 300 reserved operations leave 400 available for new allocations, even before the regions report consumption. Reissuing an apparently unused reservation after a timeout could let both holders spend it. Define expiry and ownership so the authority can safely return capacity before assigning it elsewhere. Spenders must enforce that expiry; a partitioned worker cannot retain permission indefinitely.

6. Specify distributed failure behavior

Atomicity answers one question; durability answers another. Redis guarantees atomic script execution and blocks other server activity while a script runs. Keep limiter scripts small. A single atomic decision avoids competing callers both spending the last token, but does not by itself prove that the debit survives every failover.

Redis documents asynchronous replication and a window for data loss. Decide whether a lost recent debit is acceptable for abuse throttling. For a strict purchased allowance, explain what stronger authoritative accounting or reservation record your design requires instead of assuming that a replicated cache provides that guarantee.

Handle ambiguous timeouts: the budget service may have committed the reservation before the caller lost its response. In the hypothetical reservation design, retry with the same operation identifier and return the recorded outcome. Repeated delivery must not debit twice, and a timeout must not automatically refund work that might still be running.

Also decide what happens when a request passes one budget but fails another. For this design, check inexpensive rejection conditions first, then reserve the required resources with a defined cancellation path. Distinguish releasing an unused reservation from refunding work already performed. If independent stores prevent an atomic multi-budget reservation, describe the temporary holds and reconciliation process, including who repairs an abandoned hold.

Choose the fallback by operation. A public cached read might continue through a conservative local limit for a short period. A paid export might stop when it cannot reserve allowance. Envoy exposes failure_mode_deny and records requests allowed during limiter errors. Explain your chosen policy, its duration, and how operators can distinguish limiter failure from customer overuse.

Finally, specify clock ownership. Use monotonic elapsed time for local refill calculations and a consistent authority for shared windows. Define behavior after restart and clock changes; never grant unlimited refill because an unchecked time delta is unexpectedly large.

7. Make fairness an admission policy

Per-tenant ceilings prevent one customer from exceeding its allowance, but they do not allocate contested worker time. In the hypothetical API, one export tenant could occupy the available workers before interactive customers arrive. Define fairness in terms of the protected experience: a minimum opportunity to progress, bounded waiting, or a weighted share of processing capacity.

Kubernetes API Priority and Fairness classifies requests and uses fair queuing to reduce starvation between flows. Borrow the principle rather than copying cluster configuration blindly. Propose separate bounded queues for interactive and batch traffic, tenant-aware dispatch, and explicit concurrency shares when both classes are busy.

For a hypothetical 100-worker pool, reserve 60 slots for interactive work and 40 for exports, allowing borrowing only under a stated policy. Explain whether borrowed capacity can be reclaimed promptly and how you stop continuous interactive demand from starving exports. Avoid an unbounded queue advertised as a fairness solution.

Priority should come from trusted product policy. Google describes request criticality and rejecting lower-criticality work earlier during overload. In your API, clients should not grant themselves the highest priority through an arbitrary header. Distinguish urgent user actions from background refreshes, and measure completion and waiting by class to test whether your proposed fairness actually holds.

8. Give clients a useful rejection contract

RFC 6585 defines 429 Too Many Requests, recommends explaining the condition, and permits Retry-After. It does not define your identity model or counting algorithm. In the hypothetical API, return a stable reason code, the exhausted policy scope, and safe retry guidance. Do not expose internal customer identifiers or sensitive utilization details.

RFC 9110 permits Retry-After as an HTTP date or an integer delay in seconds. Choose a representation your clients can interpret reliably. A token refill time is an estimate of when allowance becomes available, not a reservation guaranteeing success at that instant. Concurrent clients may compete for the recovered capacity.

Differentiate tenant exhaustion from server-wide unavailability and limiter errors in the response contract. Give callers enough information to reduce the relevant work. Lowering concurrency may help an occupancy limit; waiting for allowance helps a quota. Repeating an oversized request unchanged does not solve its cost problem.

Recommend bounded retries with deadlines, randomized backoff, and one clearly designated retry owner. AWS SDK documentation describes exponential backoff with jitter and a separate retry token budget. Account for library retries before adding another loop. For mutating work, preserve the operation identity so a retry cannot create a second effect. Practice explaining what the caller does after its retry budget is exhausted.

9. Protect useful work during overload

A customer can follow every published quota while the service loses half its capacity. Keep resource admission near the protected workers, using signals appropriate to the bottleneck. Possible inputs in your proposed design include active work, queue age, memory pressure, and dependency saturation. Commercial entitlement cannot guarantee that a failing dependency has capacity.

Google’s cascading-failure guidance explains that queued requests consume memory and increase latency, and recommends load shedding before resource exhaustion. Reject before expensive processing where possible. Preserve useful completions by reducing optional work: a report request might become a scheduled job, while an interactive response omits an optional enrichment.

For the hypothetical export queue, set an age ceiling and a maximum backlog consistent with the promised completion time. When accepted work exceeds what can finish within that contract, stop admitting more and expose the resulting delay. Persisting jobs can make deferred execution possible, but does not create processing capacity.

Watch admitted work, useful completions, latency by class, rejected cost, limiter errors, and time spent in fallback. A falling average latency can reflect rejection of slow requests rather than improved service. Compare completion rates and customer impact alongside response times. Define recovery criteria and restore capacity gradually so every waiting client does not rush back together.

10. Practice the decisions under pressure

Rehearse a 45-minute design exercise with the hypothetical API. Spend five minutes defining identity and guarantees, ten on workload and arithmetic, ten on the admission path, ten on failures and fairness, and ten on client behavior and validation. These are suggested practice intervals, not a prediction of an employer’s interview schedule.

Ask a partner or practice assistant to introduce three changes: a dependency becomes ten times slower, a tenant spreads traffic across regions, and the budget service times out after possibly accepting a reservation. Respond by updating the relevant assumption and guarantee. Keep the original product objective visible as you adjust the design.

Propose measurement before enforcement. NGINX provides a dry-run mode that accounts for excessive requests without limiting them. Then test boundary bursts, concurrent debits, restart behavior, and recovery after overload. Google recommends testing beyond failure and comparing gradual with sudden load increases.

The strongest practice answer connects allowance to capacity and capacity to customer experience. End with a guarantee you can defend, the measurements that would challenge it, and the next experiment needed to validate it. That gives your interviewer a concrete design to discuss.