The Staff Engineer Sharding Interview: Keys, Hotspots, and Safe Resharding

Your order service has plenty of spare database capacity, yet one large merchant cannot create orders reliably. Adding a shard helps other merchants and leaves this merchant throttled. A staff engineer sharding interview becomes interesting at this point: what determines where a request goes, and which unit of work can actually move?

A useful answer connects access patterns, key distribution, routing, and write authority. This guide uses primary documentation to examine those connections. The order service, workload figures, architecture proposals, and practice rubric below are illustrative exercises, not vendor benchmarks or a company's hiring criteria.

1. Start with requests and invariants

Before drawing shards, ask what the application must do. For an illustrative order service, list four requests: create an order, fetch it by merchant and order IDs, list a merchant's recent orders, and update an order with its line items. Separate operational requests from global reporting. Ask which writes must commit together, how long history stays online, and whether one merchant can dominate traffic.

Define correctness alongside throughput. An acknowledged order must remain retrievable. A line item must reference the intended order. During a move, the system must not let independently writable copies accept conflicting changes without a defined reconciliation rule. These requirements constrain placement and migration before a database product enters the discussion.

Clarify what “partition” means in your design. PostgreSQL's table-partitioning documentation describes dividing a logical table into physical pieces, with pruning and maintenance benefits. That feature alone does not specify a deployment across independent database servers. In this exercise, sharding distributes data ownership across database servers; each server may also partition its own tables.

2. Choose the key with a query beside it

Evaluate candidate keys against the requests you just named. MongoDB's shard-key guidance treats cardinality, frequency, monotonicity, and query patterns as separate considerations. A million distinct merchant IDs offer high cardinality, but that says little about a workload where one merchant generates a large share of writes.

Explain the tradeoff instead of declaring a winner. Merchant placement favors local reads and related writes. Order placement spreads independent orders but can scatter a merchant's history. A bucketed merchant key introduces another routing dimension and more read work. If line items use a different key from their order, explain how atomic updates or recovery work across the resulting boundary.

Hashing also changes locality. MongoDB's hashed-sharding documentation explains the distribution benefit for monotonically increasing values and the reduced targeting available for range queries. Hashing increasing order IDs can disperse new orders. Hashing the same merchant ID repeatedly still produces the same hash; it does not subdivide that merchant's traffic.

3. Diagnose the hotspot before adding capacity

Distinguish storage skew, request skew, and contention on one record. A shard holding old orders may be large but quiet. A small set of active orders may be expensive. One inventory counter may serialize updates even when every surrounding order is distributed. Ask for per-key and per-shard request rates, throttling, latency, and resource utilization; aggregate averages can hide the unit causing trouble.

For a merchant with many independent orders, propose calculated write buckets. AWS's write-sharding guidance describes random or calculated suffixes and the need to query all suffixes when reading a complete logical group. In this exercise, a stable hash of order ID selects a bucket. A lookup with merchant and order IDs can calculate its destination; a merchant-history query must combine bucket results.

Do not confuse a logical key with a permanently fixed physical partition. DynamoDB's adaptive-capacity documentation describes redistribution within table and partition limits, including isolation of frequently accessed items. Managed redistribution helps, but it does not make one record's throughput unlimited. If that record remains the bottleneck, discuss changing its update semantics, batching, or a different representation, and identify the consistency consequences.

4. Follow the request through indexes and fan-out

After balancing the base data, introduce a realistic follow-up: the operations dashboard needs every pending order. An index keyed only by status may concentrate writes again. AWS documents global-secondary-index back pressure, including hot index keys that can throttle base-table writes. “The table is evenly distributed” therefore does not finish the write-path analysis.

For the exercise, consider a status-and-bucket index or a separately maintained operational view. Then account for the new costs: multiple reads, merging, freshness, and recovery when the derived view falls behind. Keep ordering deterministic across buckets, bound the number of requests, and state what the endpoint returns when one shard times out. A global listing is a product contract, not simply an extra arrow on the diagram.

5. Separate logical placement from physical ownership

A proposed custom design can map each order to a stable logical bucket, then map that bucket to its current database owner. Moving a bucket changes the ownership directory without changing the order's identity. Define who publishes directory versions, how routers cache them, and how a database handles a request sent using stale ownership information.

The Redis Cluster specification provides a concrete example of this separation: keys map to hash slots, and slots have node owners. Its MOVED response communicates routing changes, while ASK handles a redirected operation during migration. Those are Redis-specific protocols, but they expose a general interview question: what must clients do while placement is changing?

For the custom order service, propose an ownership epoch that an old destination cannot continue serving as an independent writer after cutover. Specify how the old writer is fenced, not merely how routers learn the new address. Also explain the locality boundary for related keys. Redis hash tags colocate keys, yet its specification describes temporary TRYAGAIN failures for some multi-key operations during resharding. Placement and uninterrupted transaction behavior require separate reasoning.

6. Copy a live bucket without losing its history

Describe a move as a transfer of write authority with an evidence trail. Prepare target capacity, schema, indexes, and routing metadata first. Keep the source authoritative while copying and applying concurrent changes. Set a migration throughput budget so backfill does not consume the resources needed for foreground requests.

A raw scan followed by “start replication” leaves an unexplained interval. An order copied early may change before the scan ends. Vitess's account of a replication stream explains how snapshot chunks, binlog positions, catch-up, and fast-forward phases coordinate copying with ongoing writes. Use that as a concrete mechanism to study; name the equivalent snapshot and log boundary in your chosen system.

Trace one deletion through the move

Use an illustrative failure sequence: the copier reads an order, the source deletes it, and the delayed copy arrives at the target after the delete. Ask what prevents resurrection. A coordinated snapshot-and-log protocol may establish the necessary ordering. In a custom asynchronous copier, durable deletion markers and version checks are possible tools, but their retention must cover replay and lag. This is a design exercise, not a claim that Vitess needs that custom protocol.

Make restart behavior explicit. Persist progress against a defined range and source position. Reprocessing a chunk must not overwrite newer target state. Record which migration owns the target copy so a restarted worker cannot quietly resume an obsolete move. These are properties to demonstrate with a trace, rather than assuming that a retry flag supplies them.

7. Validate, switch authority, and preserve a real rollback

Matching total row counts is weak evidence: one missing order and one extra order can cancel out. Vitess VDiff compares source and target rows and reports missing, extra, and unmatched records. In the exercise, combine comparable-boundary data checks with routing tests and application invariants. A completed copy percentage alone does not prove that the target can serve correct orders.

Separate read migration, write cutover, and cleanup. The Vitess resharding guide demonstrates validating targets and switching traffic in distinct steps. State whether an early target read may be stale and which requests still need the authoritative source. Exercise the new routing path before retiring the old one.

Be precise about availability. The Vitess Reshard reference says write cutover temporarily makes source databases read-only while targets catch up; application writes can pause or fail. Describe the client's timeout, retry, and duplicate-prevention behavior for that interval. Calling the copy online does not establish a zero-pause write transition.

Finally, account for writes accepted by the new owner. Keeping an old database is insufficient if it stops receiving those changes. The same reference documents reverse replication, enabled by default, and the loss of post-cutover rollback when it is disabled. Define a rollback window and evidence for reversal. Treat workflow completion and source-data removal as later decisions, after the team accepts that recovery boundary.

8. Practice decisions that survive changing assumptions

Use this illustrative 40-minute rehearsal: spend eight minutes on requests and invariants, eight comparing keys, eight investigating merchant and index hotspots, ten moving one bucket, and six explaining cutover and rollback. The timing is a practice aid. Adjust it to the interview format rather than treating it as an employer's scoring system.

Have a partner introduce these follow-ups one at a time:

• One merchant doubles its traffic. Show which independent orders can move and which individual records still contend.
• Pending-order writes throttle. Inspect the secondary index before adding another base-table shard.
• A router retains the old directory. Trace its request to a redirect, rejection, or safely authorized owner.
• A copied order is deleted. Identify the ordering rule that keeps it deleted after catch-up and replay.
• Rollback is requested after new writes. Show how the old owner obtains those writes before it resumes authority.

An illustrative practice rubric

Score each follow-up from zero to two: zero if you cannot state the governing invariant, one if you name a mechanism, and two if you explain evidence that it works plus a failure response. Use the scores to choose the next rehearsal, not to predict an interview result. A clear answer keeps one order visible as its key determines placement, its writes reach indexes, and its ownership changes. That continuity makes every component in the design accountable to an actual request.