The Staff Engineer Recommendation Interview: Retrieval, Ranking, and Feedback Loops

A recommendation system learns from choices its own ranking helped create. If the first slot gets a click, did the user prefer that item, or did your service make it easiest to notice? Research on selection bias in recommendation and position bias in ranking explains why that distinction matters.

For a senior or staff engineer mock interview, make the feedback arrow as concrete as the serving path. This guide develops an illustrative learning-video feed: users select topics, videos become eligible after publication checks, and viewing produces delayed feedback. The sizing assumptions and practice rubric are editorial exercises, not disclosed employer questions or scoring criteria.

1. Separate the product goal, retrieval, and ranking

Start with a product promise: help the learner find an eligible video worth watching next. For this exercise, choose completed viewing and explicit usefulness feedback as outcomes, with latency, repetition, and hide rates as guardrails. Specify what counts as completion for different video lengths and when a label becomes final. Do not let “maximize engagement” substitute for a measurable definition.

Published industrial designs support separating candidate generation from ranking. Covington, Adams, and Sargin's 2016 YouTube paper describes those two stages. A 2019 YouTube ranking paper addresses competing objectives and selection biases. These are documented historical systems, not evidence of YouTube's exact architecture today or a reason to copy their objective into every product.

Give each stage an explicit contract

In a two-tower retriever, one model represents the user and request context; another represents the item. Their vector similarity supplies an affinity score. TensorFlow Recommenders' retrieval tutorial shows the separate representations and an item index for serving. Item representations can be indexed ahead of requests; request-specific user-item interactions belong in the later ranker when they cannot be precomputed independently.

Propose these contracts for the practice design:

• Retrieval: accept user context and an index version; return a bounded set of item IDs, candidate-source tags, and retrieval scores.
• Ranking: accept those candidates and compatible features; return an ordered list with the model version and defined prediction targets.
• Final serving: enforce current eligibility, deduplicate, apply the chosen repetition rules, and attach an exposure identifier to each returned item.

Keep two measurements separate. Candidate hit rate asks whether a held-out positive survived retrieval. Approximate-neighbor recall asks whether the index recovered the exact vector neighbors. TensorFlow's efficient-serving tutorial compares held-out top-K accuracy under brute-force and ScaNN serving, showing how approximation affects that metric. A perfect nearest-neighbor index can faithfully return a poor model's preferences; a good ranker has no opportunity to score a relevant item excluded upstream.

Cold start and eligibility are different problems

An ID-only collaborative model lacks a learned embedding for an unseen item, as Google's collaborative-filtering guidance explains. Content-based methods use item features and existing user interests, but have limited ability to expand those interests, according to Google's content-based guidance. For our exercise, route a new learner through selected topics and an eligible popularity baseline; give a new video a metadata-based candidate path and a bounded exploration allocation.

None of those paths makes an unpublished, removed, or blocked video eligible. Google's re-ranking guidance includes removing candidates with filters. Apply coarse eligibility during retrieval where practical, then recheck authoritative state before serving. Define a smaller result set when too few candidates survive, rather than backfilling from unrestricted items.

2. Log exposure before interpreting clicks

Click records alone cannot reconstruct the alternatives a user encountered. Schnabel and colleagues' 2016 recommendation study treats observations as selected by users and the recommendation system, rather than randomly missing. For this design, an unshown video has an unknown response, not an observed rejection. Sampling it as a training negative is a modeling choice that you should state and evaluate.

Define the observation boundary

Use distinct events for the server's decision, client rendering, and a viewport-qualified impression. Define the viewport rule in the exercise instead of pretending every returned item was seen. Join later clicks, completions, hides, and usefulness responses to the exposure ID. Include item position, request time, candidate source, experiment assignment, serving mode, and model and feature versions. Record the selection probability when the system intentionally randomizes an eligible choice.

Make event IDs stable across retries and define a deduplication window. Track missing client acknowledgments separately from negative outcomes. If a completion label can arrive tomorrow, retain the exposure now and finalize the training label after the agreed observation window. Keep partial outcomes distinguishable from a completed window with no action.

Distinguish position bias from selection bias

Position bias concerns how placement affects observation and clicks. Selection bias concerns which items and interactions enter the observed dataset. Joachims, Swaminathan, and Schnabel's 2017 learning-to-rank paper uses propensity weighting to address biased implicit feedback with an appropriate observation model. Merely adding a position column does not establish that your labels measure preference independently of exposure. Keep the logged probability of selecting an item distinct from the probability of examining its position; choose the estimator and event definition together.

Propensity correction also has limits. Recommendations as Treatments assumes nonzero observation probabilities and analyzes estimator variability and propensity misspecification. You cannot recover evidence for items that had no chance of observation by assigning them large weights. In a mock design, constrain exploration to eligible items, explain the probability being logged, and name the bias and variance assumptions before proposing counterfactual evaluation.

3. Preserve time and the policy that generated feedback

A per-user last-interaction split is not automatically a faithful historical replay. Another user's later interaction may still enter training before an earlier test request. In A Critical Study on Data Leakage in Recommender System Offline Evaluation, Ji and colleagues demonstrate how ignoring the global timeline leaks future interactions and unavailable items into evaluation, affecting comparative results in their experiments.

Reconstruct what the service could know

Choose a global training cutoff, a later validation period, and a later untouched test period. For each historical request, reconstruct the eligible catalog, user history, item features, and model state available then. Apply the same label window consistently. An item published after the request must not compete in its candidate set, and next week's completion must not become an input feature for today's prediction.

Feast's point-in-time joins retrieve feature values relative to historical entity timestamps. Its current documentation makes an important distinction: event-time constraints alone can include a value backfilled later. Availability-time filtering needs a created timestamp that actually represents when the feature became available, plus support from the chosen offline store. In your design, preserve both event time and availability time; a timestamp describing when something happened does not prove the service knew it then.

Use offline metrics as evidence with limits

For the practice system, inspect candidate hit rate, final ranking quality, and outcomes separately. Slice results by new users, new items, topic, and serving mode. Keep the evaluated candidate pool and relevance-label definition visible when comparing models. Add latency and eligibility violations to the release decision; a better ranking score is insufficient if the request path breaks its product contract.

Feedback-loop effects need a different boundary. Chaney, Stewart, and Engelhardt's simulation study shows that training on algorithmically confounded behavior can increase homogeneity without increasing utility in the simulated settings. It does not establish that every production recommender has the same effect or magnitude. Use it to motivate measuring repeated exposure, catalog coverage, and changes across successive retraining cycles.

Propose a staged experiment with an unchanged comparison group, predefined outcomes, and guardrails before rollout. Examine whether a short-term improvement persists across the chosen observation window. Preserve experiment assignment in the logs so a later training job can distinguish which policy produced its feedback. Treat the experiment plan as a hypothesis test for this design, not a promised business outcome.

4. Version features and keep the fallback affordable

Feature infrastructure belongs in the architecture diagram. Google's Rules of Machine Learning identifies training-serving skew caused by differing pipelines, changing data, and feedback loops; it recommends logging serving features and measuring consistency. For the mock system, give every feature a definition, transformation version, default, event timestamp, availability timestamp, and maximum tolerated age.

Make compatible versions a serving invariant

Represent a release as a manifest binding the user tower, item tower, embedding index, ranker, and feature schema. Reject incompatible combinations before they receive traffic. Build a replacement item index using the matching item model, validate it with the matching user model, and switch the compatible bundle together. This is a proposed deployment contract for our design, not an automatic guarantee supplied by a vector database.

Define freshness separately for each feature. A topic preference may tolerate an older value in this exercise; a removed-item flag must respect the product's removal deadline. Measure source-to-serving lag as well as the age attached to a returned feature. Specify what happens when a feature is missing, stale, or unavailable, and train and evaluate the ranker under those fallback conditions. Verify rollback restores a compatible bundle and still applies current eligibility.

Size work under stated assumptions

Suppose the exercise has two million videos with 128-dimensional float32 item vectors. Raw vector payload is 2,000,000 × 128 × 4 bytes: 1.024 billion bytes, about 0.95 GiB. Index structures, metadata, replicas, and headroom are additional. At an assumed 400 requests per second and 300 ranked candidates per request, the ranker receives 120,000 user-item evaluations per second. These are illustrative arithmetic inputs, not vendor benchmarks or measured production requirements.

Allocate one request deadline across feature fetches, retrieval, ranking, and final eligibility checks. Bound candidate count, concurrent feature calls, and queued inference work. Load-test both steady traffic and bursts using the chosen feature distributions. Measure end-to-end latency and degraded-path frequency rather than treating successful model inference as a successful feed response.

Google SRE's overload guidance describes cheaper degraded responses, resource-based protection, and limits on retries. Apply that principle by skipping optional ranking work or using a precomputed topic list when the expensive path exhausts its budget. Recheck eligibility on the fallback, cancel abandoned work, and keep retry budgets bounded. Exercise the fallback under overload; its feature dependencies and safety checks need capacity too.

If current eligibility cannot be established within its freshness contract, return fewer items or an explicit unavailable state. Record the fallback as its own serving policy in exposure logs. That lets the next evaluation distinguish outcomes produced by the normal ranker from outcomes produced during an incident.

5. Rehearse one request and one misleading metric

Use this editorial rubric in a 45-minute practice session. Spend five minutes defining the learner's goal and outcome window, ten specifying retrieval and ranking contracts, ten tracing exposure and delayed labels, ten defending time-correct evaluation, and ten handling a version mismatch or overloaded ranker. Adapt the timing to the mock; it is not an employer scoring model.

Ask a practice partner to introduce one failure at a time:

• A new upload never appears. Identify its eligibility state, candidate route, and the missing interaction or feature dependency.
• Clicks rise after moving an item upward. Explain the exposure change before inferring stronger preference.
• Offline results look unusually good. Audit global cutoffs, catalog availability, backfilled features, and outcome windows.
• The user tower changes alone. Point to the compatibility check that prevents querying the old embedding space.
• Ranking times out. Show the affordable fallback, current eligibility check, cancellation, and separate exposure policy.

For each response, name the invariant, the evidence that would reveal a violation, and the next decision. Finish by following one item from publication through retrieval, ranking, impression, feedback, and retraining. Your goal is an explanation in which a changed score, missing log, or delayed feature has an observable consequence.

The useful next step is to practice that trace aloud. Keep asking what the learner could see, what the system could know, and which policy produced the evidence. A recommendation design becomes defensible when every feedback arrow has the same clarity as every serving dependency.