The AI Systems Design Interview: How Senior Engineers Prove Reliability Beyond the Demo

“Design an AI assistant for customer support” sounds like a system-design prompt. At senior and staff level, the difficult part is not drawing a box labeled LLM. It is explaining how the team will know the assistant works for the real task, detect when it stops working, and limit the consequences when it is wrong.

That distinction has become a useful interview signal. AI adoption is broad, but confidence in output remains mixed: Stack Overflow’s 2025 developer survey found that 46% of respondents distrust AI-tool accuracy, versus 33% who trust it. The survey is self-reported and not a measure of production defect rates, but it captures the gap between availability and dependable use. A senior engineer should be able to close that gap with an observable product and a decision process.

This guide gives you a repeatable way to answer AI system-design prompts. It focuses on evaluation, product quality, failure handling, cost, and rollout. The goal is not to recite model terminology. It is to make your reasoning testable.

Why AI system design has become a staff-level interview

Traditional service design begins with a contract: given this input, return a result that meets a defined schema and latency objective. Generative systems add variability. A prompt change, model upgrade, retrieval change, or tool permission can alter behavior even when the API remains unchanged. A design that has no behavioral test plan therefore has no reliable way to distinguish an improvement from a regression.

Public research helps explain why this matters. Stanford’s HELM project argues for evaluating models across scenarios and multiple dimensions rather than treating one accuracy number as a complete description. NIST’s AI Risk Management Framework similarly organizes lifecycle work into govern, map, measure, and manage. Those are useful interview verbs: establish accountability, describe context, measure performance, and respond to findings.

The measurement question is active, not theoretical. In August 2026, NIST requested comment on its TEVV-Athlon framework, a structured approach to testing, evaluating, verifying, and validating AI systems across use cases including language models and agentic systems. You do not need to cite a draft framework from memory in an interview. You do need to show that evaluation is an ongoing engineering capability, not a one-time launch checkbox.

Start with the user task, not the model

Clarify the task before proposing architecture. “Customer support assistant” may mean retrieving an approved answer, drafting a reply for an agent, changing an account, or resolving a case without human review. Those tasks have different error costs and permissions. Ask who the user is, what decision or action the system supports, what counts as success, and what must remain under human control.

Then draw the boundary between the model and deterministic software. A model can classify intent, summarize a conversation, or draft language. Authorization, account ownership, payment changes, retention rules, and irreversible operations should be enforced by ordinary services. For a tool-using agent, specify allowed tools, input validation, output validation, per-action permissions, and a confirmation boundary. OWASP’s 2025 LLM risks include prompt injection, sensitive information disclosure, improper output handling, excessive agency, and misinformation; these are architectural concerns, not prompt-polishing tasks.

Describe the current baseline. Perhaps agents already search a knowledge base and spend six minutes finding an answer. Measure the task as it exists: resolution time, re-open rate, escalation rate, customer satisfaction, and policy violations. A model can improve one dimension while harming another. If a proposal promises fewer agent minutes, ask whether it shifts work to review, support, or incident response.

Set scope with a thin vertical slice. Pick one bounded queue, language, or case category with enough volume and a clear fallback. State exclusions plainly: no account changes, no legal or safety advice, or no autonomous response when retrieved evidence is absent. NIST’s Generative AI Profile recommends context-specific risk identification across the lifecycle; in an interview, translate that into an explicit initial operating envelope and a process for revising it.

Define quality as a set of measurable outcomes

“The answers should be accurate” is not an operational requirement. Convert quality into observable dimensions tied to the user task. For a grounded support response, useful dimensions might include factual correctness against policy, citation or source support, completeness, correct escalation, tone, and whether the response actually resolves the case. For an agent that takes action, include task completion, valid tool use, and whether it respected authorization and confirmation requirements.

Separate hard constraints from optimization targets. A policy violation or unauthorized account change may be a release blocker even if average answer quality is high. Latency and cost may be service objectives; user satisfaction may be a product outcome; correctness on a high-risk slice may be a safety threshold. The design should say which failures are unacceptable and who can approve a change to that threshold.

Use several measures because every proxy has blind spots. Exact-match works for structured outputs but misses equivalent wording. A model-based grader can score nuanced answers, but may prefer fluent unsupported text. Human review provides domain judgment, but costs time and can vary by reviewer. HELM’s evaluation approach emphasizes multiple scenarios and metrics; Anthropic’s statistical treatment of model evaluations recommends reporting uncertainty and paired comparisons rather than presenting a bare score as decisive.

Finally, connect product outcomes to engineering signals. Track time to first useful answer, successful resolution without re-open, escalation quality, refusal appropriateness, and cost per resolved task. Pair them with latency percentiles, retrieval coverage, tool errors, and fallback frequency. DORA’s 2025 AI research reports associations between AI adoption, delivery throughput, and stability; system quality should be judged by what users and the delivery system experience, not code volume alone.

Build an evaluation set that resembles production

A benchmark is only useful for the task distribution it represents. Start with real, permissioned examples from support tickets, user research, or domain-approved scenarios. Remove secrets and personal data, label the desired outcome, preserve relevant context, and record why a case is difficult. Do not put sensitive production conversations into an external model or evaluation vendor without an approved data-handling basis.

Stratify the set. Include common cases, long-tail intents, ambiguous requests, missing context, conflicting sources, adversarial wording, accessibility needs, and known high-risk situations. Test the languages, product versions, and user groups that will actually use the feature. Keep a held-out set that engineers do not tune against, and version both data and grading instructions so a score can be reproduced.

Mix historical examples with newly authored edge cases. Historical logs reveal what customers actually ask, but they encode existing coverage gaps and may contain repeated or outdated patterns. Purpose-built cases probe known risks, but can become artificial if they only reflect what the design team imagined. The SWE-bench paper illustrates the value and difficulty of testing systems on real repository issues: realistic tasks are richer than toy questions, yet dataset and environment details still shape what a benchmark measures.

Protect against leakage. Keep evaluation answers, hidden test cases, and prompt templates out of the model’s retrieval context. Separate development, calibration, and final test data. If an example gets added because a model failed it, label that provenance; repeated tuning on a small fixed set can make the system appear to improve while generalization stays flat. Stanford CRFM’s HELM publishes prompts and results for reproducibility, while its documentation emphasizes standardized scenarios and transparent comparisons.

Choose graders you can trust

Use deterministic checks wherever the desired behavior is deterministic. A schema validator can verify required fields; a permission test can reject unauthorized tool calls; a citation checker can ensure every quoted policy reference exists; unit tests can confirm the system does not execute a payment action without confirmation. These checks are fast and repeatable, so run them in continuous integration.

For open-ended quality, combine several forms of evidence. Have a trained reviewer score a sample against a written rubric. Use a model grader for broad coverage only after comparing its judgments with human labels. Give the grader explicit criteria, calibrated examples, and a “not enough evidence” option. Audit agreement by category and inspect disagreements instead of averaging them away.

Anthropic’s guidance on agent evaluations distinguishes simple single-turn grading from multi-turn tasks where the system calls tools and changes state. Match the harness to the behavior being evaluated: if the production agent searches, retries, and updates a case, a benchmark that only grades a final sentence misses the operational path. OpenAI’s Evals API similarly separates evaluation definitions, data sources, and graders so runs can be repeated across models and configurations.

Measure grader reliability. Ask multiple reviewers to score an overlapping sample; report agreement and adjudicate ambiguous rubric items. For model graders, test sensitivity to answer order, verbosity, and irrelevant phrasing. Anthropic’s statistical evaluation work recommends confidence intervals and paired differences when comparing systems on the same questions. This matters because a two-point score change on a small, noisy set may not justify a model migration.

Put evaluation into the system architecture

Sketch a request path with explicit observability: client, API and policy layer, retrieval or tools, model provider, output checks, and human or deterministic fallback. Show where you record a trace identifier, model and prompt version, retrieval references, tool calls, latency, token use, validation results, and the final disposition. Avoid storing raw sensitive prompts by default; define redaction, access control, retention, and consent or notice requirements.

Keep an offline evaluation harness separate from production traffic. It should replay a versioned dataset against candidate configurations and produce comparable reports. A shadow path can compare a candidate without showing its output to users, if data use and provider terms allow it. Online experiments can measure user outcomes, but require assignment discipline, guardrails, and a rollback plan. Do not treat a favorable offline score as proof of production safety.

Use staged rollout. Begin with internal users or a small eligible cohort. Route low-confidence, out-of-scope, or high-impact cases to a person. Define stop conditions before launch: policy violations, unsupported claims, a rise in re-opened cases, unacceptable latency, or cost per successful task above budget. A rollback may mean disabling generation, reverting a prompt, switching to retrieval-only behavior, or turning off a tool permission; name the switch and the owner.

Evaluation should follow the release lifecycle. Run fast structural checks for each change, a broader offline suite for significant changes, and post-launch monitoring for drift and user harm. NIST’s TEVV-Athlon draft treats evaluation as adaptable to application context; your architecture should make that adaptation possible without building a one-off pipeline every time a product team changes a prompt.

Make failures and security part of the design

List realistic failure modes before talking about mitigations. The model may invent policy, retrieval may return stale or irrelevant material, a prompt injection may try to override instructions, tool output may contain hostile text, the provider may time out, or a user may ask for an action outside their permissions. For each one, say how it is detected, what the system does next, and how the team learns about it.

Prompt injection deserves a boundary-based answer. A prompt can instruct a model to ignore malicious text, but it cannot replace authorization. Treat retrieved documents and user content as untrusted; keep secrets outside prompts; restrict tool scopes; validate every action in a trusted service; and require confirmation for consequential operations. OWASP’s LLM guidance describes prompt injection and excessive agency as distinct risks. NIST’s adversarial machine-learning taxonomy gives a broader vocabulary for attacks and mitigations across the ML lifecycle.

Handle unsupported answers explicitly. If retrieval finds no adequate evidence, ask a clarifying question, state the limitation, or escalate. Do not make a low confidence score the only gate: confidence can be miscalibrated, and a plausible answer may still be wrong. Evaluate abstention itself, including how often the model refuses a task it could have completed.

Privacy and security requirements belong in the first design pass. The NIST Generative AI Profile identifies data privacy, information security, confabulation, and intellectual property among risks to manage. Apply existing software controls too: NIST’s Secure Software Development Framework recommends integrating security practices throughout development, not appending them at release time. A senior candidate should say which data flows to a provider, which is retained, and which actions are impossible by construction.

Use evidence to make the model and cost tradeoff

Do not start by naming a provider or selecting the largest model. Compare the simplest viable design first: deterministic workflow, retrieval over approved content, a model-assisted draft, or an agent with tools. Choose based on the task, quality bar, latency, privacy, operational complexity, and cost. Stanford’s AI Index documents continuing changes in model capability and inference costs; that makes a benchmark snapshot a poor substitute for testing the exact candidate models on your task.

Estimate cost per successful task rather than cost per call. Include retries, long context, embedding and retrieval infrastructure, reviewer time, failed tasks, and escalations. If one configuration costs more but resolves cases reliably, it may be cheaper end to end. Conversely, a small model may be sufficient for routing while a larger model is only used for difficult cases. State the routing rule and how you will detect that it is sending the wrong work to the expensive path.

Be careful with productivity claims. GitHub’s controlled Copilot experiment reported that participants completed a bounded HTTP-server task 55% faster, while METR’s randomized study of experienced open-source developers working in familiar repositories found that the tested early-2025 tools increased completion time by 19% on average. METR later said its early-2025 estimate may not reflect newer tools and that its new experiment design was harder to interpret. These studies examine different tasks, users, and tool periods; together they make the interview lesson clear: measure your own workflow and avoid turning one result into a universal forecast.

Use a decision table to make the choice legible. For each option, compare expected task quality, worst-case failure, p95 latency, cost per success, reversibility, and ongoing ownership. Name the evidence that would change your recommendation. “I would start with retrieval plus a draft for agents because it has a clear human review boundary; I would consider autonomous resolution only after the held-out set and a limited pilot demonstrate reliable performance on the specific case types” is stronger than “we should use an agent.”

A practical interview answer structure

Use this sequence when the interviewer gives you an open-ended prompt. It makes your assumptions visible and gives the panel places to challenge the design.

1. Clarify the task and user. Identify the job to be done, the primary user, the decision or action, and the current baseline.
2. Set a safe scope. Define included cases, excluded cases, human-control boundaries, data constraints, and the consequence of a wrong answer.
3. Define success. Choose user outcomes, quality dimensions, critical failure thresholds, and operational measures. Separate hard gates from goals to optimize.
4. Propose the simplest architecture. Draw the deterministic services, retrieval, model, validation, and fallback path. Mark trust boundaries and state-changing tools.
5. Explain evaluation. Describe realistic test data, representative slices, held-out cases, graders, human review, and uncertainty.
6. Plan deployment and operations. Add tracing, privacy controls, staged exposure, stop conditions, model versioning, incident ownership, and rollback.
7. Compare options and name a decision. Choose one path, justify its tradeoffs, and state the evidence that would make you change course.

This is not a script to recite. If the interviewer changes the scale, risk tolerance, or user, update the design instead of defending a memorized diagram. Keep a visible assumption list. When an assumption proves false, explain which part of the design changes and which invariants remain.

If you have not shipped an LLM product, you can still demonstrate senior judgment. Use an analogous system you have built: search ranking, fraud detection, rules engines, recommendation systems, or a human-in-the-loop workflow. Explain what was probabilistic, how errors were measured, where humans entered, and how the system responded to drift. Be accurate about what you personally owned.

Practice the follow-ups that reveal senior judgment

Interviewers often test whether the design survives a changed constraint. Prepare to answer: What if the model is correct 95% of the time but every fifth failure affects a high-value customer? How do you know retrieval improved the answer? What if humans disagree with the rubric? Can a user trigger an expensive tool loop? How would you launch a new model without exposing everyone? Which data can you log? What do you disable during an outage?

Practice concise tradeoff statements. For example: “I would launch as a draft for agents, because the current evidence supports assistance but not autonomous action. I would sample both accepted and edited drafts, audit high-risk categories, and compare resolution and re-open rates against the existing process. If review time grows or a policy violation crosses the stop threshold, we disable generation and retain search.” That answer contains a decision, rationale, measures, and a rollback.