Gen AI System Design Interview: RAG, Agents, and Scaling (2026 Guide)

How Gen AI System Design Interviews Work

A gen ai system design interview is not “design Twitter.” Interviewers give ambiguous product prompts (“Design a copilot for our support team”) and expect you to clarify requirements, draw boxes, name trade-offs, and talk about evaluation and failure modes.

Unlike traditional SWE system design, Gen AI adds layers interviewers now expect you to address: non-deterministic outputs, retrieval quality, prompt and tool safety, cost per query, and offline evaluation before you ship. You are designing a pipeline where the model might hallucinate, tools might fail, and every request costs money. Strong candidates narrate failure paths as clearly as the happy path.

Key takeaway: Use the same 6-step framework for every prompt; interviewers reward structure over memorized buzzwords.

Is This Guide for You?

This guide is for you if:

• You are a mid-level AI, ML, or software engineer preparing for onsite system design
• You know RAG basics but freeze when asked to design end-to-end
• You want a repeatable whiteboard process with diagrams you can practice

Read something else first if:

• You need RAG fundamentals → RAG explained
• You want component Q&A (chunking, re-ranking) → RAG interview questions
• You want conceptual LLM Q&A → LLM interview questions
• You want agent-specific Q&A → Agentic AI interview questions

The 6-Step Framework (Use Every Time)

1. Clarify requirements: Who are the users? Expected QPS? Latency target? How fresh must data be? Say aloud: “Before I draw anything, who uses this, how often, and are write actions allowed?”
2. Architecture overview: Client → API → orchestrator → retrieval/tools → LLM → response. Show the async indexing path separately. Say: “I will split offline indexing from the online query path so we can reason about latency independently.”
3. Core components: Name only what the prompt needs: chunking, embeddings, vector store, re-ranker, prompt template, tool sandbox. Say: “Let me go deep on retrieval and evaluation; those drive quality here.”
4. Trade-offs: Name two alternatives and pick one. Say: “I considered pgvector for ops simplicity but would pick a managed vector DB if we expect rapid doc growth.”
5. Evaluation: Golden dataset, RAGAS metrics, offline regression plus online signals. Say: “I would not ship without 100 golden Q&A pairs and weekly regression when docs change.”
6. Scale and ops: Caching, batch embedding, rate limits, traces, cost per query, human escalation. Say: “Under traffic spikes, embedding rate limits break first; here is how I would cache and batch.”

RAG vs Agent vs Hybrid: What to Propose First

Before you draw boxes, match the prompt to an architecture pattern. If the interviewer pushes back, that is fine; they want to see your reasoning.

Many production systems are hybrid: a RAG retrieval step grounds the agent before it calls tools. If the prompt mentions both documents and actions, say that explicitly and ask the interviewer which path matters more for the MVP.

For agent orchestrator choices (LangGraph vs CrewAI vs simpler loops), see our LangGraph vs CrewAI vs AutoGen guide; mention it briefly in agent scenarios.

Scenario 1: Design a Production RAG Q&A System

The prompt

“Design a RAG system for 10,000 internal PDFs. Support engineers ask natural-language questions. Target p95 latency under 3 seconds.”

The interviewer is testing whether you can separate offline indexing from online retrieval, justify component choices, and close with evaluation.

Clarifying questions to ask

• How often do docs update: daily, weekly, on every upload?
• Single team or multi-tenant with ACL per document?
• English only or multilingual PDFs?
• Max PDF size and whether tables or diagrams matter for answers?
• Must every answer include a citation link to the source page?

Architecture overview

Walkthrough: what to say while drawing

Start with two paths: “On the left, uploads trigger async ingest: chunk, embed, write to the vector store. On the right, the user query never waits for ingest; it embeds the question, retrieves hybrid BM25 plus dense, re-ranks to three chunks, then generates with streaming so p95 feels fast even if the LLM takes two seconds.” Point at citations last; they build trust for support engineers.

Component deep-dive

Hybrid retrieval: Dense embeddings catch paraphrases; BM25 catches product codes and acronyms support engineers actually type. Say you would A/B hybrid vs dense-only on a golden set before committing. Re-ranking: Retrieve top-10 with fast bi-encoder similarity, then cross-encoder re-rank to top-3 for the LLM context window. This keeps latency predictable while improving precision, the most common quality win in production RAG.

Trade-offs

• pgvector (ops simplicity on existing Postgres) vs managed vector DB (scale, managed ANN)
• Larger embedding model (quality) vs smaller (cost and embed latency)

If interviewer pushes back on pgvector: “At 10K docs we are fine on Postgres; I would migrate when ANN latency exceeds our p95 retrieve budget or doc count grows 10×.”

If asked “why not fine-tune?”, point to RAG vs fine-tuning: internal docs change weekly; RAG keeps answers grounded without retraining.

Failure modes

• Stale index after upload: user queries doc that is not embedded yet
• Retrieval misses on acronyms or internal codenames: hybrid search mitigates
• Hallucination when retrieval returns empty: fall back to “I could not find this in docs” instead of guessing

Evaluation

• 100+ golden Q&A pairs from real support tickets (anonymized)
• RAGAS faithfulness + context recall; weekly regression on re-ingest
• Semantic cache for top 20 frequent queries; async re-index on upload
• Online: thumbs up/down on answers; flag sessions with follow-up questions as low quality

Follow-up questions to expect

• How do you handle PDF tables?: Layout-aware parsing (Unstructured, Docling) or table-specific chunks; mention briefly, do not derail
• How do you enforce ACL?: Metadata filter on team_id at retrieve time; never post-filter in application code alone

Scenario 2: Design a Multi-Tool Agent for Internal Knowledge

The prompt

“Design an agent that searches internal wiki, reads a page, and drafts a summary email to a teammate.”

Here the interviewer cares about tool design, loop control, prompt injection from untrusted wiki content, and human approval before side effects.

Clarifying questions to ask

• Can the agent send email autonomously or draft-only until user confirms?
• Wiki authentication: same SSO as the user or service account?
• Max runtime and step count before timeout?
• Audit log requirements for compliance?
• Expected tasks per day and concurrent users?

Architecture overview

Walkthrough: what to say while drawing

“The user states a goal. The orchestrator (I would use a graph) plans which tool to call: search wiki, fetch the page, draft email. A validator node checks the draft against the original goal. Nothing sends until the user approves. If validation fails, we loop back with a max of 12 steps so we never hang forever.”

Component deep-dive

Graph orchestrator vs linear ReAct: A graph lets you add explicit validator and approval nodes, easier to explain in an interview than magic prompt text. For orchestrator choice, see LangGraph vs CrewAI vs AutoGen.Tool schemas: Each tool returns JSON to reduce tokens and parsing errors. Sandbox wiki fetch and email APIs with rate limits per user.

Trade-offs

• Single agent with three tools vs researcher+writer crew: simpler vs parallel speed
• Sync approval UI vs async email queue after confirm

If interviewer asks single vs multi-agent: “Single agent is enough for this sequential workflow; I would split into two agents only if research and writing can run in parallel on large wikis.”

Failure modes

• Infinite loops when fetch returns empty: hard step cap + duplicate-action detection
• Prompt injection via wiki page content: never merge retrieved text into system prompt
• Tool timeout cascading: per-tool timeouts with graceful degradation to user message

Evaluation

• Task success rate on 50 scripted goals (search → read → draft quality)
• Mean steps and cost per task; p95 latency under SLA
• Zero unsent emails without explicit user approval in test suite

Follow-up questions to expect

• How prevent the agent from emailing the wrong person?: Draft only; confirm recipient in approval UI; no send tool until approved
• How debug failures?: Full trace per step (LangSmith-style); replay with same inputs

For deeper agent Q&A, see Agentic AI interview questions. Browse Agentic AI job listings when you are ready to apply.

Scenario 3: Scaling, Cost, and Latency Under Load

The prompt

“Your RAG app went viral: 50× traffic overnight. Walk through what breaks and how you fix it.”

This tests production instincts: bottlenecks by stage, cost control, and quality monitoring, not horizontal scaling hand-waving.

Clarifying questions to ask

• Current QPS and p95 latency before the spike?
• Daily LLM spend budget and whether product accepts degraded quality temporarily?
• Interactive queries only or batch jobs too?
• Can we turn off re-ranking or switch models under load?

Architecture overview

Walkthrough: what to say while drawing

“Traffic hits four stages: embed, retrieve, re-rank, generate. Embed APIs rate-limit first. Vector DB read latency spikes next. Re-ranker adds fixed cost per query. LLM spend scales linearly with tokens. I would add semantic cache before embed, batch backlog indexing off-peak, route simple queries to a smaller model, and queue non-interactive work.”

Component deep-dive

Semantic cache: Hash normalized query + metadata filters; return cached answer if cosine similarity to prior query exceeds 0.95. Cuts embed and LLM cost for FAQ-style repeats.Model routing: Cheap classifier decides simple vs complex; Haiku-class for lookup, Sonnet-class for synthesis. Track quality on golden set when you downgrade.

What breaks first (by stage)

Cost controls and observability

• Token budgets per request; daily spend alerts to Slack
• Dashboard: p95 by stage, retrieval hit rate, $/query, cache hit rate
• Horizontal API replicas; read replicas on vector store if supported
• Runbook: if faithfulness drops below threshold, re-enable re-ranker and roll back model route

Mentioning a concrete rollback trigger shows you have operated systems under pressure.

Follow-up questions to expect

• How do you know quality did not drop?: A/B on golden set; monitor RAGAS faithfulness hourly; rollback model route if below threshold
• What do you degrade first?: Re-ranker off → reduce top-K → smaller LLM → queue batch; never silent failure

What Interviewers Actually Score

• Structured thinking: you follow the 6 steps visibly; e.g. “Let me clarify requirements before the diagram”
• Requirement clarification: you ask about writes, latency, and freshness before naming Pinecone
• Named trade-offs: “pgvector vs managed DB: I pick X because Y”
• Production awareness: evaluation, logging, empty retrieval fallback, cost per query
• Communication: narrate the diagram; invite corrections: “Does this match your stack?”

Weak answers jump straight to tools; strong ones show a checklist in their head and make it visible. If you finish early, ask for a follow-up constraint (“What if we must stay under $0.02 per query?”) and walk one mitigation per pipeline stage.

Your 2-Week System Design Prep Plan

Week 1

1. Read the 6-step framework; whiteboard Scenario 1 aloud with a 45-minute timer
2. Deepen component knowledge with RAG interview questions
3. Whiteboard Scenario 2; draw the agent diagram from memory without looking

Week 2

1. Whiteboard Scenario 3 (scaling and cost)
2. Mock with a peer: random prompt from the three scenarios above
3. Browse RAG job listings and note which skills repeat in job descriptions

What to Read Next

• LLM and GenAI interview questions: conceptual depth
• RAG interview questions: retrieval and vector DB details
• Agentic AI interview questions: agent architecture and safety
• Interview prep hub: broader Gen AI interview resources
• Browse RAG jobs: see what production teams ask for

Interview patterns reflect the Gen AI hiring market as of May 2026. Confirm round format and expectations with your recruiter before onsite prep.