Summary
System design round at American for an ML Engineer role, focused entirely on building an enterprise AI assistant over internal documents. The whole interview was one big design problem with a lot of sub-questions branching off it. Dense session, probably the most technically wide-ranging interview I've done in a while.
Questions Asked(8)
Started here and it felt like a warmup but they pushed pretty hard on the boundaries.
Suggested Approach
Structure your answer as a progression from simplest to most complex, explaining each concept clearly before showing how they build upon one another. Use a concrete, relatable use case (e.g., a customer support system) to ground each concept, then explicitly articulate the trade-offs and relationships between all three paradigms.
Define the Base Language Model
Explain that a base (or foundation) LLM is a pre-trained model with parametric knowledge frozen at a training cutoff, capable of generating text but with no access to external or real-time information. Highlight its core limitations: hallucination risk, stale knowledge, and no grounding in proprietary data.
Introduce RAG as an Enhancement Layer
Describe RAG (Retrieval-Augmented Generation) as a pattern that augments the LLM with a retrieval step — fetching relevant documents from an external knowledge base at inference time and injecting them into the prompt as context. Emphasize that RAG addresses the knowledge staleness and grounding problems without retraining the model.
Define Agents as Dynamic Orchestrators
Explain that an agent wraps an LLM with the ability to reason, plan, and take actions — calling tools (APIs, code executors, search engines, databases) iteratively in a loop until a goal is achieved. Unlike RAG, which is a single retrieval-then-generate pass, agents support multi-step, conditional workflows.
Articulate the Relationships and Hierarchy
Make explicit that these are not mutually exclusive: RAG is often a tool used inside an agent, and all three rely on the base LLM as the core reasoning engine. Draw the hierarchy: Base LLM → RAG (adds grounded context) → Agent (adds planning, tool use, and multi-step execution).
Discuss Trade-offs and When to Use Each
Address the engineering trade-offs — base LLM is fastest and cheapest but least grounded; RAG adds latency and infrastructure complexity but improves factuality; agents add the most power but introduce non-determinism, higher latency, cost, and debugging complexity. Tie this back to system design decisions based on the use case requirements.
Key Points to Mention
This one I liked.
Suggested Approach
Frame your answer around the layered architecture of agentic systems, explaining how each component (tools, APIs, planners, execution policies) adds a distinct capability dimension to an agent. Use a concrete example — such as a travel booking agent or a code generation agent — to ground abstract concepts in a real-world scenario. Conclude by discussing the trade-offs involved in designing these capability layers for production systems.
Define the Agent Capability Model
Start by establishing that a bare LLM has limited real-world reach, and that tools, APIs, planners, and policies are the mechanisms that extend it into an actionable system. Briefly define each component's role before diving into details.
Explain Tools and APIs as Capability Extensions
Describe how tools (e.g., web search, code execution, calculators) and APIs (e.g., REST endpoints, database connectors) give agents access to external data and actions beyond their training knowledge. Highlight how function-calling or tool-use interfaces (like OpenAI function calling or LangChain tools) enable structured invocation.
Describe Planners as Orchestration Intelligence
Explain how planners (e.g., ReAct, Chain-of-Thought, task decomposition via frameworks like AutoGPT or LangGraph) allow agents to break complex goals into sub-tasks and decide which tools to invoke in what order. Mention the difference between single-step and multi-step planning.
Cover Execution Policies as Safety and Control Layers
Discuss how execution policies govern agent behavior — including retry logic, rate limiting, permission scopes, human-in-the-loop checkpoints, and guardrails against harmful actions. Emphasize that policies are what make agents trustworthy and production-ready.
Address Trade-offs and Design Considerations
Wrap up by discussing key trade-offs such as autonomy vs. controllability, latency vs. capability richness, and static tool registries vs. dynamic tool discovery. Tie these back to real engineering decisions you would make at scale.
Key Points to Mention
Split it into three buckets: in-context window for the current session, a session store (Redis or similar) for conversation history across turns, and a longer-term vector or key-value store for user preferences and past task outcomes.
Suggested Approach
Start by distinguishing the two memory tiers — short-term (in-session, ephemeral) and long-term (persistent, cross-session) — and explain the different storage and retrieval mechanisms each requires. Then walk through a concrete architecture that addresses latency, scalability, and relevance, tying design choices back to real ML system constraints like context window limits and retrieval accuracy.
Define Memory Tiers and Requirements
Clearly separate short-term memory (current session context, low latency, ephemeral) from long-term memory (user preferences, past tasks, persistent). Articulate the access patterns, retention policies, and freshness requirements for each tier.
Design Short-Term Conversation State
Describe an in-memory or fast cache solution (e.g., Redis) that stores the recent conversation turns, structured as a sliding window or token-budget-aware buffer to fit within model context limits. Address session lifecycle management and TTL-based expiration.
Design Long-Term Memory Storage
Propose a dual-storage approach: a relational or document store (e.g., PostgreSQL, DynamoDB) for structured user/task metadata, and a vector database (e.g., Pinecone, Weaviate, pgvector) for semantic retrieval of past interactions or knowledge. Explain how embeddings enable fuzzy, relevance-based recall.
Address Memory Retrieval and Injection
Explain how relevant long-term memories are retrieved at query time — using semantic search or keyword filtering — and injected into the model's prompt or context window without exceeding token limits. Discuss ranking or summarization strategies to prioritize the most relevant memories.
Handle Privacy, Consistency, and Scalability
Discuss user data privacy (PII handling, data retention policies, opt-out mechanisms) and consistency challenges when memory is updated concurrently. Address horizontal scalability of the memory layer to support many users and high throughput.
Key Points to Mention
Grounding was the interesting one.
Suggested Approach
Structure your answer by addressing each evaluation dimension systematically, then discuss how they interact and create trade-offs in a real production system. Ground your response in concrete metrics and tooling choices, demonstrating that you understand both offline evaluation during development and online monitoring in production.
Define Evaluation Dimensions Clearly
Briefly define what each dimension means in the context of this specific system (e.g., answer quality via ROUGE/BERTScore or human eval, grounding via citation accuracy, task success via goal completion rate). This shows precision and avoids ambiguity before diving into measurement strategies.
Establish Offline vs. Online Evaluation Strategy
Distinguish between offline evaluation (benchmark datasets, automated scoring, human annotation pipelines) and online evaluation (A/B testing, shadow deployments, real-time logging). Explain when each is appropriate and how they complement each other.
Propose Concrete Metrics and Tooling
For each dimension, name specific metrics and tools — e.g., LLM-as-judge for answer quality, faithfulness scores (RAGAs) for grounding, p50/p95 latency via distributed tracing, and token-level cost tracking via provider APIs. Concrete tooling signals hands-on experience.
Address Trade-offs and Prioritization
Discuss how improving one dimension can degrade another (e.g., adding retrieval steps improves grounding but increases latency and cost) and explain how you would set acceptable thresholds based on business requirements and user tolerance.
Describe a Continuous Monitoring and Feedback Loop
Explain how you would set up dashboards, alerting, and a data flywheel — collecting user feedback signals (thumbs up/down, session abandonment) to continuously retrain or re-rank and close the evaluation loop in production.
Key Points to Mention
Honestly the security angle surprised me a little in an ML design interview.
Suggested Approach
Frame your answer around a defense-in-depth strategy, addressing each threat vector (prompt injection, unsafe tool execution, data leakage) with layered controls at the model, application, and infrastructure levels. Demonstrate that you understand these are not purely ML problems but require a systems-thinking approach combining security engineering with ML-specific mitigations. Tie your recommendations to real-world trade-offs such as latency, cost, and user experience.
Define the Threat Model
Start by scoping the attack surface: who are the adversaries (end users, third-party content, malicious tool outputs), what assets are at risk (PII, proprietary data, system integrity), and what is the blast radius of a successful attack. This shows structured security thinking before jumping to solutions.
Defend Against Prompt Injection
Describe layered mitigations: strict input/output sanitization, separating system instructions from user content using delimiters or structured formats, using a secondary LLM-based classifier to detect injection attempts, and applying least-privilege prompting so the model has minimal context about sensitive internals.
Secure Tool and Function Execution
Enforce a strict allowlist of permitted tools and parameters, run tool calls in sandboxed environments (e.g., containers with no network egress), require human-in-the-loop confirmation for high-risk actions, and validate tool outputs before feeding them back into the model context.
Prevent Data Leakage
Apply output filtering and PII detection (regex + ML classifiers) before responses reach the user, enforce role-based access control so the model only retrieves data the user is authorized to see, and avoid storing sensitive data in long-term memory or vector stores without encryption and access controls.
Monitor, Audit, and Iterate
Implement comprehensive logging of all prompts, tool calls, and responses (with appropriate redaction), set up anomaly detection for unusual query patterns, and conduct regular red-teaming and adversarial testing to surface new attack vectors before they reach production.
Key Points to Mention
Stateless inference workers behind a load balancer, session state externalized to a shared store with user-scoped keys.
Suggested Approach
Frame your answer around a layered architecture that addresses concurrency, isolation, and reliability as distinct but interconnected concerns. Start by clarifying the scale and use case (e.g., ML inference serving, model training jobs, or interactive notebooks), then walk through concrete design decisions at each layer. Ground your answer in trade-offs rather than presenting a single 'correct' solution.
Clarify Requirements & Scale
Ask about expected concurrent users, session duration, latency SLAs, and whether sessions involve stateful resources like loaded models or user-specific data. This scoping demonstrates engineering maturity and prevents over- or under-engineering.
Design for Session Isolation
Propose mechanisms such as containerization (Docker/Kubernetes pods), namespaced resources, or per-user sandboxed environments to ensure one user's session cannot interfere with another's. For ML workloads, discuss isolating model state, feature stores, and GPU memory allocations.
Handle Concurrency & Resource Management
Describe horizontal scaling with a load balancer, request queuing (e.g., Celery, Ray, or a message broker like Kafka), and resource pooling strategies to efficiently serve many users without resource exhaustion. Address GPU sharing or batching strategies if inference is involved.
Ensure Reliability & Fault Tolerance
Introduce redundancy through replicated services, health checks, circuit breakers, and graceful degradation so that a single node or session failure does not cascade. Mention session state persistence (e.g., Redis, distributed cache) so sessions can be recovered or migrated.
Discuss Trade-offs & Monitoring
Acknowledge the cost vs. performance trade-offs (e.g., dedicated vs. shared resources, strong vs. soft isolation) and explain how you would monitor the system using metrics like latency percentiles, queue depth, and error rates to iterate on the design.
Key Points to Mention
This was the one I felt least prepared for.
Suggested Approach
Frame your answer around the core pillars of durable workflow design: state persistence, idempotency, and fault isolation. Walk through a concrete architecture using real tools (e.g., Temporal, Apache Airflow, or AWS Step Functions) while explicitly addressing how each design decision handles failure modes. Tie your answer back to ML-specific challenges like long-running training jobs, multi-agent orchestration, and non-deterministic outputs.
Define Durability Requirements
Start by scoping what 'durable' means for the specific workflow — identify the longest acceptable recovery time, acceptable data loss window, and whether tasks are idempotent. This sets the foundation for all subsequent design decisions.
Choose a State Persistence Strategy
Explain how workflow state (task progress, intermediate outputs, agent decisions) is persisted externally — e.g., in a database, object store, or a dedicated orchestration engine like Temporal or Step Functions. Emphasize that in-memory state is the enemy of durability.
Design Retry and Failure Recovery Logic
Describe tiered retry policies: exponential backoff with jitter for transient failures, dead-letter queues or human-in-the-loop escalation for permanent failures. Distinguish between task-level retries and workflow-level compensation (saga pattern).
Implement Pause and Resume Mechanisms
Explain how workflows can be paused — either via explicit signals (e.g., Temporal signals/queries) or by serializing state to a checkpoint store — and resumed without re-executing completed steps. For multi-agent tasks, discuss how agent context and memory are preserved across pauses.
Observability and Testing for Failure Scenarios
Highlight the need for distributed tracing, structured logging per task/agent, and alerting on SLA breaches. Mention chaos testing or fault injection to validate recovery paths before production deployment.
Key Points to Mention
PDFs with scanned pages, spreadsheets where meaning lives in cell relationships, support tickets that are short and noisy, policies that are long and structured.
Suggested Approach
Structure your answer by first categorizing the major document types found in enterprise environments, then systematically walking through the technical challenges each category introduces across the ingestion, indexing, and retrieval pipeline. Ground your response in real-world ML engineering concerns like OCR quality, embedding dimensionality, and chunking strategies to demonstrate hands-on experience.
Enumerate Document Categories
Classify enterprise documents into structured (spreadsheets, databases exports), semi-structured (JSON, XML, HTML, forms), and unstructured (PDFs, Word docs, emails, presentations, scanned images). This taxonomy immediately shows systematic thinking and sets up the rest of your answer.
Highlight Ingestion Challenges
Discuss format-specific parsing complexity — e.g., PDFs can be text-native or image-based requiring OCR, PowerPoint slides mix text with visual layouts, and emails contain nested threads and attachments. Mention tools like Apache Tika, PDFMiner, or Unstructured.io and their trade-offs.
Address Indexing Challenges
Cover chunking strategy decisions (fixed-size vs. semantic vs. hierarchical), handling of tables and figures that lose meaning when split, and embedding model selection for domain-specific vocabulary. Discuss metadata extraction and how it enriches retrieval without inflating index size.
Discuss Retrieval Challenges
Explain how heterogeneous document types create retrieval quality disparities — dense retrieval may underperform on highly structured tabular data where sparse BM25 excels. Address multi-modal retrieval needs when documents contain charts or diagrams critical to meaning.
Propose Mitigation Strategies and Trade-offs
Conclude with architectural decisions such as hybrid retrieval pipelines, document-type-specific preprocessing branches, re-ranking layers, and quality filtering thresholds. Acknowledge trade-offs between latency, cost, and retrieval accuracy to show engineering maturity.
Key Points to Mention
Discussion(8)
Sign in to join the discussion.
The versioning problem you hit on is one of those things that sounds like an ops detail but is actually a product correctness problem. If a policy updates and the old version is still in your index, users get stale answers with citations that look authoritative. Re-indexing is necessary but not sufficient: you also need to invalidate any cached retrievals or generated answers that were based on the old chunks, which means you need to track provenance through your caching layer. Most teams don't build that until they get burned by it.
On chunking strategy for tables and spreadsheets: the approach I've found most defensible is to treat each table as a unit and convert it to a structured text representation (markdown table or a natural language description of the row/column semantics) rather than chunking it by character count. For hierarchical policy documents, parent-child chunking where you embed at the section level but retrieve with surrounding context preserves more meaning than flat fixed-size windows. Neither approach is perfect but both are much better than naively splitting on token count.
Your ladder framing is solid and the multi-hop intuition is right, but I think the cleaner way to draw the RAG-vs-agent boundary is around who controls the retrieval loop. In a RAG setup, retrieval happens once, before generation, and the model just conditions on whatever came back. The query goes in, chunks come out, answer gets generated, done. An agent can decide to retrieve again mid-reasoning, reformulate the query based on what it found, call a different tool entirely, or decide it has enough and stop. The loop is under the model's control rather than being a fixed pipeline step.
The 'one document vs multi-hop' framing you used is a reasonable heuristic but it can break down under pressure. You can do multi-hop with RAG if you chain retrieval calls explicitly in your pipeline code, it just gets brittle and hard to generalize. The real reason you reach for agents is when you can't enumerate the retrieval strategy ahead of time, when the number of steps or which tools to call depends on what the model finds along the way.
For an enterprise doc assistant specifically, a lot of questions actually are answerable with a single well-constructed retrieval pass plus a good reranker. Agents add latency, cost, and failure modes. The honest answer to 'when do you need an agent' is probably 'later than you think, and only after you've exhausted what a well-tuned RAG pipeline can do.'
Sampling for offline eval is the right call and I think your answer was reasonable. The one thing I'd tighten: be specific about what percentage and why. Saying 'a small sample' sounds hand-wavy. Something like 1-5% with stratified sampling across query types and user cohorts is more concrete and shows you've thought about coverage versus cost. If you say 1% flat, an interviewer will ask what happens if a whole query category is underrepresented in that sample.
The human-in-the-loop routing answer is right but the interesting follow-up to that question is: what does the workflow state look like while it's paused waiting for a human? You need to serialize the full execution context, store it somewhere durable, and have a mechanism for the human's response to resume the workflow at exactly the right step with the corrected output injected. That's not trivial, especially if the workflow has been paused for hours and some upstream state has changed in the meantime.
Idempotency is the other thing worth being precise about. Idempotent doesn't just mean 'same result on retry,' it means the side effects don't compound. If a tool call sends an email or writes to a database, retrying it naively is a problem. You need either a deduplication key that the downstream system honors, or a two-phase pattern where you record intent before executing and check for prior completion before re-executing. Most people describe idempotency correctly in the abstract and then design tool execution in a way that violates it.
Your three-bucket split is solid and the summarize-and-embed-at-session-close idea is genuinely the right answer to the persistence question. The thing I'd add is that retrieval on next session start shouldn't just be by user ID lookup, it should be a semantic search over that user's past session summaries keyed to the current query. Otherwise you're loading everything the user ever did, which gets expensive and noisy fast.
The part that tripped me up when I thought through this problem was write timing. Do you persist memory synchronously during the session or async after it closes? Async is cheaper but you lose data if the session crashes before the write completes. For an enterprise assistant where users might be doing something consequential, that's not a great failure mode.
Execution policies being a distinct concept is something I also had to reconstruct on the fly once. The way I think about it now: tools define what the agent can do, the planner decides what to do and in what order, and execution policy governs how each tool call actually gets dispatched and what happens when things go wrong. They're genuinely separate concerns and conflating planner with execution policy is where a lot of agent implementations get messy.
On structured output for reliable tool dispatch, the thing worth knowing cold is that the model needs to emit something machine-parseable, and the failure mode is partial or malformed JSON when the model gets uncertain. Some teams solve this with constrained decoding, others with a retry loop that re-prompts with the parse error included. Both work but they have different latency profiles. The constrained decoding approach is cleaner but requires framework support. Worth having an opinion on that tradeoff because it comes up whenever you're designing the tool-calling layer seriously.
Indirect prompt injection through retrieved content is genuinely the hard problem here and most people don't have a clean answer because there isn't one yet. The attack surface is real: a malicious document in your index could contain instructions that hijack the model's behavior when that chunk gets retrieved and included in the prompt. The secondary classification pass you described helps for direct injection but the indirect case is harder because the retrieved content looks legitimate at the document level.
The most credible answer I've seen for this is a combination of things: treat retrieved content as untrusted input by wrapping it in a structured prompt template that clearly demarcates document content from instructions, use a separate pass that checks the final composed prompt for instruction-like patterns before it hits the model, and limit what tools are available when the model is operating over retrieved content specifically. None of these fully solve it but together they raise the bar enough to be defensible. The honest thing to say in an interview is that this is an active area and belt-and-suspenders mitigation is the current best practice, not a complete solution.
Stateless workers plus externalized session state is the right foundation and namespace isolation in the session store is exactly how you enforce the per-user boundary. The one thing I'd add is that retrieval context isolation matters as much as session state isolation. If two users' queries end up sharing a cached retrieval result that one of them isn't authorized to see, you've got a data leak that your session architecture didn't prevent. Access control has to be enforced at retrieval time, not just at the session layer.