Openai·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jun 2026

Summary

System design round at OpenAI for an MLE role, focused entirely on building a RAG pipeline from scratch. One long question that kept branching into follow-ups about evaluation at every layer. More to unpack than I expected.

Questions Asked (1)

Design a full RAG system end-to-end, covering data ingestion and chunking, embedding, vector store and retrieval, reranking, LLM generation, and post-processing. For each component, explain how you would evaluate its quality, including what metrics you'd use, whether evaluation is offline or online, and how you'd collect ground-truth data.

System DesignTechnical Trade-offsProduct Analytics & Metrics

Author's notes

This is one of those questions that sounds scoped until you realize every component is basically its own sub-question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Walk through the RAG pipeline component by component, clearly stating for each: the design choice, the evaluation metric(s), whether evaluation is offline or online, and how to obtain ground-truth data. Emphasize trade-offs and justify decisions based on scale, latency, and cost. Conclude with an end-to-end evaluation strategy that ties component metrics to overall system performance.

Pro tip: Frame evaluation as a continuous feedback loop: use offline metrics for rapid iteration, online metrics for real-world validation, and user feedback to continuously improve ground truth. Highlight that ground-truth collection is often the hardest part and propose scalable methods like weak supervision and human-in-the-loop.

1. Data Ingestion & Chunking

Describe the ingestion pipeline (e.g., document loaders, preprocessing) and chunking strategy (fixed-size, semantic, recursive). For evaluation, use offline metrics like chunk coherence and retrieval hit rate on a labeled set; collect ground truth via human annotation of relevant chunks for sample queries.

2. Embedding & Vector Store

Choose embedding model (e.g., OpenAI embeddings) and vector database (e.g., Pinecone, FAISS). Evaluate embedding quality offline via retrieval metrics (recall@k, MRR) on a benchmark dataset; ground truth from human relevance judgments or existing QA datasets. Online, monitor retrieval latency and cache hit rate.

3. Retrieval & Reranking

Implement retrieval (e.g., dense, hybrid) and reranking (e.g., cross-encoder). Offline evaluation: precision@k, NDCG for reranker; ground truth from human-annotated relevance. Online: A/B test different retrievers/rerankers measuring user engagement (CTR, dwell time).

4. LLM Generation & Post-processing

Generate answers with LLM, then post-process (e.g., citation, formatting). Offline evaluation: faithfulness, answer relevance, fluency using metrics like BLEU, ROUGE, BERTScore, or LLM-as-judge; ground truth from human-written answers. Online: user feedback (thumbs up/down), task success rate.

5. End-to-End Evaluation & Iteration

Combine component metrics into holistic evaluation: offline end-to-end QA accuracy, online A/B tests for business metrics (e.g., conversion). Collect ground truth via continuous human annotation, user feedback, and synthetic data generation. Use results to iterate on weak components.

Key Points to Mention

Trade-offs between chunk size and retrieval accuracy; overlap strategies.
Choice of embedding model and vector index (e.g., HNSW, IVF) and their impact on latency/recall.
Reranking techniques (cross-encoders, LLM-based) and their cost/benefit.
Evaluation metrics: retrieval (recall@k, MRR, NDCG), generation (faithfulness, relevance, BLEU, ROUGE), end-to-end (QA accuracy, user satisfaction).
Offline vs online evaluation: offline for fast iteration, online for real-world validation; A/B testing framework.
Ground-truth collection: human annotation, existing datasets, weak supervision, user feedback loops, synthetic data.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.