OpenAI·Machine Learning Engineer·Onsite - System Design / Architecture
May 2026
System design round at OpenAI for an ML Engineer role, focused entirely on building a RAG system end to end. Pretty intense scope, they wanted you to cover everything from chunking strategy to eval pipelines to online monitoring, not just the happy path architecture.
- Walk me through how you'd design a RAG system end to end for answering questions over a private document corpus.
- How would you handle document ingestion and preprocessing, specifically parsing, cleaning, and chunking strategy?
- What embedding strategy would you use, and how would you set up the index? Would you use pure vector search or something hybrid?
- How would you approach retrieval, including query understanding, choosing top-k, and filtering? Would you add a reranking step?
- How do you handle cases where retrieval is weak or returns irrelevant context? What guardrails or fallback behavior would you build in?
- Describe your evaluation plan across the full pipeline: chunking quality, retrieval quality, reranking, generation grounding, and overall user success.
- How would you set up online monitoring and a continuous improvement loop for a deployed RAG system?
- How would you design the system to support frequent document updates, like new or changed docs, without degrading retrieval quality?
“This is the kind of question where you think you know what to say and then realize mid-sentence you're already in the weeds.” The rest of the author's notes on Machine Learning Engineer interview at OpenAI, Onsite - System Design / Architecture round, covers how they worked through the question, what the panel pushed back on, and what they would do differently.
View Post