OpenAI·Machine Learning Engineer·Onsite - System Design / Architecture
Apr 2026
Brutal system design round at OpenAI for an MLE role. The question was essentially a full RAG architecture design from scratch, covering everything from ingestion to A/B testing to capacity math. Felt like they were asking for a 2-hour design doc in 45 minutes.
- Design a production-grade RAG system for a customer support assistant that meets strict p99 latency under 1.5 seconds and has real cost constraints. Walk through the full architecture: document ingestion, chunking, embedding model choice, index type, caching, re-ranking, prompt orchestration, and safety guardrails.
- How would you handle document updates and deletions in the vector index, and how do you enforce data isolation across multiple tenants?
- What's your plan for handling failure modes like empty retrieval results, timeouts mid-request, and stale cached responses?
- Define the key metrics you'd track for this system, both offline and online, and design an experiment comparing a BM25 plus cross-encoder re-ranker setup against a dense-only retrieval approach.
- Give a back-of-the-envelope capacity estimate for a system handling 10 million documents at 50 QPS. Cover index size, throughput requirements, and rough cost.
- If you only had 15 minutes to present a coherent version of this plan, what would you prioritize and what would you explicitly cut?
“This is basically a full system design in one breath.”