OpenAI·Software Engineer·Onsite - System Design / Architecture
May 2026
System design round at OpenAI for a software engineer role, focused entirely on designing a chatbot service from scratch. The scope was broader than I expected, covering everything from streaming to safety pipelines, and I felt like I was playing catch-up the whole time.
- Design a chatbot service that supports multi-turn conversations, multiple concurrent conversations per user, streaming responses, and persisted conversation history.
- How would you handle context window limits as conversations grow longer, and what tradeoffs exist between truncation, summarization, and retrieval-based approaches?
- Walk through how you'd architect the LLM inference layer to support high throughput with dynamic batching and KV-cache reuse for streaming.
- What does your safety pipeline look like, covering input moderation, output moderation, jailbreak detection, and PII redaction?
- How would you approach observability for this system, including cost tracking, latency monitoring, and supporting A/B tests for prompt or model changes?
“This is the kind of question where you think you know where to start and then realize you have no idea how deep it goes.” The rest of the author's notes on Software Engineer interview at OpenAI, Onsite - System Design / Architecture round, covers how they worked through the question, what the panel pushed back on, and what they would do differently.
View Post