Openai·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

May 2026

Summary

System design round at OpenAI for a software engineering role. The whole thing was one big open-ended question about video generation infrastructure, and it went deep fast. Walked away feeling like I'd covered maybe 60% of what they wanted.

Questions Asked (1)

Design the backend system for a video generation service similar to Sora. A user submits a request and you need to design the full scheduling and worker pipeline end to end, covering request queuing, worker assignment, GPU pool integration, progress tracking, failure handling with retry and resume, idempotency, cost and priority scheduling, and SLAs for long-running jobs. Each worker can only process one video at a time, the GPU pool size fluctuates, and workers can be terminated at any time.

System DesignTechnical Trade-offsAPI & Integrations

Author's notes

This is one of those questions that sounds like a single question but is actually eight questions stacked on top of each other.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints, then design a high-level architecture that decouples request submission from processing using a durable queue and a scheduler. Dive into the worker lifecycle, GPU pool integration, and failure handling, emphasizing idempotency, retries, and SLAs. Conclude by discussing trade-offs and how the design meets the specific challenges of fluctuating GPU availability and worker termination.

Pro tip: Emphasize that idempotency and checkpointing are critical for long-running jobs on preemptible workers; design the system so that any worker can resume a job from the last checkpoint without duplicating work or corrupting state.

1. Clarify Requirements and Constraints

Ask questions to understand expected scale, latency SLAs, priority levels, cost constraints, and failure semantics. Confirm that workers are ephemeral and GPUs are heterogeneous and dynamically available.

2. High-Level Architecture

Propose a decoupled system: API gateway for request submission, durable queue (e.g., Kafka/SQS) for buffering, scheduler for assigning jobs to workers, and a state store for job metadata and progress. Include a GPU pool manager that tracks available resources.

3. Scheduling and Worker Assignment

Design a scheduler that considers priority, cost, and GPU availability. Use a pull-based model where workers request jobs when ready, or a push-based model with leases. Ensure each worker processes one video at a time and handle worker termination via heartbeats and lease expiration.

4. Failure Handling and Idempotency

Implement retries with exponential backoff, checkpointing for resume, and idempotent job execution using unique job IDs and deduplication. Use a state machine to track job status and ensure exactly-once semantics for critical steps.

5. Monitoring, SLAs, and Trade-offs

Define SLAs for job completion and progress tracking. Discuss monitoring, alerting, and how to handle long-running jobs. Address trade-offs between cost, latency, and reliability, and how the design adapts to fluctuating GPU pools.

Key Points to Mention

Durable queue with at-least-once delivery and idempotent consumers to handle retries and duplicates.
Checkpointing and resume: periodically save intermediate state so jobs can resume after worker termination.
Lease-based worker assignment with heartbeats to detect and recover from worker failures.
Priority and cost-aware scheduling: use multiple queues or weighted fair queuing to balance SLAs and cost.
GPU pool integration: dynamic registration/deregistration of workers, resource tracking, and autoscaling.
Progress tracking and SLAs: expose job status via API, use metrics for monitoring, and define SLOs for completion time.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.