Openai·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jun 2026

Summary

Onsite system design round at OpenAI for an ML Engineer role. The main question was a deep debugging scenario around a broken Transformer training pipeline, and they really wanted to see how you'd systematically tear it apart rather than just throw fixes at it.

Questions Asked (1)

You have a Transformer-based sequence model that intermittently throws shape or dtype mismatch errors and stops converging after several thousand training steps. Walk through your complete debugging approach from start to finish, covering tokenization, attention masks, gradient issues, data bugs, numerical instability, checkpointing, multi-GPU problems, and memory profiling.

Root Cause AnalysisSystem DesignTechnical Trade-offs

Author's notes

This question is enormous and I think that's the point.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by reproducing the error in a minimal, deterministic setting to isolate the cause, then systematically rule out common culprits in order: data/tokenization, model architecture (masks, dtypes), optimization (gradients, instability), and distributed training issues. Use profiling and logging to gather evidence at each stage, and always verify fixes with a controlled experiment before scaling up.

Pro tip: Treat the training pipeline as a series of contracts (shapes, dtypes, value ranges) and add assertions or logging at each boundary; this turns intermittent bugs into deterministic failures you can debug. Also, keep a known-good baseline (e.g., a small dataset and model) to quickly test changes.

1. Reproduce and isolate

Create a minimal, deterministic reproduction (fixed seed, small batch, single GPU) that triggers the error. Log shapes, dtypes, and value ranges at key points to identify where the mismatch first occurs.

2. Audit data and tokenization

Verify tokenizer consistency (vocab, special tokens, padding/truncation), check for out-of-range token IDs, and ensure input_ids, attention_mask, and labels are correctly aligned and typed. Inspect batches for anomalies like all-padding sequences.

3. Validate model and masks

Check attention mask construction (causal, padding, combined) and dtype compatibility (e.g., float16 vs float32). Ensure positional encodings and layer norms are correctly applied, and that no operations silently cast tensors.

4. Diagnose optimization and numerical stability

Monitor gradient norms, check for NaNs/Infs, and adjust learning rate, warmup, or gradient clipping. Consider loss scaling for mixed precision and verify that loss computation ignores padding tokens correctly.

5. Investigate distributed and memory issues

For multi-GPU, verify DDP setup, gradient synchronization, and batch splitting. Use memory profiling to detect leaks or fragmentation, and check checkpoint saving/loading for consistency across devices.

Key Points to Mention

Tokenization pitfalls: mismatched vocab, incorrect special tokens, padding/truncation errors, and out-of-vocab IDs.
Attention mask bugs: incorrect causal masking, padding mask not combined, or mask dtype causing silent errors.
Gradient issues: vanishing/exploding gradients, NaN losses, and the need for gradient clipping and learning rate warmup.
Numerical instability: mixed precision (fp16/bf16) causing overflow/underflow, loss scaling, and accumulation dtype.
Checkpointing: saving/loading optimizer state, model state, and ensuring compatibility across restarts and devices.
Multi-GPU problems: DDP synchronization, uneven batch sizes, and device placement errors.
Memory profiling: using tools like PyTorch profiler or TensorBoard to track memory usage and identify leaks.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.