OpenAI·Machine Learning Engineer·Onsite - Coding / Algorithms
Jun 2026
Multi-part ML coding round at OpenAI for an MLE role, all centered on cleaning a human-annotation dataset. The problem built up in stages, each one asking for a more sophisticated filtering approach than the last. Solid problem if you've worked with noisy label pipelines before, rough if you haven't.
- Given a dataset where each item has been labeled by multiple annotators, implement a majority-vote filter to remove low-quality annotations and return a cleaned dataset.
- Extend your solution to compute a per-annotator agreement rate against their peers, and use that to down-weight or exclude annotators who consistently disagree with the group.
- Add confidence scores to the annotations and implement a confidence-weighted aggregation scheme to produce a cleaner final label per item.
- Design a method to identify and remove annotators whose disagreement with peers consistently exceeds some threshold across the full dataset, and explain how you'd set that threshold and handle edge cases like sparse annotators.
“This part felt manageable.” The rest of the author's notes on Machine Learning Engineer interview at OpenAI, Onsite - Coding / Algorithms round, covers how they worked through the question, what the panel pushed back on, and what they would do differently.
View Post