Openai·Software Engineer·Technical Phone Screen·Senior

Senior

May 2026

Summary

Technical screen for a Research Engineer role at OpenAI. One meaty ML question about annotator quality filtering and iterative retraining. The kind of problem that sounds scoped until you start thinking about edge cases.

Questions Asked (1)

You have a classification dataset with human-provided labels and per-annotator IDs. Some annotators are noisy. Walk through how you'd score each annotator's reliability, filter out low-quality labels, retrain the classifier, and confirm whether validation accuracy actually improves. Also address how you'd pick thresholds, avoid label leakage, and handle annotators who only have a few labels.

Technical Trade-offsRoot Cause AnalysisA/B Testing & Experimentation

Author's notes

This one took me longer to get into than I expected.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the problem as a data quality improvement pipeline: estimate annotator reliability using agreement and model-based methods, filter labels with statistically justified thresholds, retrain the classifier, and rigorously validate the impact using a clean, held-out test set. Emphasize avoiding leakage by splitting data before any reliability estimation and handling sparse annotators with smoothing or hierarchical models.

Pro tip: Use a small, high-quality gold set to calibrate annotator reliability and validate filtering decisions, and always compare against a baseline trained on all labels to isolate the effect of filtering.

1. Estimate annotator reliability

Compute per-annotator agreement with majority vote or a model (e.g., Dawid-Skene) to get reliability scores, using cross-validation to avoid overfitting. For annotators with few labels, apply Bayesian smoothing or hierarchical modeling to stabilize estimates.

2. Filter low-quality labels

Set a threshold on reliability scores (e.g., via precision-recall on a validation set or cost-benefit analysis) and remove labels from annotators below it. Consider soft filtering by weighting labels by reliability instead of hard removal.

3. Retrain and validate

Retrain the classifier on the filtered dataset and evaluate on a held-out test set that was never used for reliability estimation or filtering. Compare accuracy against a baseline trained on all labels to confirm improvement.

4. Avoid leakage and handle sparse annotators

Split data into train/validation/test before any reliability estimation, and ensure annotator reliability is computed only on training data. For annotators with few labels, use partial pooling or exclude them from filtering decisions if estimates are unreliable.

Key Points to Mention

Inter-annotator agreement metrics (e.g., Cohen's kappa, Fleiss' kappa) and model-based reliability (Dawid-Skene, EM).
Threshold selection via validation set performance or cost-sensitive optimization, and the trade-off between removing noise and losing data.
Data splitting strategy: separate test set before any reliability estimation to prevent leakage.
Handling sparse annotators: Bayesian smoothing, hierarchical models, or setting a minimum label count.
Baseline comparison: always compare filtered model against model trained on all data to measure true improvement.
Validation accuracy confirmation: use statistical significance testing (e.g., bootstrap) to ensure improvement is not due to chance.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.