This one sprawled in a way I didn't expect.
Start by clarifying requirements and data characteristics, then propose a hybrid approach combining rule-based heuristics (e.g., confidence thresholds, time anomalies) with statistical methods (e.g., agreement with gold set, annotator reliability scores). Emphasize iterative refinement, validation against the gold set, and trade-offs between precision and recall in filtering.
Pro tip: Demonstrate awareness of the cold-start problem for new annotators and propose a Bayesian or smoothing approach to avoid over-penalizing them. Also, mention the importance of monitoring filtered data's impact on downstream model performance.
Ask about the size of the gold set, whether annotator IDs are consistent, and the cost of false positives vs. false negatives in filtering. Understand the downstream task and how filtered data will be used.
Establish metrics such as annotator accuracy on gold set, inter-annotator agreement, confidence calibration, and anomaly detection in timestamps. Use these to score annotations and annotators.
Combine heuristics (e.g., drop low-confidence, outlier timestamps) with a probabilistic model (e.g., Dawid-Skene) to estimate true labels and annotator reliability. Output per-annotation drop decisions and per-annotator weights.
Evaluate the filtering on the gold set and via cross-validation. Monitor precision/recall of bad annotation detection and adjust thresholds. Consider A/B testing impact on model training.
Build a scalable pipeline that can handle large datasets, with logging and dashboards for annotator performance. Allow for periodic re-evaluation and updating of weights as more data arrives.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.