Bytedance·Data Scientist·Technical Phone Screen·Intermediate

Intermediate

Jun 2026

Summary

Bytedance data scientist interview with a meaty ML question on classification thresholds. One question but it had a bunch of sub-parts that each could've been their own conversation.

Questions Asked (1)

You've built a binary classifier that outputs probabilities. Walk through how you'd pick a decision threshold on a validation set, how you'd factor in asymmetric costs between false positives and false negatives, how class imbalance and calibration issues affect your choice, and give examples of when you'd want a high vs low threshold.

Technical Trade-offsProduct Analytics & MetricsAlgorithms & Data Structures

Author's notes

This question looks like one thing but it's really four.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the business objective and cost structure, then describe a systematic threshold selection process using validation set metrics like expected cost or F-beta score. Address class imbalance and calibration as factors that influence threshold choice, and illustrate with concrete examples of high vs low threshold scenarios.

Pro tip: Always tie the threshold to a business metric (e.g., revenue, cost, or user experience) and mention that you would validate the chosen threshold on a holdout set to avoid overfitting. Also, consider that thresholds may need to be re-evaluated over time as data distributions shift.

1. Define the cost matrix and business objective

Quantify the cost of false positives and false negatives in business terms (e.g., dollars, user churn). Determine whether the goal is to minimize expected cost, maximize F-beta, or meet a specific precision/recall requirement.

2. Evaluate threshold performance on validation set

Compute relevant metrics (precision, recall, F1, expected cost) across a range of thresholds on the validation set. Plot these metrics to visualize trade-offs and identify candidate thresholds.

3. Adjust for class imbalance and calibration

If classes are imbalanced, use metrics like precision-recall AUC or cost-sensitive learning. Check calibration (e.g., reliability diagram) and apply techniques like Platt scaling or isotonic regression if probabilities are miscalibrated, as this affects threshold interpretation.

4. Select threshold based on cost-benefit analysis

Choose the threshold that optimizes the business objective, such as minimizing expected cost or maximizing F-beta. Consider operational constraints (e.g., limited human review capacity) that may impose a maximum false positive rate.

5. Validate and monitor

Validate the chosen threshold on a holdout set to ensure generalization. Set up monitoring to detect drift and re-evaluate the threshold periodically or when business costs change.

Key Points to Mention

Cost matrix: explicitly define the relative cost of false positives vs false negatives.
Metrics: use precision, recall, F-beta, and expected cost; avoid accuracy due to imbalance.
Class imbalance: discuss how it affects threshold choice and metrics like PR-AUC.
Calibration: mention reliability diagrams and methods to recalibrate probabilities.
Examples: high threshold for spam detection (avoid false positives), low threshold for cancer screening (avoid false negatives).
Validation: use a separate validation set to tune threshold and a test set to confirm.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.