Anthropic·Machine Learning Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say

Jun 2026Remote

Summary

Anthropic ML interview, one meaty technical question on probability calibration that covered a surprising amount of ground for a single prompt. Felt like a design question dressed up as a theory question.

Questions Asked (1)

You have a binary classifier whose raw output scores aren't well-calibrated. Walk through how you'd fix that using Platt scaling and isotonic regression, explain the tradeoffs between the two, and describe how you'd actually measure whether your calibration is any good.

Technical Trade-offsAlgorithms & Data StructuresSystem Design

Author's notes

I knew Platt scaling well enough, it's just fitting a logistic regression on top of your model's scores using a held-out set.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining calibration and why it matters, then explain Platt scaling and isotonic regression with their mathematical intuitions and tradeoffs. Finish by describing evaluation metrics like reliability diagrams, ECE, and proper scoring rules, and how to choose between methods based on data size and model complexity.

Pro tip: Emphasize that calibration should be validated on a separate calibration set and that isotonic regression can overfit with small data, so Platt scaling is often safer unless you have ample data. Mention that calibration doesn't change ranking, only the probability estimates.

1. Define calibration and its importance

Explain that a calibrated classifier's output scores reflect true probabilities (e.g., among predictions with score 0.8, ~80% should be positive). Highlight why calibration matters for decision-making, cost-sensitive applications, and interpretability.

2. Describe Platt scaling

Explain that Platt scaling fits a logistic regression model to the raw scores (or logits) using a separate calibration set, learning parameters A and B to map scores to probabilities. Mention it's parametric, works well with small data, but assumes a sigmoid relationship.

3. Describe isotonic regression

Explain that isotonic regression is a non-parametric method that fits a piecewise constant, monotonically increasing function to the scores. It's more flexible and can capture any monotonic relationship, but requires more data to avoid overfitting.

4. Compare tradeoffs

Discuss that Platt scaling is less prone to overfitting and works with limited data, but may underfit complex calibration curves. Isotonic regression is more expressive but needs large datasets and can overfit, leading to step-like, non-smooth probability estimates.

5. Measure calibration quality

Describe using reliability diagrams (calibration curves) to visualize alignment, and quantitative metrics like Expected Calibration Error (ECE), Maximum Calibration Error (MCE), and proper scoring rules (Brier score, log loss). Mention that these should be computed on a held-out test set.

Key Points to Mention

Calibration set must be separate from training and test sets to avoid leakage.
Platt scaling learns parameters via maximum likelihood on the calibration set.
Isotonic regression uses pool-adjacent-violators algorithm (PAVA) to enforce monotonicity.
Tradeoff: bias-variance tradeoff—Platt scaling has higher bias, lower variance; isotonic has lower bias, higher variance.
Evaluation metrics: reliability diagram, ECE, MCE, Brier score, log loss.
Calibration does not affect ranking metrics like AUC; it only changes probability estimates.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.