Anthropic·Software Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say

Jul 2026

Summary

Anthropic SWE interview that was basically one long design problem about building a text classifier on top of a black-box LLM scoring API. Dense question, a lot of moving parts, and they clearly wanted to see how far you could push the design before you ran out of ideas.

Questions Asked (1)

Given a helper function that returns per-token probabilities from an LLM, design a complete binary text classifier: construct prompts for each class, produce a continuous probability score per input, handle numerical stability with log-probabilities, aggregate token-level scores to sequence level, define batching, select thresholds, evaluate with ROC-AUC and F1, propose improvements like prompt ensembling and calibration, discuss failure modes, and give pseudocode with complexity analysis.

System DesignTechnical Trade-offsA/B Testing & Experimentation

Author's notes

This was one question but it kept expanding.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the problem scope and assumptions, then walk through the design in logical layers: prompt construction, token-level scoring with log-probabilities, aggregation, batching, threshold selection, and evaluation. Emphasize trade-offs and practical considerations, and finish with improvements, failure modes, and pseudocode with complexity analysis.

Pro tip: Anchor your answer in the LLM's next-token probabilities: treat the classifier as a probabilistic model over class tokens, and always discuss calibration and threshold selection in terms of the downstream decision cost, not just accuracy.

1. Clarify requirements and assumptions

Confirm the input/output format, class labels, available helper function signature, and constraints (latency, cost, batch size). State assumptions about tokenization and probability access.

2. Design prompt construction and scoring

Construct prompts for each class (e.g., 'This text is positive/negative'), extract per-token probabilities for class tokens, and compute log-probabilities for numerical stability. Aggregate token scores to a sequence-level score via sum or average of log-probs.

3. Define batching and threshold selection

Batch inputs for efficient LLM calls, ensuring consistent prompt formatting. Select a threshold by maximizing F1 on a validation set or using cost-sensitive optimization; consider calibration methods like Platt scaling.

4. Evaluate and iterate

Use ROC-AUC and F1 to evaluate performance, analyze failure modes (e.g., ambiguous inputs, prompt sensitivity), and propose improvements like prompt ensembling, calibration, and data augmentation.

5. Provide pseudocode and complexity analysis

Write clear pseudocode covering prompt construction, scoring, aggregation, and thresholding. Analyze time and space complexity, noting LLM call overhead and batching effects.

Key Points to Mention

Use log-probabilities to avoid underflow and enable stable aggregation (sum of log-probs).
Aggregate token-level scores by summing log-probs for the class token sequence, then normalize by length if needed.
Batch inputs to amortize LLM call overhead, but be mindful of padding and prompt length consistency.
Select thresholds via ROC curve or F1 maximization on validation data; consider cost-sensitive thresholds.
Evaluate with ROC-AUC (threshold-independent) and F1 (threshold-dependent), and report calibration metrics like ECE.
Improve with prompt ensembling (multiple prompts per class), calibration (Platt/isotonic), and handle failure modes like prompt sensitivity and out-of-distribution inputs.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.