Jane Street·Software Engineer·Technical Phone Screen·Intermediate

IntermediatePrefer not to say

Apr 2026

Summary

Jane Street SWE interview with a heavy ML/systems flavor. The whole thing was built around a CNN evaluation notebook for FashionMNIST, and each part layered on more complexity. Felt more like a research problem set than a traditional coding screen.

Questions Asked (4)

You're given a trained CNN and a test set. Implement a row-wise reveal evaluation where at step k, the top k rows are visible and the rest are replaced by a fixed mask value. Record predictions at each k, compute accuracy vs k across the test set, plot the curve, and explain what it tells you about how much information the model actually needs.

Technical Trade-offsProduct Analytics & Metrics

Author's notes

This was the setup question but it's deceptively meaty.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the evaluation setup: the model is fixed, only the input is progressively revealed from top to bottom. Then outline the implementation steps: create masked inputs for each k, run inference, compute accuracy, and plot accuracy vs k. Finally, interpret the curve in terms of information sufficiency and model reliance on spatial features.

Pro tip: Mention that you would also track per-class accuracy or confidence to see if certain classes need more rows, and discuss the computational cost of repeated inference, suggesting batching or caching to optimize.

1. Clarify the setup and assumptions

Confirm that the model is pre-trained and frozen, and that masking replaces hidden rows with a constant value (e.g., 0). Discuss whether the mask value should be the dataset mean or a neutral value, and how it might affect the model.

2. Implement the row-wise reveal evaluation

For each k from 1 to H (number of rows), create a copy of the test set where rows k+1 to H are set to the mask value. Run the model on these masked inputs and record predictions.

3. Compute and plot accuracy vs k

Calculate accuracy across the entire test set for each k. Plot accuracy on the y-axis and k on the x-axis to visualize how performance improves as more rows are revealed.

4. Interpret the curve

Analyze the shape: a steep initial rise indicates the model relies on top rows; a plateau suggests additional rows add little information. Compare to baseline (random guessing) and full-image accuracy.

5. Discuss implications and extensions

Explain what the curve reveals about the model's information needs, potential biases (e.g., positional bias), and suggest further experiments like revealing from bottom or random rows.

Key Points to Mention

Masking strategy: using a fixed value (e.g., 0 or mean) and its potential impact on model behavior.
Computational efficiency: batching multiple k values or caching intermediate activations to avoid redundant forward passes.
Baseline comparisons: accuracy at k=0 (all masked) and k=H (full image) to contextualize the curve.
Curve interpretation: identifying the point of diminishing returns and what it implies about feature importance.
Potential pitfalls: overfitting to the mask value, positional bias in the model, and the need for statistical significance (e.g., confidence intervals).
Extensions: per-class analysis, different reveal orders (bottom-up, random), and using the curve to guide model pruning or input reduction.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Define a reward function for partial image reveals where the reward equals the number of still-masked pixels if the final prediction is correct, and zero otherwise. Using your accuracy-vs-k results, propose and implement a method to find a single global mask fill value that maximizes expected reward across the dataset. Also discuss trade-offs like class imbalance and distribution shift introduced by masking.

A/B Testing & ExperimentationTechnical Trade-offsAlgorithms & Data Structures

Author's notes

This is where it got interesting.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

First, clarify the reward function and the goal: find a single global mask fill value that maximizes expected reward across the dataset. Then, using your accuracy-vs-k results, model the expected reward as a function of the fill value and optimize it, likely via grid search or another method. Finally, discuss trade-offs such as class imbalance and distribution shift introduced by masking.

Pro tip: Emphasize that the optimal fill value depends on the model's accuracy at different mask levels and the reward structure; a value that maximizes accuracy may not maximize expected reward due to the reward being zero for incorrect predictions.

1. Understand the reward function and objective

Restate the reward: if the final prediction is correct, reward equals the number of still-masked pixels; otherwise, zero. The goal is to find a single global mask fill value that maximizes expected reward across the dataset.

2. Leverage accuracy-vs-k results

Use your existing accuracy-vs-k results, which show how model accuracy varies with the number of revealed pixels (k). This relationship is crucial for estimating the probability of correct prediction for a given fill value.

3. Model expected reward as a function of fill value

For a candidate fill value, determine the number of revealed pixels (k) it produces for each image (or on average). Then, using accuracy-vs-k, estimate the probability of correct prediction. The expected reward for an image is (number of masked pixels) * P(correct | k). Average over the dataset to get overall expected reward.

4. Optimize the fill value

Search over possible fill values (e.g., grid search over pixel intensity range) to find the one that maximizes the estimated expected reward. If the relationship is smooth, use gradient-based optimization or other search methods.

5. Discuss trade-offs and implementation

Address class imbalance: masking may disproportionately affect minority classes, and the fill value could bias predictions. Distribution shift: the masked images may not match the training distribution, affecting accuracy. Implement the chosen fill value and validate on a held-out set.

Key Points to Mention

The reward function incentivizes correct predictions with more masked pixels, so the optimal fill value balances accuracy and the number of masked pixels.
Accuracy-vs-k results provide the necessary link between fill value and prediction accuracy, but may need to be interpolated or extrapolated.
Expected reward calculation: E[reward] = E[ (number of masked pixels) * I(prediction correct) ].
Class imbalance: masking might cause the model to favor majority classes, reducing reward for minority classes; consider per-class analysis.
Distribution shift: the fill value introduces artificial patterns; the model may not generalize well, so validate on realistic masked data.
Implementation: use a validation set to estimate expected reward and avoid overfitting the fill value to the training set.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Propose a training-time augmentation strategy that masks contiguous row blocks so the model learns to handle partially revealed images. Specify the augmentation policy including probability, region size range, and fill value. Explain how you'd prevent degenerate cases like masking nearly all pixels, and how you'd tune the policy. If you only had two retraining runs, what exact configurations would you try and what metrics would you use to compare them?

Technical Trade-offsA/B Testing & ExperimentationSystem Design

Author's notes

The two-run constraint is what makes this hard.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by proposing a concrete augmentation policy with specific parameters (probability, region size, fill value) and justify each choice. Then discuss safeguards against degenerate cases and a tuning strategy. Finally, outline two retraining configurations and the metrics to compare them, emphasizing controlled experimentation and trade-offs.

Pro tip: Frame the augmentation as a regularizer that simulates real-world occlusion, and highlight that the fill value should be chosen to avoid introducing artificial patterns—zero is often safe but consider dataset mean if normalization is used.

1. Define the augmentation policy

Specify probability (e.g., 0.5), region size range (e.g., 10-30% of image height for contiguous rows), and fill value (e.g., 0 for normalized images or dataset mean). Explain that masking contiguous rows encourages the model to use global context.

2. Prevent degenerate cases

Cap the total masked area (e.g., at most 40% of pixels) by limiting the number of blocks or using a cumulative mask. Also, ensure at least some rows remain unmasked to avoid trivial solutions.

3. Tune the policy

Use a small validation set to sweep probability and region size, monitoring validation accuracy and robustness to masked test images. Start with a coarse grid and refine around the best performing settings.

4. Design two retraining runs

Run A: conservative masking (p=0.3, 10-20% rows). Run B: aggressive masking (p=0.7, 20-40% rows). Keep other hyperparameters fixed to isolate the augmentation effect.

5. Select comparison metrics

Compare validation accuracy on clean data, accuracy on masked test data (with same masking applied), and training stability (loss curves). Also consider inference latency if model changes.

Key Points to Mention

Contiguous row masking simulates occlusion and encourages robustness to partial information.
Fill value should be consistent with data normalization (e.g., zero for zero-centered data).
Degenerate cases: masking too much can destroy signal; enforce a maximum mask ratio.
Tuning: use validation performance on both clean and masked data to balance robustness and accuracy.
Two runs: vary only the augmentation strength to isolate its effect.
Metrics: clean accuracy, masked accuracy, and possibly area under the robustness curve.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

With the model fixed, pixels are revealed one at a time at test time. Design an early-exit policy that decides when to stop revealing pixels in order to maximize expected reward. Propose a concrete strategy, describe how to calibrate it offline, and explain how you'd handle ties or oscillations in the model's predictions.

Algorithms & Data StructuresTechnical Trade-offsSystem Design

Author's notes

Sliding window stability plus confidence threshold was my answer.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame the problem as a sequential decision process where you trade off the cost of revealing more pixels against the expected gain in reward. Propose a threshold-based policy on the model's confidence or expected reward, calibrated offline via simulation to maximize expected reward. Address ties/oscillations with deterministic tie-breaking and hysteresis.

Pro tip: Emphasize that the policy must be calibrated on a validation set that reflects the test distribution, and that you'd monitor for distribution shift to avoid overfitting the threshold.

1. Define the objective and state

Formalize the expected reward as a function of the number of revealed pixels and the model's prediction. Define the state as the current set of revealed pixels and the model's output distribution.

2. Choose a policy class

Select a simple, interpretable policy such as a threshold on the maximum predicted probability or on the expected reward gain. Consider more complex policies like dynamic programming if the state space is manageable.

3. Calibrate offline

Use a validation set to simulate the pixel-revealing process and estimate the expected reward for different policy parameters. Choose the parameter that maximizes the average reward, possibly with cross-validation to avoid overfitting.

4. Handle ties and oscillations

Implement deterministic tie-breaking (e.g., stop at the first occurrence of the maximum confidence) and add hysteresis (e.g., require the confidence to exceed a higher threshold to stop and drop below a lower threshold to continue) to prevent oscillation.

5. Evaluate and iterate

Test the policy on a held-out test set, compare against baselines (e.g., always reveal all pixels), and analyze failure cases. Iterate on the policy or calibration if needed.

Key Points to Mention

Trade-off between computational cost (revealing more pixels) and accuracy/reward.
Use of a validation set to calibrate the stopping threshold, avoiding overfitting.
Threshold on model confidence or expected reward gain as a simple and effective policy.
Deterministic tie-breaking and hysteresis to handle oscillations.
Consideration of distribution shift and the need for periodic recalibration.
Comparison against baselines and analysis of failure modes.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.