← Applied intuition Interview Insights

Applied intuition·Machine Learning Engineer·Take-home Assignment·Senior

Senior
Jul 2026

Summary

Applied Intuition ML engineer interview centered on a take-home Colab notebook with a broken GPT-style model. The task was to find and fix three specific bugs, write tests, and explain the reasoning. Pretty hands-on, no fluff.

Questions Asked (1)

Q1

Given a minimal GPT-style language model in a Colab notebook, identify and fix three bugs: a broken attention mask, a training loop error, and missing positional encoding. Also write unit tests that would have caught each bug and explain your reasoning.

System DesignTechnical Trade-offsAlgorithms & Data Structures
Author's notes

This took me longer than I expected on the attention masking piece.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by outlining a systematic debugging process: reproduce the issue, isolate each bug, and verify fixes with targeted unit tests. For each bug, explain the root cause, the fix, and the test that would catch it, emphasizing how the tests validate the fix and prevent regression.

Pro tip: Demonstrate maturity by discussing how you would prioritize bugs based on their impact on model training and how you would structure tests to be fast and deterministic, avoiding flaky tests due to randomness.

1. Reproduce and Isolate

Run the notebook to observe symptoms (e.g., loss not decreasing, NaN outputs). Use print statements or a debugger to isolate each bug by examining intermediate tensors and shapes.

2. Fix Attention Mask

Identify if the mask is incorrectly applied (e.g., not masking future tokens or wrong shape). Fix by ensuring the mask is a lower triangular matrix of correct shape and applied before softmax.

3. Fix Training Loop

Check for common errors like missing optimizer.zero_grad(), incorrect loss calculation, or not detaching hidden states. Fix by following standard training loop structure.

4. Add Positional Encoding

Implement sinusoidal or learned positional encodings and add them to token embeddings before the transformer blocks.

5. Write Unit Tests

For each bug, write a test that would fail before the fix and pass after. Use assertions on shapes, values, and behavior (e.g., mask prevents attention to future tokens).

Key Points to Mention

  • Attention mask: ensure it's a lower triangular matrix with -inf for masked positions, applied before softmax.
  • Training loop: verify optimizer.zero_grad(), loss.backward(), optimizer.step() order, and that loss is computed correctly.
  • Positional encoding: add to embeddings, either fixed sinusoidal or learned, to provide sequence order information.
  • Unit tests: test mask shape and values, test that training loop reduces loss on a simple task, test that positional encoding changes outputs for different positions.
  • Debugging strategy: use small synthetic data, check tensor shapes, and validate each component independently.
  • Trade-offs: discuss fixed vs learned positional encodings, and the importance of deterministic tests.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.