← Amazon Interview Insights

Amazon·Machine Learning Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say
Jun 2026Remote

Summary

Technical phone screen for an ML Engineer role at Amazon, focused entirely on LLM internals. Two topics: tokenization design and the SFT training objective. No behavioral questions, no coding, just "explain why it works this way" probing for about an hour.

Questions Asked (6)

Q1

Why do we tokenize text at all, and why do practitioners use subword schemes like BPE or unigram instead of whole-word or character-level tokenization?

Technical Trade-offsSystem Design
Author's notes

This felt like a warm-up but it actually has teeth.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by explaining the fundamental purpose of tokenization: converting raw text into discrete units that models can process. Then compare whole-word, character-level, and subword tokenization in terms of vocabulary size, handling of rare words, and computational efficiency. Conclude by highlighting why subword schemes like BPE and unigram strike a balance, especially for large-scale systems like those at Amazon.

Pro tip: Mention that subword tokenization enables open-vocabulary handling and reduces the number of parameters, which is crucial for deploying models at scale. Also, note that it aligns with Amazon's focus on efficiency and robustness in production systems.

1. Define Tokenization

Explain that tokenization is the process of splitting text into smaller units (tokens) that a model can understand and process.

2. Discuss Whole-Word Tokenization

Highlight its simplicity but note the drawbacks: huge vocabulary, out-of-vocabulary (OOV) issues, and inability to handle morphological variations.

3. Discuss Character-Level Tokenization

Mention its small vocabulary and no OOV, but point out that sequences become very long, making it computationally expensive and harder to capture semantic meaning.

4. Introduce Subword Tokenization

Explain that subword schemes like BPE and unigram find a middle ground by splitting words into frequent subword units, balancing vocabulary size and sequence length.

5. Highlight Practical Benefits

Emphasize that subword tokenization handles rare words, reduces OOV, improves model efficiency, and is widely adopted in state-of-the-art NLP models.

Key Points to Mention

  • Vocabulary size trade-off: whole-word leads to huge vocabularies, character-level to very small but long sequences.
  • Out-of-vocabulary (OOV) handling: subword methods can represent unseen words by combining known subwords.
  • Morphological awareness: subword tokenization captures prefixes, suffixes, and roots, beneficial for morphologically rich languages.
  • Computational efficiency: shorter sequences than character-level, fewer parameters than whole-word.
  • Empirical success: BPE and unigram are used in models like BERT, GPT, and Amazon's own NLP systems.
  • Open-vocabulary support: subword tokenization allows models to handle any text without a fixed vocabulary.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

Could you build a tokenizer using just the 26 letters of the English alphabet? Would it technically work, and what breaks in practice?

Technical Trade-offsAlgorithms & Data Structures
Author's notes

Weird question and I kind of loved it.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying that a tokenizer using only the 26 English letters is technically feasible, but it would be a character-level tokenizer with a very small vocabulary. Then discuss the practical limitations: increased sequence length, loss of subword information, and poor handling of non-English text, numbers, and punctuation. Finally, relate this to real-world trade-offs in ML engineering, such as model efficiency and generalization.

Pro tip: Acknowledge that while such a tokenizer could work for simple, English-only tasks, it would be highly inefficient for modern large-scale models. Emphasize that the choice of tokenizer is a trade-off between vocabulary size, sequence length, and model performance, and that Amazon often deals with diverse, multilingual data.

1. Clarify the question

Confirm that the tokenizer would map each character to a token, resulting in a vocabulary of 26 plus possibly special tokens. State that it is technically possible.

2. Discuss technical feasibility

Explain that it would work for basic tasks: any English text can be encoded as a sequence of letters. Mention that it's a character-level tokenizer, which is simple but has drawbacks.

3. Analyze practical limitations

Highlight issues: longer sequences (increasing computational cost), loss of subword semantics, inability to handle numbers, punctuation, and non-English characters, and poor performance on out-of-vocabulary words.

4. Compare with modern tokenizers

Contrast with subword tokenizers like BPE or WordPiece, which balance vocabulary size and sequence length, and handle multilingual text better.

5. Relate to ML engineering trade-offs

Conclude that while it could technically work, it's impractical for most real-world applications, especially at Amazon where data is diverse and models need to be efficient.

Key Points to Mention

  • Character-level tokenization with a 26-letter vocabulary is possible but leads to very long sequences.
  • Longer sequences increase memory and compute requirements, especially for transformer models.
  • It cannot represent numbers, punctuation, or non-English characters without additional tokens.
  • Subword tokenization (e.g., BPE, WordPiece) is standard because it balances vocabulary size and sequence length.
  • Practical ML systems require handling diverse text, so a 26-letter tokenizer would fail on real-world data.
  • Trade-offs: smaller vocabulary vs. longer sequences; simplicity vs. performance.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

In supervised fine-tuning, can you add a KL divergence term to the loss? If so, what are the two distributions involved, what does the combined objective look like, and why would you want it?

Technical Trade-offsSystem Design
Author's notes

This is where things got serious.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by confirming that yes, a KL divergence term can be added to the supervised fine-tuning loss, typically as a regularizer. Then clearly define the two distributions: the model's predictive distribution and a reference distribution (e.g., the pretrained model or a prior). Finally, present the combined objective and explain the motivation, such as preventing catastrophic forgetting, encouraging smoother outputs, or incorporating a prior.

Pro tip: Mention that the KL term is often weighted by a hyperparameter β, and that tuning β is crucial to balance task performance and regularization. Also, note that in practice, the reference distribution is often the pretrained model's output, which is fixed during fine-tuning.

1. Confirm and clarify

State that yes, a KL divergence term can be added to the supervised fine-tuning loss. Clarify that it acts as a regularizer to keep the fine-tuned model close to a reference distribution.

2. Identify the distributions

Specify the two distributions: the model's predicted distribution p(y|x) and a reference distribution q(y|x), such as the pretrained model's distribution or a uniform prior.

3. Write the combined objective

Present the combined loss: L = L_SFT + β * KL(p || q), where L_SFT is the standard cross-entropy loss, β is a hyperparameter, and KL is the Kullback-Leibler divergence.

4. Explain the rationale

Discuss why: to prevent overfitting, mitigate catastrophic forgetting, encourage the model to stay within a trusted region, or incorporate prior knowledge. Mention trade-offs: too high β may underfit the task.

5. Provide practical considerations

Mention implementation details: the reference distribution is often fixed (e.g., pretrained model), β is tuned via validation, and KL can be computed per example or batch.

Key Points to Mention

  • KL divergence as a regularizer in supervised fine-tuning
  • Two distributions: model's predictive distribution and reference distribution (e.g., pretrained model)
  • Combined loss: L_SFT + β * KL(p || q)
  • Motivations: prevent catastrophic forgetting, overfitting, and maintain generalization
  • Hyperparameter β controls the strength of regularization
  • Reference distribution is typically fixed during fine-tuning

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

Tokenizers often split numbers or code identifiers into multiple subword pieces in ways that hurt arithmetic and code reasoning. Why does this happen and what tokenizer-level choices can help?

Technical Trade-offsAlgorithms & Data Structures
Author's notes

Follow-up that I was not fully ready for.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Explain the root cause: tokenizers are trained on natural language corpora where numbers and code identifiers are rare, so they get fragmented into subwords. Then discuss tokenizer-level choices such as digit-level tokenization, custom vocabularies, and pre-tokenization rules that preserve numeric and identifier integrity.

Pro tip: Mention that while tokenizer changes help, they must be paired with model architecture adjustments (e.g., position encodings) and evaluated on downstream tasks to ensure no regression on general language understanding.

1. Identify the Problem

Explain why tokenizers split numbers and code identifiers: subword algorithms like BPE or WordPiece optimize for frequent character sequences in natural text, where digits and code symbols are underrepresented, leading to fragmentation.

2. Consequences for Reasoning

Describe how fragmentation harms arithmetic and code reasoning: models must learn to reassemble numeric values from pieces, losing positional and magnitude information, and code identifiers lose semantic coherence.

3. Tokenizer-Level Solutions

Propose choices: use digit-level tokenization (each digit as a token), add special tokens for numbers, train tokenizers on domain-specific corpora (code, math), or apply pre-tokenization rules to keep numbers and identifiers intact.

4. Trade-offs and Evaluation

Discuss trade-offs: larger vocabularies, longer sequences, and potential impact on general language performance. Emphasize the need to evaluate on both arithmetic/code tasks and standard NLP benchmarks.

Key Points to Mention

  • Subword tokenization algorithms (BPE, WordPiece, Unigram) are data-driven and biased toward natural language.
  • Numbers and code identifiers are often split into meaningless subwords, e.g., '1234' -> '12', '34'.
  • Digit-level tokenization or number-specific tokens can preserve numeric structure.
  • Domain-specific tokenizer training on code/math corpora improves tokenization of identifiers and numbers.
  • Pre-tokenization rules (e.g., splitting on whitespace and punctuation) can keep numbers and identifiers whole.
  • Trade-offs: increased sequence length, vocabulary size, and potential degradation on general language tasks.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

The KL regularization term requires a second forward pass through a frozen reference model at every training step. How would you make this cheaper, and what do you give up by approximating it?

System DesignTechnical Trade-offs
Author's notes

Short answer from me: you can cache reference logits offline if the reference is truly frozen, or approximate KL with a clipped ratio like PPO does.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging the computational cost of the second forward pass and propose practical optimizations like caching reference logits or using a smaller proxy model. Then discuss the trade-offs of each approximation, emphasizing that the core challenge is balancing fidelity of the KL penalty with training efficiency. Conclude by suggesting a hybrid approach that adapts based on resource constraints.

Pro tip: Quantify the overhead (e.g., 'the reference forward pass can add 30-50% to step time') and mention that in production, you'd monitor KL divergence drift to detect when approximation degrades too much.

1. Identify the bottleneck

Explain that the second forward pass through the frozen reference model doubles the forward computation and increases memory usage, especially for large models.

2. Propose caching strategies

Suggest precomputing and storing reference model outputs for the training dataset, or caching them in a replay buffer if data is reused across epochs.

3. Explore model approximations

Consider distilling the reference model into a smaller proxy or using a low-rank approximation of its outputs to reduce compute per step.

4. Discuss algorithmic alternatives

Mention using a running estimate of the KL term (e.g., exponential moving average) or sampling-based approximations to avoid full forward passes.

5. Analyze trade-offs

Detail what is sacrificed: increased variance in gradient estimates, potential bias in the KL penalty, and reduced regularization effectiveness, which may lead to worse final performance or instability.

Key Points to Mention

  • Caching reference logits for static datasets or when data is revisited
  • Using a smaller distilled model as a proxy for the reference
  • Approximating KL with a running average or sampling
  • Trade-off: reduced accuracy of KL estimate vs. computational savings
  • Impact on training stability and final model quality
  • Memory vs. compute trade-offs in caching

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

After SFT, the model follows instructions well but has clearly forgotten pretraining knowledge. How do you use the KL coefficient, data mixing, or reference checkpoint choice to diagnose and fix this?

Root Cause AnalysisTechnical Trade-offs
Author's notes

Probably the hardest follow-up.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the problem as catastrophic forgetting during SFT, then systematically diagnose using the three levers: KL coefficient (too low allows drift), data mixing (too little pretraining data), and reference checkpoint (wrong base or stale reference). Propose a diagnostic plan that isolates each factor, then a fix that balances instruction following and knowledge retention.

Pro tip: Emphasize that you would first establish a baseline by evaluating the SFT model on a held-out set of pretraining tasks (e.g., cloze, QA) to quantify forgetting, and then use ablation studies to attribute the cause—this shows rigor and avoids premature fixes.

1. Quantify the forgetting

Evaluate the SFT model on a suite of pretraining knowledge benchmarks (e.g., MMLU, TriviaQA, cloze) and compare to the base model to measure the gap. This confirms the issue and sets a baseline for improvements.

2. Diagnose via KL coefficient

Check the KL penalty value used during SFT. If it's too low or zero, the model can drift far from the pretrained distribution; if too high, it may underfit instructions. Run a sweep to find a balance that preserves knowledge while following instructions.

3. Diagnose via data mixing

Inspect the SFT dataset composition. If it contains little to no pretraining data, the model overfits to instruction formats. Experiment with mixing in a small percentage (e.g., 5-20%) of pretraining data or replay examples to retain knowledge.

4. Diagnose via reference checkpoint

Verify the reference model used for KL is the correct pretrained checkpoint (not an earlier SFT model). Also consider using a frozen copy of the base model as reference to anchor the distribution. If the base model itself lacks knowledge, the issue is upstream.

5. Implement and validate fixes

Apply the most promising fix (e.g., increase KL coefficient, add pretraining data, correct reference) and re-evaluate on both instruction-following and knowledge benchmarks. Iterate until both metrics are satisfactory.

Key Points to Mention

  • Catastrophic forgetting is a known issue in sequential fine-tuning; KL regularization helps by penalizing divergence from the reference policy.
  • The KL coefficient controls the trade-off between instruction adherence and knowledge retention; too low leads to forgetting, too high leads to underfitting.
  • Data mixing: incorporating a small fraction of pretraining data (or replay) during SFT can mitigate forgetting without sacrificing instruction-following ability.
  • Reference checkpoint choice: using the original pretrained model (or a frozen copy) as the KL reference is crucial; using an SFT model as reference can propagate forgetting.
  • Diagnostic approach: run controlled ablations—vary KL coefficient, data mix ratio, and reference model—to isolate the primary cause.
  • Evaluation should include both instruction-following metrics (e.g., win rate vs. reference) and knowledge retention metrics (e.g., accuracy on pretraining tasks).

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.