← Microsoft Interview Insights

Microsoft·Software Engineer·Technical Phone Screen·Senior

Senior
May 2026

Summary

Microsoft SWE interview that went deep on language model decoding. One question, but it had real teeth: implement greedy decoding and beam search from scratch given a next-token log-prob API. Not a leetcode grind, more of an applied algorithms design problem.

Questions Asked (1)

Q1

Given a function that returns log-probabilities for the next token given a prefix, implement both greedy decoding and beam search. Greedy should pick the highest log-prob token at each step and stop at EOS or max_len. Beam search should maintain the top-k candidate sequences by cumulative log-probability, expanding all beams each step and pruning back to k. Discuss edge cases like early-finished beams.

Algorithms & Data StructuresTechnical Trade-offsSystem Design
Author's notes

This one took me a minute to even parse.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the function signature and constraints, then implement greedy decoding as a simple loop, followed by beam search with a priority queue or sorted list to maintain top-k beams. Discuss edge cases like early-finished beams and how to handle them, and analyze time/space complexity.

Pro tip: Mention that beam search can be optimized by only expanding beams that haven't finished, and that finished beams should be set aside but still considered for final output if they have high scores. Also, note that using log-probabilities avoids underflow and that cumulative log-prob is a monotonic sum.

1. Clarify the problem and constraints

Ask about the function signature, whether it returns a single log-prob or a distribution, and the expected input/output format. Confirm max_len and k values.

2. Implement greedy decoding

Write a loop that at each step picks the token with the highest log-prob, appends it to the sequence, and stops if EOS or max_len is reached. Return the sequence.

3. Implement beam search

Initialize beams with the start prefix. At each step, expand each beam by considering all possible next tokens (or top candidates), compute cumulative log-probs, and keep the top-k sequences. Stop when all beams have finished or max_len is reached.

4. Handle edge cases

Address early-finished beams by either removing them from active beams but keeping them in a separate list for final selection, or by allowing them to continue with a padding token. Also handle ties, empty prefixes, and k larger than vocabulary.

5. Analyze complexity and trade-offs

Discuss time complexity: greedy is O(max_len * vocab_size), beam search is O(max_len * k * vocab_size) if expanding all tokens. Mention memory and potential optimizations like pruning low-probability tokens.

Key Points to Mention

  • Use log-probabilities to avoid numerical underflow and because they sum instead of multiply.
  • Greedy decoding is a special case of beam search with k=1.
  • Beam search maintains top-k sequences by cumulative log-prob, expanding all beams each step and pruning.
  • Early-finished beams should be stored separately and considered for final output; they should not be expanded further.
  • Time complexity: greedy O(max_len * V), beam search O(max_len * k * V) where V is vocabulary size.
  • Edge cases: EOS token handling, max_len cutoff, ties in scores, and k > V.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.