Walmart Labs·Data Scientist·Technical Phone Screen·Intermediate

Intermediate

May 2026

Summary

Interviewed for a Data Scientist role at Walmart Labs and got a technical question about LLM inference that I thought I was prepared for until I actually had to explain it out loud.

Questions Asked (1)

During LLM inference, the same prompt can produce different outputs each time. Where does that randomness come from? Walk through the role of temperature, top-k and top-p sampling, and how greedy decoding fits into the picture.

Technical Trade-offsSystem Design

Author's notes

I knew the surface-level answer but fumbled when they pushed on the interaction between temperature and top-p.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by explaining that randomness in LLM inference comes from the sampling strategy applied to the model's output logits. Then describe how temperature, top-k, and top-p modify the probability distribution, and contrast with greedy decoding which is deterministic. Use a concrete example to illustrate the effect of each parameter.

Pro tip: Mention that in production systems like Walmart Labs, you often need to balance creativity and consistency; for example, using temperature=0 for deterministic outputs in A/B tests, or top-p for diverse recommendations. Also note that even with greedy decoding, floating-point non-determinism can cause slight variations across runs.

1. Source of Randomness

Explain that LLMs output a probability distribution over the vocabulary at each step. Randomness is introduced when we sample from this distribution instead of always picking the most likely token.

2. Temperature Scaling

Describe how temperature divides the logits before softmax. Lower temperature (e.g., 0.1) makes the distribution sharper (more deterministic), while higher temperature (e.g., 1.5) flattens it, increasing diversity.

3. Top-k and Top-p Sampling

Explain that top-k restricts sampling to the k most likely tokens, while top-p (nucleus) sampling restricts to the smallest set of tokens whose cumulative probability exceeds p. Both truncate the distribution to avoid unlikely tokens.

4. Greedy Decoding

Contrast with greedy decoding, which always selects the token with the highest probability (argmax). This is deterministic (temperature effectively 0) and produces the same output for the same input, but can lead to repetitive or bland text.

5. Trade-offs and Use Cases

Discuss when to use each method: greedy for tasks requiring consistency (e.g., factual QA), temperature for creative tasks, top-k/top-p for balancing coherence and diversity. Mention that these can be combined (e.g., temperature with top-p).

Key Points to Mention

Logits and softmax: the model outputs raw scores (logits) that are converted to probabilities via softmax.
Temperature: scales logits before softmax; T→0 approximates greedy, T>1 increases randomness.
Top-k: truncates to k highest-probability tokens, then renormalizes and samples.
Top-p (nucleus): truncates to smallest set of tokens with cumulative probability ≥ p, then samples.
Greedy decoding: always picks argmax token; deterministic but can be suboptimal.
Practical implications: reproducibility, creativity vs. consistency, and computational efficiency.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.