Amazon·Machine Learning Engineer·Technical Phone Screen
- Walk me through LoRA and its variants like QLoRA, DoRA, LoRA+, and AdaLoRA. How do they differ and when would you pick one over another?
- Explain RLHF, DPO, and the basics of PPO as they apply to aligning large language models.
- What is a KV cache, how does it accelerate autoregressive decoding, and what are the memory implications? Also touch on paged attention.
- Compare different positional encoding schemes: absolute, sinusoidal, learned, RoPE, and ALiBi. What are the tradeoffs?
- How does FlashAttention achieve speedups over standard attention? Explain the IO-aware design, tiling, and recomputation.
“This is where I spent most of my prep and it showed.”