I knew the punchline but fumbled the derivation live.
Start by defining cross-entropy and KL divergence mathematically, then derive the relationship step-by-step. Emphasize that cross-entropy equals entropy plus KL divergence, and explain the implications for machine learning.
Pro tip: Connect the derivation to practical ML: since entropy is constant for a fixed true distribution, minimizing cross-entropy is equivalent to minimizing KL divergence, which is why cross-entropy is the standard loss for classification.
Define entropy H(p), cross-entropy H(p,q), and KL divergence D_KL(p||q) for discrete distributions p (true) and q (predicted).
Express cross-entropy as H(p,q) = -∑ p(x) log q(x) and entropy as H(p) = -∑ p(x) log p(x).
Show that H(p,q) = H(p) + D_KL(p||q) by substituting the definition of KL divergence: D_KL(p||q) = ∑ p(x) log(p(x)/q(x)).
Discuss that since H(p) is constant with respect to the model q, minimizing cross-entropy is equivalent to minimizing KL divergence.
Mention that this justifies using cross-entropy loss in classification, where p is the true label distribution (often one-hot) and q is the model's predicted probabilities.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Inverted dropout I could explain fine, scaling activations at train time so you don't touch anything at inference.
Start by explaining the core mismatch: dropout randomly zeroes activations during training, but at inference all neurons are active, causing a scale shift. Then describe the two main fixes: scaling during training (inverted dropout) or scaling during inference (Monte Carlo dropout). Finally, discuss when each is appropriate, emphasizing that inverted dropout is the standard for deterministic inference, while MC dropout is used for uncertainty estimation.
Pro tip: Mention that inverted dropout is the default in frameworks like PyTorch and TensorFlow because it keeps inference efficient and deterministic, but MC dropout is valuable for Bayesian approximation and can be used in production for confidence scores—just be aware of the computational cost.
Describe how dropout randomly deactivates neurons during training, but at inference all neurons are active, leading to a different expected output scale. This mismatch can cause degraded performance if not addressed.
Introduce the two common solutions: (1) scaling activations during training by 1/(1-p) (inverted dropout), and (2) scaling during inference by (1-p) (classic dropout). Mention that inverted dropout is now standard.
Clarify that inverted dropout is used for standard training and deterministic inference, while Monte Carlo dropout keeps dropout active at inference and averages multiple stochastic forward passes to estimate uncertainty.
Explain when to use each: inverted dropout for typical supervised learning where speed and determinism matter; MC dropout for Bayesian deep learning, uncertainty quantification, or when you need confidence intervals.
Mention that MC dropout increases inference time and variance, but provides uncertainty estimates. Inverted dropout is computationally free at inference but doesn't provide uncertainty.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by contrasting the core mechanics of LayerNorm and BatchNorm, emphasizing that LayerNorm normalizes across features per sample while BatchNorm normalizes across the batch. Then, explain how this difference makes LayerNorm inherently suited for sequence models with variable lengths and small batch sizes, and discuss the implications for training stability and distributed training.
Pro tip: Mention that LayerNorm's per-sample normalization avoids cross-sample dependencies, which is crucial for autoregressive generation and handling padding in variable-length sequences. Also, note that BatchNorm's running statistics can be problematic when batch statistics are noisy or when the model is used for inference with different batch sizes.
Briefly define BatchNorm and LayerNorm, highlighting that BatchNorm computes statistics across the batch dimension for each feature, while LayerNorm computes statistics across the feature dimension for each sample.
Explain that in NLP, sequences have variable lengths and are often padded. BatchNorm would compute statistics including padding tokens, leading to biased estimates, whereas LayerNorm treats each sequence independently, ignoring padding.
Note that training large language models often requires micro-batching due to memory constraints, resulting in small per-device batch sizes. BatchNorm's performance degrades with small batches because statistics are noisy, while LayerNorm is unaffected.
Highlight that LayerNorm provides consistent normalization across different batch sizes and devices, simplifying distributed training. BatchNorm requires synchronization of statistics across devices, which can be complex and unstable.
Summarize that LayerNorm is preferred for its robustness to variable lengths, small batches, and distributed settings, making it the standard choice for Transformers and large language models.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Structure your answer by first defining each optimizer's update rule, then comparing their convergence and generalization properties, and finally explaining how you choose based on problem characteristics. Emphasize practical trade-offs and real-world experience.
Pro tip: Mention that Adam often requires less hyperparameter tuning but can generalize worse than SGD with momentum, so many practitioners use Adam for rapid prototyping and switch to SGD with momentum for final training when squeezing out performance.
Clearly state the update equations for SGD, SGD with momentum, and Adam, highlighting the key differences such as the use of momentum and adaptive learning rates.
Explain how each optimizer converges: SGD has slow convergence but good theoretical guarantees; momentum accelerates convergence; Adam converges quickly but may not always reach the best minimum.
Describe how SGD with momentum often generalizes better than Adam due to implicit regularization, while Adam may overfit or find sharper minima.
Outline factors for choosing an optimizer: problem type, dataset size, computational budget, need for fast prototyping, and desired final performance.
Give examples from your experience where you chose one optimizer over another and the outcomes, demonstrating applied knowledge.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Knew the clipping version better than the KL penalty version.
Start by explaining the core problem PPO addresses: policy updates that are too large can cause performance collapse. Then describe how clipping and KL penalties constrain updates, and how this stabilizes training by keeping the new policy close to the old one. Finally, discuss key hyperparameters and common failure modes, tying them to practical tuning insights.
Pro tip: Emphasize that PPO's clipping is a first-order approximation of a trust region, and that the KL penalty is an alternative that can be more stable but requires careful tuning. Mention that in practice, clipping is often preferred for its simplicity and robustness.
Explain that in policy gradient methods, large updates can lead to a bad policy that collects poor data, causing a vicious cycle. Constraining updates ensures monotonic improvement.
Detail how PPO clips the probability ratio between new and old policies to a range [1-ε, 1+ε], removing incentive to move outside this range. This limits the update size.
Explain that instead of clipping, one can add a KL divergence penalty to the objective, with a coefficient β. This penalizes large deviations from the old policy.
Discuss how both methods prevent destructive updates, maintain a trust region, and lead to more stable and reliable learning curves.
List key hyperparameters: clip range ε, KL coefficient β, learning rate, number of epochs per update, and batch size. Discuss failure modes: too large ε or small β leads to instability; too small ε or large β leads to slow learning; improper learning rate can cause divergence.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.