I wrote the code fine but fumbled the explanation of the scaling factor.
Start by writing a clear, vectorized implementation of inverted dropout, then explain the training vs. inference behavior and the rationale for scaling during training. Address the follow-up by describing the consequences of leaving dropout on at test time, emphasizing increased variance and degraded performance.
Pro tip: Mention that inverted dropout ensures the expected value of activations remains unchanged between training and inference, so no scaling is needed at test time—this shows you understand the mathematical justification and practical deployment benefits.
Write a function that during training generates a binary mask with probability keep_prob, multiplies the input by the mask, and divides by keep_prob. During inference, simply return the input unchanged.
During training, dropout randomly zeroes neurons to prevent co-adaptation and force robustness. At inference, no neurons are dropped; the full network is used deterministically.
Dividing by keep_prob during training ensures that the expected sum of activations remains the same as without dropout, so the network's output scale is consistent between training and inference.
If dropout is left on at test time, predictions become stochastic and the expected output is scaled down by keep_prob, leading to inconsistent and typically worse performance. It also increases variance and breaks deterministic inference.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.