I knew the textbook answers: gradient clipping, lower learning rate, maybe batch norm.
Start by acknowledging that exploding gradients are a common issue and then systematically walk through the diagnostic and mitigation strategies, from quick fixes like gradient clipping to architectural changes. Emphasize that the best approach depends on the specific context, such as model architecture and data, and that you would monitor and iterate.
Pro tip: Mention that while gradient clipping is a standard fix, it can mask underlying issues like poor initialization or unstable architecture; therefore, it's crucial to also investigate root causes and consider long-term solutions like normalization or architectural adjustments.
Confirm exploding gradients by monitoring gradient norms during training and checking for sudden increases in loss or NaN values.
Implement gradient clipping (by value or norm) and reduce the learning rate to stabilize training quickly.
Consider using weight initialization techniques like Xavier or He, adding batch normalization, or switching to architectures less prone to exploding gradients (e.g., LSTMs with gradient clipping, or residual connections).
Experiment with learning rate schedules (e.g., warm-up, decay), smaller batch sizes, or gradient accumulation to improve stability.
Continuously track gradient norms and loss curves, and be prepared to combine multiple strategies or revisit data preprocessing if issues persist.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.