Amazon·Software Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say

May 2026

Summary

Applied Scientist interview at Amazon with a tricky neural network theory question that apparently almost nobody gets right. Short post but the question itself is genuinely interesting.

Questions Asked (1)

You have two two-layer neural networks: one with very few neurons per layer, one with many neurons per layer. Which one is more likely to get stuck in local minima during training?

Technical Trade-offsAlgorithms & Data Structures

Author's notes

The interviewer mentioned fewer than 2 out of 100 candidates answer this correctly, which made me feel pretty good about getting it.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying that the question is about the likelihood of getting stuck in local minima, not about which network is better. Then explain that the network with very few neurons per layer is more prone to local minima due to its limited capacity and simpler loss landscape, while the larger network has more parameters and a higher-dimensional loss landscape with many saddle points but fewer bad local minima. Conclude by noting that in practice, local minima are less of a problem in deep learning, and saddle points are the main challenge.

Pro tip: Mention that modern deep learning rarely suffers from local minima due to overparameterization, and that smaller networks are more likely to get stuck in local minima because they have less capacity to escape. This shows you understand both theory and practical realities.

1. Clarify the question

Restate the question to ensure you understand it: 'You're asking which network is more likely to get stuck in local minima during training, not which one performs better.'

2. Define local minima and saddle points

Briefly explain that local minima are points where the loss is lower than all nearby points but not the global minimum, while saddle points are points where the gradient is zero but not a local minimum.

3. Analyze the small network

Explain that a network with very few neurons per layer has fewer parameters, leading to a simpler loss landscape with more local minima. Its limited capacity makes it harder to escape local minima.

4. Analyze the large network

Explain that a network with many neurons per layer has more parameters, resulting in a high-dimensional loss landscape with many saddle points but fewer bad local minima. Overparameterization tends to smooth the landscape, making local minima less problematic.

5. Conclude and mention practical implications

Conclude that the small network is more likely to get stuck in local minima. Add that in practice, local minima are not a major issue in deep learning; saddle points and plateaus are more common challenges.

Key Points to Mention

Local minima vs. saddle points: In high-dimensional spaces, saddle points are more common than local minima.
Overparameterization: Large networks have many parameters, which tends to eliminate bad local minima and create a smoother loss landscape.
Capacity and expressiveness: Small networks have limited capacity, making them more prone to getting stuck in local minima.
Gradient descent dynamics: Small networks may have fewer directions to escape local minima, while large networks have more paths to explore.
Practical deep learning: Modern deep learning rarely suffers from local minima; techniques like stochastic gradient descent and batch normalization help escape them.
Empirical evidence: Studies show that larger networks often generalize better and are less likely to get stuck in poor local minima.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.