Start by clearly defining the network architecture (input size, hidden size, output size) and the forward pass equations. Then derive the backward pass using the chain rule, explicitly showing gradients for each parameter. Finally, implement SGD updates and train on a toy dataset, printing the loss over epochs to demonstrate convergence.
Pro tip: Vectorize operations across the batch to avoid loops, and use gradient checking to verify your backprop implementation. This shows production-level coding and rigor.
Specify layer sizes and activation functions (e.g., ReLU for hidden, softmax for output). Write the forward pass equations: Z1 = XW1 + b1, A1 = ReLU(Z1), Z2 = A1W2 + b2, A2 = softmax(Z2).
Compute gradients: dZ2 = A2 - Y, dW2 = A1.T @ dZ2, db2 = sum(dZ2), dA1 = dZ2 @ W2.T, dZ1 = dA1 * ReLU'(Z1), dW1 = X.T @ dZ1, db1 = sum(dZ1). Explain each step.
Update parameters: W1 -= lr * dW1, b1 -= lr * db1, W2 -= lr * dW2, b2 -= lr * db2. Choose a learning rate and iterate over epochs.
Generate or load a simple dataset (e.g., XOR or linearly separable). Train for several epochs, printing loss every few epochs to show it decreasing.
Perform gradient checking to ensure correctness. Discuss trade-offs: batch vs stochastic, activation choices, learning rate tuning, and potential overfitting.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I knew the zero-init answer cold, symmetry breaking and all that.
Start by explaining the symmetry problem: if all weights are zero, every neuron in a layer computes the same output and receives the same gradient, so the network cannot break symmetry and learn diverse features. Then introduce Xavier initialization as a way to set weights so that the variance of activations and gradients remains constant across layers, preventing vanishing/exploding signals. Finally, connect this to practical benefits like faster convergence and better performance in deep networks.
Pro tip: Mention that Xavier initialization assumes linear activations and works best with tanh or sigmoid, while He initialization is preferred for ReLU—showing awareness of when Xavier might not be optimal. Also note that modern frameworks like PyTorch default to Kaiming/He for ReLU, so you'd choose Xavier deliberately for saturating activations.
Describe how setting all weights to zero causes all neurons to be identical, leading to symmetric gradients and no learning. Emphasize that this symmetry cannot be broken by training alone.
Explain that random initialization with too small or too large variance leads to vanishing or exploding activations/gradients, making deep networks hard to train.
State that Xavier (Glorot) initialization sets weights with variance 2/(fan_in + fan_out), derived to keep the variance of activations and back-propagated gradients constant across layers.
Highlight that Xavier enables faster convergence, reduces the need for careful hyperparameter tuning, and allows training of deeper networks without batch normalization.
Mention that Xavier assumes linear activations and is less effective for ReLU, where He initialization (variance 2/fan_in) is preferred. This shows depth of understanding.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining the vanishing gradient problem and how it arises in deep networks with sigmoid/tanh activations. Then explain how ReLU's derivative (0 or 1) avoids the multiplicative decay of gradients, and contrast with sigmoid/tanh whose derivatives are always <1. Conclude by noting that while ReLU mitigates vanishing gradients, it introduces other trade-offs like dying ReLU, showing balanced understanding.
Pro tip: Mention that ReLU's constant gradient for positive inputs also enables faster training and that variants like Leaky ReLU address the dying ReLU problem, demonstrating awareness of practical deployment considerations.
Explain that in deep networks, gradients are multiplied through layers during backpropagation. If each layer's gradient is small, the product shrinks exponentially, making early layers learn very slowly.
Sigmoid derivative max is 0.25, tanh derivative max is 1 but typically <1. In deep networks, repeated multiplication of these small values causes gradients to vanish.
ReLU's derivative is 1 for positive inputs and 0 for negative inputs. For active neurons, the gradient is preserved without decay, allowing gradients to flow through many layers.
Highlight that ReLU does not saturate for positive inputs, unlike sigmoid/tanh. However, note that ReLU can suffer from dying neurons (zero gradient for negative inputs) and mention variants like Leaky ReLU.
Conclude that ReLU enables training of deeper networks by mitigating vanishing gradients, leading to faster convergence and better performance in practice.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Explain the finite difference method for gradient checking, emphasizing the central difference formula and its O(ε^2) accuracy. Describe a practical implementation that compares analytical and numerical gradients, using relative error and tolerances to flag discrepancies. Highlight the importance of testing on simple functions first and then scaling to complex models.
Pro tip: Use a relative error metric with a threshold like 1e-7 for float64, and always disable dropout or other stochastic components during gradient checking to avoid false positives.
Explain that finite differences approximate gradients by perturbing each input dimension and computing the change in output. Use the central difference formula for better accuracy: (f(x+ε) - f(x-ε)) / (2ε).
For each parameter, compute the analytical gradient (from your code) and the numerical gradient using finite differences. Compare them using a relative error metric: ||g_analytical - g_numerical|| / (||g_analytical|| + ||g_numerical|| + ε).
Select a small ε (e.g., 1e-5) to balance truncation and round-off errors. Set a tolerance (e.g., 1e-7) for relative error; if the error exceeds it, investigate potential bugs.
Validate the gradient checking implementation on simple functions with known gradients (e.g., linear, quadratic) to ensure correctness before applying to complex models.
Use gradient checking on your actual model, but be mindful of computational cost. Check a subset of parameters or use random projections if the parameter space is large.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.