Start by clearly defining the network architecture and notation, then walk through the forward pass step-by-step, computing intermediate values. Next, derive the gradients using the chain rule, beginning from the loss and propagating backward layer by layer, explicitly showing the computation of weight and bias gradients. Finally, summarize the update rule and mention any practical considerations.
Pro tip: Use a concrete example with small dimensions (e.g., 2 input features, 2 hidden units) to make the derivation tangible and less error-prone. Emphasize the modularity of backpropagation by breaking it into local gradients that can be reused, which is key for efficient implementation.
Specify the number of layers, units per layer, activation functions (sigmoid/ReLU for hidden, softmax for output), and loss (cross-entropy). Introduce notation for weights, biases, pre-activations, and activations.
Compute the pre-activation and activation for each layer sequentially, starting from the input and ending with the softmax output. Write down the equations for each step.
Derive the gradient of the loss with respect to the output pre-activations (logits). For softmax with cross-entropy, this simplifies to the difference between predicted probabilities and true labels.
Propagate the gradient backward through each hidden layer using the chain rule. Compute the gradient of the loss with respect to pre-activations, then derive weight and bias gradients for each layer.
List the final expressions for weight and bias gradients at each layer. Mention how these are used in gradient descent (e.g., w := w - learning_rate * grad).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clearly defining a small, fully-connected network architecture (e.g., 2-layer MLP) with ReLU activation and MSE loss. Implement the forward pass using NumPy, then derive and code the backward pass manually using the chain rule, storing intermediate values for gradient computation. Finally, replicate the same network in PyTorch, run autograd, and compare gradients numerically using np.allclose with a tight tolerance.
Pro tip: Emphasize the importance of numerical gradient checking (e.g., finite differences) as a sanity check before comparing to PyTorch, and discuss how this builds intuition for debugging custom layers in production.
Choose a simple architecture (e.g., input -> hidden -> output) with ReLU and MSE loss. Implement the forward pass in NumPy, caching intermediate values (pre-activations, activations) needed for backprop.
Apply the chain rule to compute gradients for each parameter and input. Code the backward pass step-by-step, ensuring correct shapes and using cached values.
Recreate the same network in PyTorch with identical weights and inputs. Run forward and backward passes to obtain gradients via autograd.
Use np.allclose to compare manual gradients with PyTorch's. If mismatch, debug by checking individual layer gradients and using numerical gradient checking.
Talk about when manual backprop is useful (e.g., custom ops, debugging) versus autograd, and mention potential pitfalls like numerical stability or broadcasting errors.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining the core mechanics of autograd: how the computational graph is built during the forward pass and how backward() computes gradients via reverse-mode differentiation. Then discuss the practical implications of gradient accumulation and the roles of zero_grad(), detach(), and no_grad() in controlling gradient flow and memory. Finally, tie it together with a concrete example or two to illustrate correct usage in training loops and inference.
Pro tip: Emphasize that forgetting zero_grad() leads to unintended gradient accumulation, which can silently degrade model performance—this is a common pitfall even for experienced engineers. Also, clarify that detach() creates a new tensor that shares data but is detached from the graph, while no_grad() disables gradient tracking entirely for a block of code, which is crucial for memory efficiency during evaluation.
Describe how PyTorch builds a dynamic computational graph during the forward pass, tracking operations on tensors that require gradients. Mention that only leaf tensors with requires_grad=True accumulate gradients.
Explain that calling backward() on a scalar (e.g., loss) triggers backpropagation, computing gradients for all tensors that require gradients. Note that gradients are accumulated into the .grad attribute, not overwritten.
Explain that without zero_grad(), gradients from previous iterations accumulate, leading to incorrect updates. Describe how optimizer.zero_grad() resets gradients before the next backward pass.
Differentiate: detach() returns a new tensor detached from the graph (useful for logging or stopping gradient flow), while no_grad() is a context manager that disables gradient tracking entirely for operations within it (e.g., during evaluation).
Highlight when to use each: always zero gradients in training loops, use detach() to prevent gradients through specific tensors, and use no_grad() for inference to save memory and compute.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I knew these but explained the in-place leaf tensor one poorly.
For each bug, explain the root cause, the typical error or symptom, and the correct fix. Use concrete code examples to illustrate how the bug manifests and how to avoid it. Emphasize best practices like zeroing gradients, avoiding in-place ops on leaf tensors, and managing graph lifetime.
Pro tip: Mention that using `torch.autograd.set_detect_anomaly(True)` can help catch these bugs early, and that understanding the computational graph is key to debugging autograd issues.
Explain that gradients accumulate by default, so not calling `optimizer.zero_grad()` leads to incorrect updates. Show the fix: zero gradients before each backward pass.
Describe how in-place ops on leaf tensors that require grad can corrupt the graph and cause errors. Suggest using non-in-place operations or cloning.
Explain that after `backward()`, the graph is freed by default, so calling backward again raises an error. Mention `retain_graph=True` if multiple backward passes are needed.
Recommend tools like anomaly detection and best practices such as using `with torch.no_grad()` for inference and detaching tensors when appropriate.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining the core idea of finite-difference gradient checking: comparing analytical gradients from backpropagation to numerical approximations. Then describe the symmetric difference formula and its error properties, and finally discuss when to use it (debugging) and when not to (training).
Pro tip: Mention that gradient checking should be done with double precision and on a small subset of parameters, and that it's disabled during training due to cost. Also, note that it can catch bugs like incorrect gradient signs or missing terms.
Explain that gradient checking verifies the correctness of analytical gradients computed by backpropagation by comparing them to numerical approximations.
Detail the central difference formula: (f(θ+ε) - f(θ-ε)) / (2ε), which has O(ε^2) error, and mention the choice of ε (e.g., 1e-4 to 1e-7).
Compute the relative error between analytical and numerical gradients, and use a threshold (e.g., 1e-7) to flag potential bugs.
Use it during debugging of new models or layers, before training, and on a small number of parameters to save computation.
Mention that it's computationally expensive, sensitive to ε, and should be disabled during training; also note issues with non-differentiable points and kinks.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I talked through the variance scaling argument for Xavier and how He accounts for ReLU killing half the activations.
Start by explaining the mathematical root cause: repeated multiplication of gradients through layers leads to exponential decay or growth. Then describe how Xavier and He initialization set initial weights to preserve variance across layers, mitigating these issues. Finally, connect to activation functions and practical implications.
Pro tip: Mention that Xavier is designed for tanh/sigmoid while He is for ReLU, and that modern architectures often use batch normalization or residual connections in addition to proper initialization.
Explain what vanishing and exploding gradients are: during backpropagation, gradients can become extremely small or large, making training unstable or impossible.
Describe how repeated matrix multiplications in deep networks cause gradients to shrink or grow exponentially, depending on weight scale and activation function derivatives.
Introduce Xavier (Glorot) and He initialization as methods that set initial weights to maintain variance of activations and gradients across layers.
Explain that Xavier uses variance 2/(fan_in + fan_out) for tanh/sigmoid, while He uses 2/fan_in for ReLU to account for its zero-half activation.
Conclude that these methods mitigate but don't eliminate the problem; other techniques like batch norm, residual connections, and gradient clipping are also used.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining the core problem both methods address: internal covariate shift and the resulting unstable gradient flow. Then contrast their normalization axes (batch vs. features) and explain how this affects gradient propagation and training stability. Conclude with practical trade-offs and when to use each.
Pro tip: Mention that BatchNorm's batch-dependent statistics introduce noise that can act as a regularizer, but also causes issues with small batches or RNNs, while LayerNorm's per-sample normalization makes it ideal for sequence models and online inference. This shows depth beyond textbook definitions.
Explain internal covariate shift: as parameters update, the distribution of layer inputs changes, forcing later layers to continuously adapt. This slows training and makes gradients unstable.
BatchNorm normalizes each feature across the batch dimension, using batch statistics during training and running averages at inference. It reduces internal covariate shift, allows higher learning rates, and acts as a regularizer.
LayerNorm normalizes each sample across the feature dimension, independent of batch size. It is effective for sequence models and small batches, and provides consistent behavior between training and inference.
Both methods mitigate vanishing/exploding gradients by keeping activations in a stable range. BatchNorm's gradient depends on batch statistics, introducing noise; LayerNorm's gradient is per-sample, offering more stable updates for recurrent and transformer architectures.
BatchNorm works well for CNNs with large batches but fails with small batches or RNNs. LayerNorm is preferred for transformers and RNNs, and is robust to batch size variations. Mention that both can be combined with other techniques like residual connections.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.