Start by clearly defining the network architecture and notation, then systematically derive the forward pass equations and the backward pass gradients using the chain rule. Finally, implement the computations in NumPy with vectorized operations, ensuring dimensional consistency and numerical stability.
Pro tip: Emphasize numerical stability by using the log-sum-exp trick for softmax cross-entropy and avoid explicit one-hot encoding by using integer labels. Also, mention that you would verify gradients using finite differences to catch implementation errors.
Specify the two-layer MLP: input dimension D, hidden dimension H, output dimension C. Define weight matrices W1 (D x H), W2 (H x C), biases b1 (H), b2 (C). Use ReLU activation for hidden layer and softmax for output.
Write equations for the forward pass: Z1 = X W1 + b1, A1 = ReLU(Z1), Z2 = A1 W2 + b2, and probabilities P = softmax(Z2). For a batch of N samples, X is N x D, Z1 is N x H, etc.
Derive the cross-entropy loss for a batch and compute gradients analytically. Start with dZ2 = (P - Y)/N, then dW2 = A1^T dZ2, db2 = sum(dZ2, axis=0). Backpropagate: dA1 = dZ2 W2^T, dZ1 = dA1 * (Z1 > 0), dW1 = X^T dZ1, db1 = sum(dZ1, axis=0).
Implement the forward and backward passes using NumPy operations, ensuring vectorization over the batch. Use np.max for numerical stability in softmax and avoid loops where possible.
Mention gradient checking with finite differences, handling of ReLU derivative at zero, and ensuring shapes match. Also discuss initialization (e.g., He initialization) and potential overfitting.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by outlining a systematic verification process: set up the same input tensors with requires_grad=True, compute gradients via both your manual implementation and PyTorch's autograd, then compare them numerically. Emphasize using torch.allclose with appropriate tolerances and investigating any discrepancies by checking for common pitfalls like non-differentiable operations or incorrect gradient formulas.
Pro tip: Use double precision (float64) for gradient checking to reduce numerical noise, and always verify with a small, controlled input first to isolate issues before scaling up.
Create input tensors and parameters with requires_grad=True, ensuring both manual and autograd computations use the exact same values and data types.
Run your manual gradient computation and PyTorch's autograd (using backward() or torch.autograd.grad) to obtain gradients for all relevant tensors.
Use torch.allclose with a small tolerance (e.g., 1e-5 for float32, 1e-8 for float64) to check for closeness, and also compute max absolute difference to quantify discrepancies.
If mismatches occur, check for common issues: incorrect gradient formulas, non-differentiable operations, in-place modifications, or numerical instability; debug by simplifying the computation or using gradcheck.
For a more rigorous check, use torch.autograd.gradcheck on your manual function to compare against numerical Jacobian, ensuring correctness for small inputs.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clearly defining the toy dataset and model, then implement SGD step-by-step, explaining the gradient computation and parameter update. Run a few iterations, tracking the loss to demonstrate convergence, and discuss potential improvements or pitfalls.
Pro tip: Emphasize the importance of setting a appropriate learning rate and monitoring loss; mention that in practice, you'd use mini-batches and learning rate schedules, but for this toy example, full-batch SGD suffices.
Specify a simple model (e.g., linear regression) and a small synthetic dataset (e.g., 10 points). State the loss function (e.g., mean squared error).
Write code to compute the gradient of the loss w.r.t. parameters and update them using the SGD rule: param = param - lr * grad.
Loop for a fixed number of iterations (e.g., 10), computing loss and updating parameters each time. Print or log the loss to observe progress.
Comment on whether the loss decreased, the effect of learning rate, and any convergence issues. Mention extensions like momentum or adaptive methods.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I talked about summing or averaging gradients across the batch dimension and making sure matrix shapes broadcast correctly.
Start by explaining how to extend the forward pass to handle a batch dimension, then detail the necessary adjustments in the backward pass to compute gradients over the batch. Emphasize the use of matrix operations and the importance of averaging or summing gradients appropriately.
Pro tip: Mention that while summing gradients over a batch is mathematically correct, most frameworks average them to keep the learning rate independent of batch size. This shows awareness of practical implementation details.
Describe how inputs become a matrix of shape (batch_size, input_dim), and weights are applied via matrix multiplication to produce outputs of shape (batch_size, output_dim).
Explain that gradients for weights and biases are computed by summing or averaging over the batch dimension, using matrix operations like the transpose of the input.
Note that activation functions are applied element-wise, so their derivatives are also element-wise and must be multiplied with the upstream gradient.
Discuss whether to sum or average gradients across the batch, and how this choice affects the learning rate and optimization.
Mention vectorization for efficiency, avoiding loops, and ensuring correct broadcasting for bias addition.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Blanked for a second on the exact name but knew the fix: subtract the row max before exponentiating.
Start by explaining the numerical instability in softmax and cross-entropy when dealing with large logits, then introduce the log-sum-exp trick as a solution. Show how it stabilizes computations by factoring out the maximum value, and discuss its application in cross-entropy loss. Emphasize the practical importance in training deep neural networks.
Pro tip: Mention that the log-sum-exp trick is not just for softmax but also used in other areas like mixture models and attention mechanisms, and that frameworks like PyTorch implement it internally for stability.
Define softmax as converting logits to probabilities and cross-entropy as the negative log-likelihood of the true class. Highlight that both involve exponentials and logarithms.
Describe how large logits cause overflow in exp (e.g., exp(1000) = inf) and underflow in log (log(0) = -inf), leading to NaN or incorrect gradients.
Show that softmax(x) = softmax(x - c) for any constant c, and choosing c = max(x) prevents overflow. Similarly, log-sum-exp(x) = c + log(sum(exp(x - c))).
Demonstrate how to compute cross-entropy loss stably using log-sum-exp: loss = -x_y + logsumexp(x), avoiding explicit softmax and log.
Mention that this trick is standard in deep learning frameworks and crucial for training stability, especially with large logits or many classes.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Backprop requires storing intermediate activations for the backward pass, so memory scales with depth.
Start by defining the memory and compute costs of standard backpropagation, then explain how gradient checkpointing trades increased compute for reduced memory. Conclude with criteria for when this trade-off is beneficial, such as limited GPU memory or very deep models.
Pro tip: Quantify the trade-off: gradient checkpointing typically increases compute by about 30% but can reduce memory usage by up to the square root of the number of layers, enabling larger batch sizes or models. Mention that it's especially useful when memory is the bottleneck, not compute.
Describe how backpropagation stores intermediate activations for all layers, leading to O(n) memory where n is the number of layers, and O(n) compute for the backward pass.
Define gradient checkpointing as a technique that recomputes activations during the backward pass instead of storing them, reducing memory to O(sqrt(n)) at the cost of extra forward passes.
Discuss the memory-compute trade-off: memory savings allow larger models or batch sizes, but compute increases due to recomputation, typically by 20-40%.
Specify scenarios where gradient checkpointing makes sense: when memory is the limiting factor, for very deep networks (e.g., transformers), or when training on hardware with limited GPU memory.
Summarize that the decision depends on the specific bottleneck and mention alternatives like mixed precision or model parallelism.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.