Start by clearly defining the network architecture (e.g., input size, hidden layers, activations) and the loss function. Then implement each component in NumPy: forward pass, loss, backward pass, and parameter update, ensuring vectorized operations for efficiency. Finally, test with a simple dataset and verify gradients numerically.
Pro tip: Emphasize vectorization and numerical stability (e.g., using log-sum-exp for softmax) to demonstrate production-level awareness. Also, mention that you would verify gradients with finite differences to catch bugs early.
Specify the number of layers, neurons per layer, activation functions (e.g., ReLU, sigmoid), and weight initialization (e.g., Xavier/He). Initialize weights and biases randomly.
Compute layer outputs by applying linear transformations followed by activations. Use vectorized operations for batch processing.
Calculate the loss (e.g., cross-entropy for classification, MSE for regression) between predictions and true labels. Ensure numerical stability.
Derive gradients of the loss with respect to weights and biases using the chain rule. Propagate errors backward through the network.
Apply gradient descent (or variant) to update weights and biases. Repeat for multiple epochs, monitoring loss.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
They asked this as a follow-up and I was glad I knew it.
Start by explaining the numerical instability of the naive softmax due to overflow/underflow, then introduce the max-subtraction trick and the log-sum-exp trick as solutions. Emphasize the mathematical equivalence and practical implementation, and discuss trade-offs like computational overhead and stability guarantees.
Pro tip: Mention that in practice, frameworks like PyTorch and TensorFlow already implement these tricks internally, but understanding them is crucial for custom implementations and debugging. Also, note that the log-sum-exp trick is used in many other contexts, such as computing log probabilities and in the forward algorithm for HMMs.
Explain that softmax involves exponentiating large values, which can overflow to infinity, and small values, which can underflow to zero, leading to NaN or incorrect results.
Describe how subtracting the maximum value from the input vector before exponentiation prevents overflow, as the largest exponent becomes 0, and the results are mathematically equivalent.
State that log-sum-exp computes log(sum(exp(x_i))) stably by factoring out the maximum: log(sum(exp(x_i - m))) + m, where m is the maximum of x_i.
Show how softmax can be computed using the max-subtraction trick, and how log-softmax (often used in loss functions) directly uses log-sum-exp for stability.
Mention that while these tricks add a small computational cost (finding the max), they are essential for stability. Also, note that many libraries implement them, but understanding is key for custom code.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by contrasting single-sample and mini-batch tensor shapes, then explain how broadcasting rules dictate the placement of batch dimensions and the handling of operations like normalization and loss. Emphasize that consistent shape conventions and explicit broadcasting prevent subtle bugs and enable efficient vectorized computation.
Pro tip: Mention that broadcasting can silently introduce unintended behavior (e.g., when a bias term is added to the wrong dimension), so always validate shapes with assertions or unit tests on small examples before scaling up.
Establish a clear convention for batch dimensions (e.g., batch-first vs. batch-last) and ensure all tensors adhere to it. This avoids confusion when extending from single sample to mini-batch.
Explain how broadcasting aligns dimensions from the right and how to insert singleton dimensions (e.g., using None or unsqueeze) to make operations compatible across batch and feature dimensions.
Discuss how common operations (e.g., matrix multiplication, normalization, loss computation) change when a batch dimension is added, and how to adjust reductions (e.g., mean over batch) accordingly.
Describe using assertions or shape-printing to catch broadcasting errors early, and testing with small dummy batches to ensure correctness before full-scale training.
Highlight how proper broadcasting enables vectorized operations that leverage hardware acceleration, and discuss trade-offs like memory usage when batch size increases.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.