Start by clearly defining the network architecture and forward pass equations, then derive the backward pass using the chain rule, and finally implement a numerical gradient check to validate your analytical gradients. Emphasize modularity and vectorization for efficiency, and discuss trade-offs between different activation and loss functions.
Pro tip: When implementing the backward pass, always cache intermediate values from the forward pass to avoid recomputation and reduce errors. Also, use a relative error metric in the gradient check to account for scale differences.
Specify the number of layers, sizes, activation functions (ReLU/sigmoid), and loss function (MSE/cross-entropy). Implement the forward pass using matrix operations, caching intermediate activations for use in backpropagation.
Using the chain rule, derive gradients for weights and biases for each layer, starting from the loss and propagating backwards. Express gradients in terms of cached activations and pre-activation values.
Code the backward pass efficiently, computing gradients layer by layer. Ensure correct handling of activation derivatives (e.g., ReLU derivative is 0 or 1, sigmoid derivative is s*(1-s)).
Implement a numerical gradient checker using finite differences (central difference) to compare against analytical gradients. Use a small epsilon and compute relative error to validate correctness.
Test with different architectures, activations, and losses. Discuss trade-offs such as computational efficiency, numerical stability, and choice of activation/loss combinations.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.