This one took me a minute to get grounded.
Explain that 1-NN can be seen as a neural network with a linear layer that computes negative squared distances between the input and each training point, followed by a softmax with a very low temperature to approximate a hard nearest-neighbor selection. Then discuss how the softmax output can be interpreted as a probability distribution over classes, and how the temperature parameter controls the sharpness of the decision.
Pro tip: Mention that while this construction is theoretically valid, it is not practical for large datasets due to the O(N) memory and compute cost; however, it illustrates the connection between non-parametric methods and neural networks, which is useful for understanding modern architectures like prototypical networks.
Set the weight matrix W to be the training data points (each row is a training example) and the bias to be related to the squared norm of the training points. This allows the linear layer to compute inner products between the input and each training point.
Use the identity ||x - x_i||^2 = ||x||^2 - 2 x·x_i + ||x_i||^2 to express the negative squared distance as a linear operation on x plus a bias term. The linear layer output can be set to -||x - x_i||^2 by appropriate scaling and bias.
Apply a softmax function to the negative distances divided by a temperature parameter T. As T approaches 0, the softmax output approaches a one-hot vector indicating the nearest neighbor.
Aggregate the softmax outputs by class labels (e.g., sum probabilities for training points of the same class) to obtain class probabilities. With T→0, this yields the 1-NN prediction.
Note that this construction requires storing all training points and is computationally expensive for large N. Mention that it can be extended to k-NN by using a different aggregation, and that it relates to metric learning and prototypical networks.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.