This is where I spent most of my prep and it still felt shaky in the moment.
Start by defining the linear regression model and the mean squared error cost function. Then derive the gradient of the cost with respect to the parameters and show the gradient descent update rule. Finally, compare the iterative gradient descent approach with the closed-form solution in terms of computational complexity, convergence, and practical considerations.
Pro tip: Mention that feature scaling (e.g., standardization) is crucial for gradient descent to converge quickly, while the closed-form solution does not require it. Also, discuss the trade-offs in terms of memory and computation for large datasets.
State the linear regression hypothesis: h_θ(x) = θ^T x (or θ0 + θ1*x for simple case). Define the cost function as J(θ) = (1/2m) * Σ (h_θ(x^(i)) - y^(i))^2, explaining the 1/2 for convenience in derivation.
Compute the partial derivative of J(θ) with respect to each parameter θ_j. Show that ∂J/∂θ_j = (1/m) * Σ (h_θ(x^(i)) - y^(i)) * x_j^(i). This is the key step to get the update rule.
Write the simultaneous update rule: θ_j := θ_j - α * ∂J/∂θ_j for all j. Emphasize that all parameters must be updated simultaneously and discuss the role of the learning rate α.
Derive or state the normal equation: θ = (X^T X)^{-1} X^T y. Mention that it directly minimizes the cost without iteration, but requires matrix inversion which is O(n^3) for n features.
Discuss trade-offs: gradient descent scales better to large datasets (especially with stochastic or mini-batch variants), while the closed-form is exact but computationally expensive for many features. Also mention that gradient descent requires tuning learning rate and feature scaling, whereas closed-form does not.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying the problem scope (e.g., distance metric, k value, data size) and then outline the core components: data structures for storing training data, the fit method (which is essentially a no-op), and the predict method that computes distances and finds nearest neighbors. Discuss time complexity for both training and prediction, and mention potential optimizations like KD-trees or ball trees for low-dimensional data.
Pro tip: Emphasize the trade-off between simplicity and efficiency: brute-force KNN is O(n*d) per query, but for Waymo's real-time systems, you'd likely need approximate nearest neighbor methods or spatial indexing. Showing awareness of scalability and production constraints sets you apart.
Ask about dataset size, dimensionality, distance metric, and whether approximate results are acceptable. This shows you consider practical constraints before diving into implementation.
Store training data in a simple array or list for brute-force, or use a KD-tree/ball tree for efficient nearest neighbor search. Mention that the choice depends on dimensionality and dataset size.
Fit just stores the training data and labels. Predict computes distances from the query point to all training points, selects the k smallest, and returns the majority label (or weighted vote).
Training is O(1) (or O(n) to store data). Prediction is O(n*d) for brute-force, where n is number of training samples and d is dimensionality. With KD-tree, average query is O(log n) for low d, but degrades to O(n) in high dimensions.
Mention vectorization (e.g., using NumPy), parallelization, approximate methods (LSH, HNSW), and the curse of dimensionality. Relate to real-world constraints like latency and memory.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Blanked briefly on the entropy formula and had to reconstruct it from scratch mid-explanation.
Start by defining entropy and information gain, then walk through the algorithm step-by-step: compute the dataset's entropy, evaluate the information gain for each feature, and select the feature with the highest gain. Conclude by explaining how this process recurses to build the tree, and mention practical considerations like handling continuous features and avoiding overfitting.
Pro tip: Emphasize that information gain is biased toward features with many distinct values, and mention that alternatives like gain ratio or Gini impurity are often used in practice—this shows you understand real-world trade-offs beyond textbook definitions.
Explain entropy as a measure of impurity or uncertainty in a dataset, and information gain as the expected reduction in entropy after splitting on a feature. Use the formulas: Entropy(S) = -Σ p_i log2(p_i), and InformationGain(S, A) = Entropy(S) - Σ (|S_v|/|S|) * Entropy(S_v).
Calculate the entropy of the target variable for the current node using the class distribution. This represents the baseline uncertainty before any split.
For each feature, partition the data by its values, compute the weighted average entropy of the subsets, and subtract from the original entropy to get the information gain. For continuous features, consider threshold splits.
Choose the feature with the highest information gain as the splitting criterion. Create child nodes for each value (or threshold) of that feature and assign the partitioned data accordingly.
Recursively apply the same process to each child node until a stopping condition is met, such as pure nodes, maximum depth, or minimum samples per leaf. Optionally, prune the tree to reduce overfitting.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The Laplace smoothing question caught me a little flat-footed.
Start by clarifying the problem scope (e.g., text classification, feature types) and then outline the Naive Bayes algorithm, focusing on the data structures for storing class priors and conditional probabilities. Walk through the fit and predict methods step-by-step, and explicitly address zero probabilities with smoothing techniques like Laplace smoothing. Emphasize computational efficiency and trade-offs, especially for large-scale or real-time applications relevant to autonomous driving.
Pro tip: Mention that in practice, you'd use log probabilities to avoid underflow and that Laplace smoothing is a special case of Dirichlet priors, showing deeper statistical understanding. Also, relate the choice of smoothing parameter to the specific domain (e.g., sensor data) to demonstrate domain awareness.
Ask about the type of features (categorical, continuous), dataset size, and whether online learning is needed. State assumptions like feature independence and choice of likelihood model (e.g., multinomial for text, Gaussian for continuous).
Propose storing class priors as a dictionary or array, and conditional probabilities as a nested dictionary (class -> feature -> value -> probability) or a 2D array for each feature. For efficiency, consider sparse representations if features are high-dimensional.
Compute class priors by counting class occurrences. For each feature and class, estimate conditional probabilities using maximum likelihood estimation, applying smoothing (e.g., Laplace) to avoid zeros. Use log probabilities to prevent underflow.
For a new sample, compute the log posterior for each class by summing log priors and log conditional probabilities. Return the class with the highest log posterior. Optionally, provide probability estimates via softmax.
Explain Laplace smoothing (add-one) or Lidstone smoothing (add-alpha) to handle unseen feature values. Discuss trade-offs: smoothing parameter selection, bias-variance, and computational cost vs. accuracy. Mention alternative approaches like using a small epsilon.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.