Define overfitting and underfitting clearly, then explain how to identify each through training/validation performance patterns and learning curves. Emphasize that the goal is to find the right balance between bias and variance, and mention practical techniques to address each issue.
Pro tip: Demonstrate maturity by discussing the bias-variance trade-off and how it relates to model complexity, and mention that in practice, you often start with a simple model and gradually increase complexity while monitoring validation performance.
Clearly state that overfitting occurs when a model learns noise in the training data and performs poorly on new data, while underfitting occurs when a model is too simple to capture underlying patterns.
Describe how to compare training and validation metrics: overfitting shows low training error but high validation error, while underfitting shows high training and validation error.
Mention that plotting training and validation error against training set size or model complexity helps visualize the gap and diagnose overfitting or underfitting.
Briefly outline solutions: for overfitting, use more data, regularization, or simpler models; for underfitting, use more complex models or better features.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining Bayesian inference as a method to update beliefs based on evidence, then explain likelihood as the probability of observed data given a hypothesis. Use a simple example like coin flips to illustrate how prior, likelihood, and posterior relate, and connect it to software engineering contexts such as A/B testing or spam filtering.
Pro tip: Emphasize the trade-offs between Bayesian and frequentist approaches, and mention how Bayesian methods can be computationally intensive but provide a principled way to incorporate prior knowledge—showing you understand practical implications.
Explain that Bayesian inference updates the probability of a hypothesis as more evidence becomes available, using Bayes' theorem.
Define likelihood as the probability of observing the data given a specific hypothesis or parameter value, distinct from the probability of the hypothesis.
Describe how the prior belief is combined with the likelihood of the data to produce the posterior distribution, using Bayes' theorem: P(H|D) = P(D|H) * P(H) / P(D).
Walk through a simple example, such as estimating the bias of a coin after observing flips, to illustrate the concepts.
Discuss applications like A/B testing, spam filtering, or recommendation systems, and mention trade-offs such as computational cost and the need for prior selection.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining perplexity as the exponentiated average negative log-likelihood per token, then explain that it measures how well a language model predicts a sample, with lower values indicating better performance. Emphasize its role as an intrinsic evaluation metric and discuss its limitations, such as sensitivity to tokenization and vocabulary size.
Pro tip: Mention that perplexity is not comparable across models with different tokenizers or vocabularies, and that while it's useful for tracking training progress, it doesn't always correlate with downstream task performance.
State that perplexity is the exponential of the cross-entropy loss, representing the model's uncertainty in predicting the next token. Formally, it's 2^H or e^H, where H is the average negative log-likelihood.
Describe perplexity as a measure of how well a probability model predicts a sample. In language modeling, it quantifies the model's confidence in its predictions; lower perplexity means the model is less 'perplexed' and assigns higher probability to the actual data.
Explain that a perplexity of k means the model is as uncertain as if it were choosing uniformly among k equally likely options at each step. For example, perplexity equal to vocabulary size indicates random guessing.
Highlight that perplexity depends on tokenization and vocabulary, so it's not comparable across models with different tokenizers. Also, it may not reflect performance on downstream tasks, and optimizing perplexity can sometimes lead to overfitting or less useful representations.
Mention that perplexity is commonly used during training to monitor convergence and compare models with identical tokenization, but for final evaluation, task-specific metrics are preferred.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by contrasting the sequential nature of RNNs with the parallelizable architecture of Transformers, emphasizing how this impacts training efficiency and long-range dependency modeling. Then, discuss the self-attention mechanism and its ability to capture global context without recurrence, and finally touch on practical trade-offs like computational complexity and memory usage.
Pro tip: Acknowledge that Transformers have quadratic complexity in sequence length, but highlight that their parallelizability and effective long-range modeling often outweigh this for many tasks, especially with modern optimizations like sparse attention.
Explain that RNNs process sequences step-by-step, creating a sequential bottleneck, while Transformers process all positions simultaneously via self-attention.
Discuss how Transformers enable parallel computation across sequence positions, leading to faster training on modern hardware like GPUs/TPUs.
Explain that RNNs struggle with vanishing/exploding gradients over long sequences, while Transformers' self-attention directly connects any two positions, capturing long-range dependencies effectively.
Mention that Transformers have O(n^2) complexity in sequence length, but this is often manageable with optimizations, and the benefits outweigh the costs for many applications.
Summarize that these advantages have led to Transformers' dominance in NLP and beyond, despite RNNs' theoretical efficiency for certain tasks.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The matrix multiplication part is straightforward but linking it to autoregressive decoding tripped me up a bit.
Start by stating the standard O(n^3) complexity for multiplying two n×n matrices, then mention sub-cubic algorithms like Strassen and the theoretical lower bound. Next, connect this to autoregressive decoding in Transformers, explaining that each token generation step involves matrix-vector multiplications (O(n^2) per step) and that the overall decoding is O(n^3) for a sequence of length n, but with different constants and parallelization characteristics.
Pro tip: Emphasize the practical distinction: during training, matrix multiplications are batched and highly parallelizable, while autoregressive decoding is sequential and memory-bandwidth-bound, making it latency-sensitive despite similar asymptotic complexity.
State that multiplying two n×n matrices takes O(n^3) time using the naive algorithm, and mention that sub-cubic algorithms like Strassen (O(n^2.807)) exist but are rarely used in practice due to overhead.
Describe that autoregressive decoding generates one token at a time, where each step involves computing attention and feed-forward layers using matrix-vector products, not full matrix-matrix products.
Show that for a sequence of length n, the total decoding cost is O(n^3) because each of the n steps involves operations on growing key/value caches, but per-step cost is O(n^2) and sequential.
Highlight that despite similar asymptotic complexity, decoding is often memory-bound and latency-limited due to sequential dependencies, unlike training which is compute-bound and parallelizable.
Briefly note techniques like KV caching, quantization, and speculative decoding that mitigate the sequential bottleneck in practice.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Covered momentum in SGD, adaptive learning rates in Adam, and then AdamW decoupling weight decay from the gradient update.
Start by defining each optimizer's core mechanism, then compare their update rules and practical implications. Highlight how AdamW addresses a key flaw in Adam's weight decay implementation, and discuss when to choose each based on problem characteristics.
Pro tip: Mention that AdamW's decoupled weight decay often leads to better generalization, especially in large-scale deep learning tasks, and that SGD with momentum can still outperform adaptive methods in some computer vision tasks when tuned properly.
Explain that Stochastic Gradient Descent updates parameters using the gradient of the loss with respect to each parameter, optionally with momentum to accelerate convergence. Mention that it uses a single learning rate for all parameters.
Describe Adam as an adaptive optimizer that computes individual learning rates for each parameter using estimates of first and second moments of the gradients. Note that it incorporates bias correction and typically includes L2 regularization via weight decay added to the loss.
Explain that AdamW modifies Adam by decoupling weight decay from the gradient update, applying it directly to the weights. This corrects the issue where Adam's L2 regularization is scaled by the adaptive learning rate, leading to suboptimal regularization.
Contrast the update equations: SGD uses a fixed learning rate, Adam scales updates by the inverse of the square root of the second moment, and AdamW separates weight decay. Emphasize how these differences affect convergence and generalization.
Summarize when to use each: SGD for simplicity and sometimes better generalization in vision, Adam for fast convergence and sparse gradients, AdamW for improved regularization in large models like Transformers. Mention that hyperparameter tuning is crucial.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.