This is where I spent most of my mental energy and still felt shaky at the end.
Start by defining the strata based on query frequency (head, torso, tail) and explain how to allocate the 2,000 weekly judgments using Neyman allocation to minimize variance for the overall NDCG@10 estimate. Then, present the importance-weighted estimator that combines stratified sampling with inverse probability weighting to correct for the non-uniform sampling, and derive its variance formula. Finally, discuss how to extend this to measure long-tail NDCG@10 changes by focusing on the tail strata and possibly using a separate estimator.
Pro tip: Emphasize that Neyman allocation requires estimating the variance of NDCG@10 within each stratum, which can be approximated from historical data or pilot studies; also note that the importance weights should be the inverse of the sampling probabilities, and that the estimator's variance can be computed using the delta method or bootstrap for ratio metrics like NDCG.
Divide queries into strata based on their frequency (e.g., head: top 1%, torso: next 9%, tail: remaining 90%) and define the sampling unit as a query. Determine the population size N_h for each stratum.
Allocate the 2,000 judgments per week across strata proportionally to N_h * S_h, where S_h is the standard deviation of NDCG@10 within stratum h. This minimizes the variance of the overall NDCG@10 estimate for a fixed budget.
For each sampled query i in stratum h, compute its NDCG@10 for both rankers, then form the importance-weighted estimator: \hat{\mu} = \sum_{h} \sum_{i \in S_h} w_{hi} * NDCG_{hi}, where w_{hi} = 1 / \pi_{hi} and \pi_{hi} is the inclusion probability. Normalize weights to sum to 1 if needed.
The variance of \hat{\mu} is approximately \sum_{h} (1 - n_h/N_h) * S_h^2 / n_h, where n_h is the number of samples in stratum h. For the difference between two rankers, use the variance of the difference, accounting for paired sampling.
For the long-tail, focus on the tail stratum and use a similar estimator but with weights adjusted to represent the tail population. Alternatively, use a separate Neyman allocation for the tail if the budget allows, or combine with the overall estimator by conditioning on the tail.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Knew IPS in theory but when they pushed on propensity requirements I got a bit vague.
Start by framing the problem as combining a small unbiased human-labeled set with large-scale biased click data to estimate ranking performance. Propose using doubly robust estimation to combine a propensity-weighted IPS estimator from clicks with a direct method (outcome model) trained on human labels, ensuring robustness if either model is misspecified. Then detail the propensity requirements (positivity, correct specification, logging policy) and how to collect them via randomization or exploration.
Pro tip: Emphasize that propensity scores must reflect the probability of observation under the logging policy, and that even small amounts of randomization in production can enable unbiased evaluation. Also, mention that doubly robust estimation can reduce variance by incorporating a reward model, making it practical for Google-scale data.
Explain that the goal is to estimate the performance of a new ranking model using limited human judgments and abundant click logs. Highlight that human labels are unbiased but scarce, while clicks are biased but plentiful.
Describe IPS as a technique that reweights clicked documents by the inverse probability of being observed under the logging policy. Note that this yields an unbiased estimate if propensities are known and positive.
Propose using doubly robust estimation, which combines an IPS estimator with a direct method (e.g., a regression model trained on human labels). This provides robustness: if either the propensity model or the outcome model is correct, the estimate remains unbiased.
State that propensities must be strictly positive for all query-document pairs and correctly specified. Collect them by logging the probability of each document being shown under the production policy, or by injecting randomization (e.g., epsilon-greedy) to ensure exploration.
Discuss challenges like high variance, propensity estimation errors, and distribution shift. Suggest validating with A/B tests or simulation, and using techniques like clipping or normalization to stabilize IPS weights.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying the metric and test design: NDCG@10 is a per-query metric, so you need to estimate the variance of the difference in mean NDCG@10 between two rankers. Use the standard sample size formula for comparing two means, but explicitly state assumptions about the variance of NDCG@10 and the correlation between paired observations (if using a paired test). Then compute the required number of queries (or sessions) and discuss how variance assumptions affect the result.
Pro tip: Mention that NDCG@10 is bounded between 0 and 1, so its variance is at most 0.25, but in practice it's often much smaller (e.g., 0.01–0.05). Also note that using a paired test (same queries for both rankers) can dramatically reduce required sample size by accounting for correlation.
Confirm that NDCG@10 is computed per query and averaged over queries. Determine whether the test is paired (same queries evaluated by both rankers) or unpaired, as this affects variance calculation.
Assume a variance for NDCG@10 (e.g., from historical data or pilot). For a paired test, also assume a correlation between the two rankers' scores on the same query, or directly assume the variance of the difference.
For an unpaired test: n = 2 * (z_{α/2} + z_β)^2 * σ^2 / Δ^2. For a paired test: n = (z_{α/2} + z_β)^2 * σ_diff^2 / Δ^2, where σ_diff^2 = Var(NDCG_A - NDCG_B).
Use Δ = 0.01, 95% confidence (z_{α/2}=1.96), and power 80% (z_β=0.84). For example, if σ = 0.1 (unpaired), n ≈ 2 * (2.8)^2 * 0.01 / 0.0001 = 1568 per group. If paired with σ_diff = 0.05, n ≈ (2.8)^2 * 0.0025 / 0.0001 = 196 per group.
Highlight that the required sample size is highly sensitive to variance assumptions. Suggest using pilot data or historical A/B tests to estimate variance, and consider clustering (e.g., queries within sessions) if applicable.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I gave a reasonable answer about using model uncertainty (entropy or margin sampling) to select which query-doc pairs to label next, and prioritizing the long-tail stratum since that's where label coverage is worst.
Start by framing the problem as a budget-constrained optimization: maximize model improvement per labeling dollar by targeting high-uncertainty, high-impact examples. Describe a concrete active learning loop that combines uncertainty sampling with diversity and representativeness, and explain how you'd measure its ROI. Then outline a monitoring plan for distribution drift using statistical tests and business metrics, emphasizing proactive retraining triggers.
Pro tip: Tie the active learning loop to a clear business metric (e.g., reduction in annotation cost per F1 point gained) and propose a small-scale A/B test to validate the strategy before full rollout. This shows you think like a product-minded data scientist, not just a modeler.
Clarify the goal: minimize annotation cost while maximizing model performance on long-tail queries. Identify budget, labeling capacity, and target metrics (e.g., recall on rare classes).
Use a hybrid acquisition function: combine uncertainty (e.g., entropy, margin) with diversity (e.g., clustering in embedding space) and representativeness (e.g., density weighting). Prioritize long-tail by upweighting rare classes or using class-balanced sampling.
Train a model on the current labeled set, score unlabeled data, select a batch for labeling, and retrain. Track cost per improvement and adjust acquisition strategy based on feedback.
Week over week, compute drift metrics (e.g., KL divergence, PSI) on input features and predicted labels. Use statistical tests (e.g., KS test) and track changes in query distribution, especially long-tail frequency.
Define thresholds for drift metrics and automate alerts. When drift exceeds threshold, trigger model retraining or re-prioritization of labeling budget toward newly emerging long-tail queries.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.