This was basically a full lecture compressed into one interview question.
Start by defining the problem and the importance sampling estimator, then derive both unnormalized and self-normalized forms. Discuss bias-variance tradeoffs, ideal proposal properties, effective sample size, and illustrate with a numerical example. Finally, cover failure modes and practical implications.
Pro tip: Emphasize that while self-normalized importance sampling is biased, it often has lower variance and is preferred in practice, especially when the normalizing constant is unknown. Also, relate ESS to the reliability of estimates.
Explain the goal: estimate E_p[f(X)] when sampling from p is difficult. Introduce proposal distribution q and derive the unnormalized estimator: (1/N) sum f(x_i) p(x_i)/q(x_i). Then derive the self-normalized estimator: sum f(x_i) w_i / sum w_i, where w_i = p(x_i)/q(x_i).
Compare unnormalized (unbiased but potentially high variance) vs self-normalized (biased but often lower variance, especially when p is unnormalized). Mention that self-normalized is consistent and often preferred in practice.
State that the ideal q minimizes variance, often proportional to |f(x)| p(x). Define effective sample size (ESS) as (sum w_i)^2 / sum w_i^2, which measures the efficiency of the proposal. Relate ESS to the variance of the weights.
Provide a simple example, e.g., estimating E_p[X] for a standard normal p using a shifted normal q. Compute weights and estimates for a few samples, showing how ESS and estimates change with different q.
Discuss weight degeneracy (few dominant weights leading to high variance) and heavy-tailed proposals (which can cause infinite variance if q has heavier tails than p). Mention diagnostics like ESS and weight histograms.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.