This one took me a minute to even set up properly.
Frame the problem as estimating a population proportion with a fixed budget, comparing the variance of a simple random sample (spread) versus a cluster sample with majority voting (concentrated). Argue that spreading is generally better unless there is high within-subject correlation, and then outline how to construct a confidence interval using the appropriate variance formula.
Pro tip: Acknowledge that the optimal design depends on the intraclass correlation (ICC) and that in practice you might use a hybrid approach, but for a statistical argument, spreading is the default because it maximizes effective sample size when ICC is low.
Clarify that you want to estimate the true accuracy (proportion correct) of the classifier on the population. Assume each human review is an unbiased binary judgment of the classifier's prediction on that subject.
For spread (K subjects, 1 review each), the estimator is the sample proportion with variance p(1-p)/K. For concentrated (n subjects, m reviews each, K=nm), the majority vote estimator has variance that depends on the intraclass correlation (ICC) ρ. Derive that the variance is approximately p(1-p)/K * [1 + (m-1)ρ] for the concentrated design, showing it is larger when ρ>0.
Majority vote can reduce variance if individual reviews are noisy but unbiased, but it introduces bias if the noise is systematic (e.g., correlated errors). Also, concentrating reduces the number of unique subjects, increasing the risk of selection bias if the chosen subjects are not representative.
Unless there is strong evidence of high ICC (e.g., from prior data), spread the reviews across K subjects to maximize the effective sample size and minimize variance. If ICC is high, a concentrated design with majority vote might be better, but you need to estimate ICC to decide.
For the spread design, use the standard Wald or Wilson interval for a proportion. For the concentrated design, use a cluster-adjusted variance estimate (e.g., with ICC) or a bootstrap resampling at the subject level to account for within-subject correlation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.