Pretty foundational but I second-guessed myself for a second because the numbers felt like they were designed to trick you.
First, identify that the population variance is unknown and the sample size is small (n=15), which points to a t-test. Then, explain that the t-test accounts for the additional uncertainty from estimating the variance, while the z-test would be appropriate only if the population variance were known or the sample size were large. Finally, mention that you would check normality assumptions and possibly consider a non-parametric test if violated.
Pro tip: Emphasize that in practice, with small samples, the t-test is robust to moderate departures from normality, but you should always verify assumptions and consider the impact on Type I and Type II errors.
Determine whether the population variance is known or unknown, and note the sample size (n=15).
Since the population variance is unknown and the sample size is small, select the t-test over the z-test.
Discuss how the t-test uses the sample standard deviation and accounts for the extra uncertainty via the t-distribution with n-1 degrees of freedom.
Verify that the data are approximately normally distributed or that the sample size is sufficient for the Central Limit Theorem (though n=15 is small).
If normality is questionable, mention non-parametric options like the Wilcoxon signed-rank test.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clearly stating the null and alternative hypotheses for the two-sided one-sample t-test. Then compute the t-statistic using the formula t = (x̄ - μ₀) / (s/√n), find the p-value from the t-distribution with n-1 degrees of freedom, and construct the 95% confidence interval using x̄ ± t_{α/2, n-1} * (s/√n). Finally, interpret the p-value and confidence interval in the context of the problem.
Pro tip: Always check the assumptions for the t-test (e.g., normality, independence) and mention them briefly; also, when interpreting the confidence interval, avoid saying 'there is a 95% probability the true mean lies in the interval'—instead say 'we are 95% confident' or 'the method captures the true mean 95% of the time.'
Define H₀: μ = μ₀ (often 10 or a specified value) and H₁: μ ≠ μ₀. Identify that this is a two-sided one-sample t-test since the population standard deviation is unknown.
Calculate the standard error (SE = s/√n = 2.1/√15 ≈ 0.542) and the t-statistic: t = (10.4 - μ₀) / SE. For μ₀=10, t ≈ 0.738.
Determine the degrees of freedom (df = n-1 = 14) and use a t-distribution table or software to find the two-tailed p-value corresponding to the computed t-statistic. For t ≈ 0.738, p ≈ 0.473.
Find the critical t-value for 95% confidence (t_{0.025,14} ≈ 2.145). Compute the margin of error: ME = t * SE ≈ 2.145 * 0.542 ≈ 1.163. The CI is x̄ ± ME = 10.4 ± 1.163, giving (9.237, 11.563).
Interpret the p-value: if p < 0.05, reject H₀; otherwise, fail to reject. Interpret the CI: we are 95% confident that the true population mean lies between 9.237 and 11.563. Relate back to the business context if applicable.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Expected false positives under the null is just 20 times 0.05, so 1.
Start by clarifying the assumptions: 20 independent tests, each with α=0.05, and no correction. Then compute the expected number of false positives as 20 * 0.05 = 1, and the family-wise error rate (FWER) as 1 - (1 - 0.05)^20 ≈ 0.6415. Finally, discuss the implications for multiple testing and briefly mention correction methods like Bonferroni or FDR.
Pro tip: Emphasize that the expected number of false positives is a long-run average, while FWER is the probability of at least one false positive; also note that independence is a simplifying assumption that may not hold in practice, but it's standard for this calculation.
Confirm that the tests are independent, each uses α=0.05, and no multiple testing correction is applied. Define false positive (Type I error) and family-wise error rate (FWER).
Since each test has a 5% chance of a false positive, the expected number is the sum of individual probabilities: 20 * 0.05 = 1.
FWER is the probability of at least one false positive. Under independence, FWER = 1 - (1 - α)^m, where m=20. So FWER = 1 - 0.95^20 ≈ 0.6415.
Explain that with 20 tests, we expect 1 false positive on average, and there's a 64% chance of at least one false positive. This highlights the multiple comparisons problem and the need for corrections like Bonferroni or FDR control.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
State the Bonferroni formula (adjusted alpha = original alpha / number of tests) and compute 0.05/20 = 0.0025. Then briefly explain the rationale: controlling the family-wise error rate by making each test more stringent, and mention the trade-off of increased false negatives.
Pro tip: Acknowledge that Bonferroni is conservative and can be overly strict when tests are correlated; in practice, consider alternatives like Benjamini-Hochberg for false discovery rate control, especially in large-scale A/B testing.
Recognize the original alpha (0.05) and the number of tests (20).
Divide the original alpha by the number of tests: 0.05 / 20 = 0.0025.
Explain that each individual test must have a p-value less than 0.0025 to be considered statistically significant at the 0.05 family-wise error rate.
Mention that this correction reduces false positives but increases false negatives (Type II errors), potentially missing true effects.
Briefly note that other methods like Holm-Bonferroni or Benjamini-Hochberg may be more appropriate depending on the context.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Both control FWER but Tukey is designed specifically for all pairwise comparisons so it's less conservative when that's exactly what you're doing.
Start by clarifying the difference between family-wise error rate (FWER) and per-comparison error rate, then explain that Tukey's HSD controls FWER exactly for all pairwise comparisons when group sizes are equal and variances are homogeneous, while Bonferroni controls FWER conservatively for any set of comparisons. Conclude with practical guidance: prefer Tukey when you need all pairwise comparisons and assumptions hold; prefer Bonferroni when you have a small number of planned comparisons or need flexibility.
Pro tip: Mention that Tukey's HSD is more powerful than Bonferroni for all pairwise comparisons because it uses the studentized range distribution, which accounts for the correlation among comparisons, whereas Bonferroni assumes independence and is overly conservative. Also note that with unequal sample sizes, Tukey-Kramer is the appropriate variant.
Explain that both methods control the family-wise error rate (FWER), but Tukey's HSD is exact for pairwise comparisons while Bonferroni is conservative and controls FWER for any set of comparisons.
Tukey's HSD assumes independent observations, normally distributed residuals, homogeneous variances, and equal sample sizes (or use Tukey-Kramer for unequal sizes).
Tukey's HSD is more powerful for all pairwise comparisons because it leverages the studentized range distribution; Bonferroni is more flexible and can be used for any planned or post-hoc comparisons, but is less powerful.
Prefer Tukey's HSD when you need all pairwise comparisons and assumptions are met; prefer Bonferroni when you have a small number of comparisons, unequal variances, or non-normal data, or when you want a simple, conservative method.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clearly defining Type I and Type II errors in the context of hypothesis testing, then provide a concrete real-world example for each that is relevant to A/B testing or product analytics at Amazon. Emphasize the trade-offs between the two and how they impact business decisions.
Pro tip: Tie the examples to Amazon's business metrics (e.g., conversion rate, customer satisfaction) and mention how you would balance the risks of each error based on the cost of false positives versus false negatives.
Explain that a Type I error occurs when we reject a true null hypothesis, often called a false positive. In A/B testing, this means concluding that a new feature has an effect when it actually does not.
Explain that a Type II error occurs when we fail to reject a false null hypothesis, often called a false negative. In A/B testing, this means missing a real effect of a new feature.
Give a concrete example: Amazon tests a new recommendation algorithm and sees a statistically significant increase in click-through rate, but in reality, the algorithm has no effect. This leads to a false positive and potentially wasted resources.
Give a concrete example: Amazon tests a new checkout flow that actually improves conversion rate, but the test fails to detect the improvement due to insufficient sample size or high variance. This leads to a missed opportunity to improve customer experience.
Explain how the choice between Type I and Type II errors depends on the business context. For example, in a high-stakes decision like changing the homepage layout, a Type I error might be more costly, while in a low-risk experiment, a Type II error might be acceptable.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
LLN says the sample mean converges to the true mean as n grows.
Start by clearly defining both theorems and contrasting their focus: LLN concerns convergence of sample averages to the population mean, while CLT describes the distribution of the sample mean. Then, discuss how heavy-tailed distributions violate CLT assumptions, leading to slow convergence, and illustrate the impact on statistical inference with a concrete example.
Pro tip: Emphasize that in practice, with heavy-tailed data, relying on CLT-based methods (like t-tests) can be misleading; instead, consider robust or non-parametric alternatives, and always check convergence diagnostics.
State the Law of Large Numbers: as sample size grows, the sample mean converges to the population mean. State the Central Limit Theorem: as sample size grows, the distribution of the sample mean approaches normality, regardless of the population distribution (with finite variance).
Explain that LLN tells us about the consistency of an estimator (where it converges to), while CLT tells us about the shape of its sampling distribution (how it varies), enabling hypothesis testing and confidence intervals.
Describe heavy-tailed distributions (e.g., Pareto, Cauchy) where extreme values are more likely. Note that if variance is infinite (e.g., Cauchy), CLT does not apply; if variance is finite but heavy-tailed, convergence can be very slow.
Discuss how slow convergence means the sample mean distribution may not be approximately normal even for large samples, leading to inaccurate p-values, confidence intervals, and error rates in A/B tests.
Give an example: measuring average revenue per user when a few users generate huge revenue. Show that t-tests may fail; suggest using bootstrap, permutation tests, or robust methods, and emphasize the need for larger samples or alternative metrics.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.