Google·Data Analyst·Technical Phone Screen·Intermediate

Intermediate

Apr 2026

Summary

Interviewed at Google, probably for a data or analytics role. One question on statistical vs practical significance. Short post, not much else to go on.

Questions Asked (1)

Why can a result be statistically significant but still not matter in practice?

A/B Testing & ExperimentationProduct Analytics & Metrics

Author's notes

I knew the textbook answer here but fumbled explaining it cleanly.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining statistical significance and practical significance, then explain how large sample sizes can make tiny effects statistically significant. Use a concrete example to illustrate the difference, and emphasize the importance of effect size and business impact.

Pro tip: Always tie statistical significance back to business metrics—ask 'so what?' and quantify the impact in terms of revenue, user engagement, or other KPIs. This shows you think like a product analyst, not just a statistician.

1. Define the concepts

Clearly distinguish between statistical significance (probability the result is not due to chance) and practical significance (whether the effect is large enough to matter in the real world).

2. Explain the role of sample size

Highlight that with very large samples, even trivial effects can become statistically significant because the standard error shrinks. This is common in big tech companies like Google.

3. Introduce effect size and confidence intervals

Discuss how effect size (e.g., Cohen's d, relative lift) and confidence intervals provide a range of plausible effects, helping assess practical relevance.

4. Connect to business impact

Translate the effect into business terms: does a 0.1% increase in click-through rate justify the engineering cost? Consider implementation costs, opportunity costs, and strategic alignment.

5. Provide a concrete example

Illustrate with a scenario: e.g., a new feature increases average revenue per user by $0.01, statistically significant with millions of users, but the cost to build and maintain it far exceeds the gain.

Key Points to Mention

Statistical significance depends on sample size and variance; large samples can detect tiny effects.
Practical significance is about the magnitude of the effect and its real-world implications.
Effect size measures (e.g., relative lift, Cohen's d) quantify the magnitude.
Confidence intervals show the range of plausible effects, not just whether it includes zero.
Business metrics (ROI, revenue, user retention) determine if an effect matters.
Cost-benefit analysis and strategic alignment are crucial for decision-making.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.