← CVS Health Interview Insights
I went straight to factorial and felt pretty good about it, but I fumbled the interference piece.
Start by clarifying the business goal: to maximize flu vaccination rates while minimizing cost. Then compare a simple A/B test (SMS vs. Email) with a 2x2 factorial (SMS/Email x message variant) by discussing interaction effects, interference, and send costs. Conclude with a recommendation based on whether you expect interactions and whether the incremental cost of the additional cells is justified.
Pro tip: In practice, factorial designs are powerful but can suffer from interference when channels overlap (e.g., patients receiving both SMS and Email). Always consider whether the treatment groups are independent and whether the interaction effect is of primary interest or a nuisance.
Define the primary metric (e.g., vaccination rate) and secondary metrics (e.g., cost per vaccination, engagement). Identify constraints such as budget, timeline, and available sample size.
Outline a simple A/B test (SMS vs. Email) and a 2x2 factorial (SMS/Email x message variant). Discuss the trade-offs in terms of statistical power, ability to detect interactions, and operational complexity.
Explain that a 2x2 factorial allows testing whether the effect of channel depends on message variant (and vice versa). If no interaction is expected, a simple A/B test may suffice; if interaction is plausible, factorial is preferred.
Discuss potential interference: patients may receive both SMS and Email if not properly randomized, leading to contamination. Also, sending multiple messages increases cost; factorial requires more sends, so weigh the cost against the value of interaction insights.
Based on the above, recommend either a simple A/B test (if resources are limited, no interaction expected, or interference is high) or a 2x2 factorial (if interaction is of interest and costs are manageable). Justify with expected impact on vaccination rates and ROI.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying the experiment's goal and the business context at CVS Health, then systematically define each component: eligibility criteria, exclusions, randomization unit, and stratification variables. Explain how each choice impacts validity, bias, and statistical power, and tie them to practical constraints like data availability and ethical considerations.
Pro tip: Always align your design with the company's existing data infrastructure and privacy policies; for example, at CVS Health, consider how HIPAA and member consent affect eligibility and data collection. Also, pre-register your analysis plan to avoid p-hacking and demonstrate rigor.
Understand the hypothesis, target population, and success metrics. This ensures all design choices support the experiment's goal and business relevance.
Specify who can participate based on demographics, behavior, or tenure. Include criteria that ensure sufficient exposure to the treatment and relevance to the business question.
Identify groups to exclude, such as employees, bots, or users with prior exposure, to reduce confounding and interference. Justify each exclusion with data or logic.
Select the unit of randomization (e.g., user, session, store) based on the experiment's nature and potential spillover. Consider clustering and analysis implications.
Pick variables correlated with the outcome to balance groups and increase power. Use pre-treatment covariates and ensure strata are large enough for stable estimates.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The formula part was fine, standard two-proportion z-test, came out around 3,500 per arm give or take.
First, compute the required sample size per arm for a two-proportion z-test using the given baseline rate, MDE, alpha, and power. Then, adjust for household clustering by calculating the design effect (1 + (average household size - 1) * ICC) and multiply the initial sample size by this factor. Finally, discuss the implications for the experiment, such as increased recruitment needs and potential trade-offs.
Pro tip: Always clarify whether the MDE is absolute or relative; here it's absolute (1.5 percentage points). Also, mention that clustering reduces effective sample size, so you might need to randomize at the household level or use cluster-robust standard errors.
Use the formula for two-proportion z-test: n = (Z_{α/2} + Z_β)^2 * (p1*(1-p1) + p2*(1-p2)) / (p2-p1)^2, where p1=0.08, p2=0.095 (8% + 1.5pp), Z_{α/2}=1.96, Z_β=0.84. Calculate n per arm.
Design effect (DE) = 1 + (average cluster size - 1) * ICC. Here, average household size = 1.3, ICC = 0.01, so DE = 1 + (1.3 - 1)*0.01 = 1.003.
Multiply the base sample size by the design effect to get the required sample size per arm accounting for clustering. This yields a slight increase (0.3% more).
Explain that the inflation is minimal due to small household size and low ICC, but in other scenarios it could be substantial. Mention that if randomization is at household level, the number of households needed is the adjusted sample size divided by average household size.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining the ITT estimator as the difference in means between randomized groups, then introduce deliverability as a binary instrument to compute the TOT via the Wald estimator. Clearly show the ITT/compliance relationship (ITT = TOT × compliance rate) and list the key assumptions (randomization, exclusion restriction, monotonicity, SUTVA) that validate the instrumental variable approach.
Pro tip: Emphasize that ITT is the causal effect of assignment, not treatment, and that deliverability is a valid instrument only if it affects the outcome solely through treatment receipt. Also, mention that in practice, compliance rates are often estimated from the data, and you should check for weak instruments.
State that ITT is the average causal effect of being randomized to the treatment group, regardless of actual treatment received. Formally, ITT = E[Y | Z=1] - E[Y | Z=0], where Z is the randomization indicator.
Explain that deliverability (e.g., whether the email/message was successfully delivered) is a binary instrument that affects treatment receipt but is independent of potential outcomes given randomization. It must satisfy relevance, exclusion restriction, and monotonicity.
Use the formula TOT = ITT / compliance rate, where compliance rate = E[D | Z=1] - E[D | Z=0] and D is the treatment receipt indicator. This is the local average treatment effect (LATE) for compliers.
Derive that ITT = TOT × compliance rate, meaning the intention-to-treat effect is diluted by non-compliance. This relationship highlights why ITT underestimates the treatment effect when compliance is imperfect.
List the required assumptions: (1) random assignment (Z independent of potential outcomes), (2) exclusion restriction (Z affects Y only through D), (3) monotonicity (no defiers), and (4) stable unit treatment value assumption (SUTVA). Also note that the instrument must be relevant (compliance rate > 0).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Proposed unique tracking codes per channel per customer, message timestamps to establish sequence, and then factorial contrasts to estimate main effects and the interaction term.
Start by describing a 2x2 factorial design with SMS and Email as factors, including a holdout group to measure incremental impact. Explain how to instrument the experiment with randomized assignment and track user-level conversions, then analyze using ANOVA or regression to estimate main effects and interactions. Finally, discuss why self-reported attribution is biased due to recall and social desirability, but note it can be useful for qualitative insights or debugging.
Pro tip: Emphasize the importance of pre-registering the analysis plan and checking for interaction effects, as they can significantly alter channel attribution. Also, mention that self-reported attribution can complement experimental data by providing context on user perception.
Set up a 2x2 factorial design with SMS and Email as factors, including a control group receiving neither. Randomize customers to each cell to ensure unbiased assignment.
Implement tracking for exposures (SMS/Email sent, delivered, opened) and outcomes (conversions, revenue). Use unique user IDs to link exposures to outcomes and ensure data quality.
Use ANOVA or linear regression to estimate main effects of each channel and their interaction, controlling for covariates. Calculate incremental lift relative to control.
Explain biases like recall bias, social desirability, and attribution errors. Discuss how self-reported data can misattribute conversions, especially when multiple channels are present.
Suggest using self-reported data for qualitative insights, hypothesis generation, or debugging tracking issues, but not for causal inference.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
ITT lift is just 9.0% minus 7.5% = 1.5pp, pretty clean.
Start by computing the ITT lift as the difference in vaccination rates between the SMS arm and control (9.0% - 7.5% = 1.5 percentage points). Explain that the self-report of 50 customers is subject to attribution bias and does not affect the ITT estimate because ITT measures the causal effect of assignment, not treatment received. Then, outline a plan to test next steps if the lift falls below 1.5 pp, focusing on power, segmentation, and mechanism.
Pro tip: Emphasize that ITT is the gold standard for causal inference in randomized experiments because it preserves randomization and avoids selection bias. Mention that self-reports are often unreliable due to social desirability and recall bias, so they should be used for exploratory insights, not causal claims.
Calculate the difference in vaccination rates between the SMS arm and control: 9.0% - 7.5% = 1.5 percentage points. This is the intent-to-treat (ITT) effect.
The self-report (50 customers) is a subset of the 100 vaccinated in the SMS arm and is subject to attribution bias. ITT measures the effect of being assigned to the SMS arm, regardless of whether customers saw or acted on the message, so it remains valid.
If the lift is below 1.5 pp, first check statistical significance and power. Then, explore heterogeneity by customer segments, message timing, and frequency. Consider whether the control group had contamination or if the SMS delivery failed.
Suggest A/B tests with different message content, send times, or incentives. Also, test alternative channels (e.g., email, app notifications) and measure incremental lift via holdout groups.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Talked through sequential testing with alpha spending to allow early stopping without inflating type I error.
Structure your answer around a clear experimentation lifecycle: monitoring, analysis, and decision-making. Emphasize statistical rigor, business impact, and budget constraints. Use a concrete example to illustrate how you would handle each phase.
Pro tip: Show that you balance statistical significance with practical significance—especially in healthcare where small lifts can have large financial or clinical impact. Mention how you would handle multiple comparisons and guardrail metrics to avoid false positives.
Clearly state primary, secondary, and guardrail metrics (e.g., conversion, revenue, customer satisfaction, safety). Establish minimum detectable effect and required sample size to ensure adequate power.
Set up real-time dashboards to track sample ratio mismatch, metric trends, and guardrail violations. Use sequential testing or alpha spending to allow early stopping for harm or futility without inflating Type I error.
After the experiment concludes, perform hypothesis tests (e.g., t-test, bootstrap) and calculate confidence intervals. Adjust for multiple comparisons if needed, and check for heterogeneous treatment effects across key segments.
Estimate the incremental lift and its monetary value. Compare against implementation costs and budget constraints. Consider scalability, operational feasibility, and potential risks.
Propose a rollout plan (e.g., full scale, phased, or targeted) based on ROI and budget. Define monitoring and re-evaluation cadence, and suggest a decision framework for future experiments under fixed budget.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.