CVS Health·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Brutal technical screen for a DS role at CVS Health. The whole thing was basically one massive multi-part prompt about designing a flu vaccination email experiment from scratch, and they wanted you to go deep on every single layer of it.

Questions Asked (1)

Design an end-to-end email campaign experiment to increase verified flu vaccinations within 30 days among eligible members. Walk through targeting and exclusion rules, randomization unit, holdout structure, primary and guardrail metrics with ground-truth outcome sourcing, power analysis for a 2% baseline with 20% relative uplift at alpha=0.05 and 80% power, frequency capping and creative testing, seasonality controls, HIPAA and opt-out compliance implications, and your full analysis plan including a fallback causal method if randomization breaks.

A/B Testing & ExperimentationProduct Analytics & MetricsSystem Design

Author's notes

This was basically nine questions stuffed into one.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer around the experiment lifecycle: design (targeting, randomization, holdout), measurement (metrics, power, ground truth), execution (frequency capping, creative testing, seasonality), and analysis (compliance, causal methods). Emphasize healthcare-specific constraints like HIPAA and opt-out compliance, and show how you'd handle randomization breaks with a fallback causal method.

Pro tip: In healthcare experiments, always pre-specify how you'll source ground-truth vaccination status (e.g., claims data with a 30-day lag) and account for it in your power analysis; otherwise, you risk underpowering your study or missing the primary endpoint.

1. Define Population and Randomization

Specify inclusion criteria (eligible members: age, risk factors, no prior flu shot this season) and exclusion rules (opt-outs, recent vaccinations, contraindications). Choose randomization unit (member-level) and holdout structure (e.g., 10% pure control, 90% treatment with nested creative tests).

2. Select Metrics and Ground Truth

Identify primary metric (verified flu vaccination within 30 days) and guardrail metrics (opt-out rate, complaint rate, email engagement). Source ground truth from claims data, immunization registries, or EHRs, and define verification windows.

3. Conduct Power Analysis and Design

Calculate sample size for 2% baseline, 20% relative uplift (2.4% absolute), alpha=0.05, power=0.80. Use power analysis to determine required sample per arm, then design frequency capping (e.g., max 2 emails/week) and creative testing (A/B/n subject lines, content) within the treatment group.

4. Address Seasonality and Compliance

Control for seasonality by randomizing within blocks (e.g., by geography or enrollment date) or using time-series methods. Ensure HIPAA compliance (de-identified data, secure transmission) and opt-out mechanisms (unsubscribe link, preference center) are in place.

5. Plan Analysis and Fallbacks

Pre-specify analysis: intention-to-treat (ITT) with logistic regression, covariate adjustment, and subgroup analyses. If randomization breaks, use instrumental variables, propensity score matching, or difference-in-differences as fallback causal methods.

Key Points to Mention

Targeting and exclusion rules: include eligible members based on age, risk, and no prior vaccination; exclude opt-outs, recent vaccines, and contraindications.
Randomization unit and holdout: member-level randomization with a pure holdout (e.g., 10%) to measure absolute lift and a treatment group for creative testing.
Primary and guardrail metrics: primary = verified flu vaccination within 30 days; guardrails = opt-out rate, complaint rate, email engagement.
Power analysis: baseline 2%, relative uplift 20% (absolute 2.4%), alpha=0.05, power=0.80; calculate sample size per arm (approx. 3,800 per arm).
Frequency capping and creative testing: cap emails (e.g., 2 per week) to avoid fatigue; test subject lines, content, and send times within treatment.
Seasonality controls and compliance: block randomization by geography/time; ensure HIPAA-compliant data handling and easy opt-out.
Fallback causal method: if randomization breaks, use instrumental variables, propensity score matching, or difference-in-differences.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.