CVS Health·Data Scientist·Technical Phone Screen·Senior

Senior

Jul 2026

Summary

CVS Health data scientist interview with a deep-dive case question on diagnosing a failed vaccination lift experiment. The question was layered enough that I kept second-guessing whether I was going too deep or not deep enough.

Questions Asked (1)

Your vaccination outreach experiment shows no lift overall and a negative effect in the senior population. Walk through how you'd diagnose what went wrong, from exposure verification to re-test design, and explain what data you'd pull and what you'd actually do with it.

A/B Testing & ExperimentationRoot Cause AnalysisProduct Analytics & Metrics

Author's notes

This one wrecked me a little.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by validating the experiment's integrity—checking exposure, randomization, and instrumentation—before interpreting results. Then segment the analysis to understand the negative effect in seniors, and propose a re-test with design improvements. Emphasize a structured, data-driven diagnostic process.

Pro tip: Always verify that the treatment was actually delivered as intended (e.g., vaccination outreach messages were sent and received) before diving into effect analysis; many 'failed' experiments are actually implementation failures.

1. Exposure and Instrumentation Verification

Confirm that users in the treatment group were properly exposed to the outreach and that data logging is accurate. Check for issues like message delivery failures, tracking bugs, or sample ratio mismatch.

2. Randomization and Sample Balance Check

Validate that treatment and control groups are comparable on key covariates (age, gender, geography, baseline vaccination status). Look for confounding or selection bias, especially in the senior subgroup.

3. Segmented Effect Analysis

Analyze the negative effect in seniors: examine dosage, timing, message content, and channel. Consider heterogeneity—maybe the outreach was perceived as pushy or confusing for this group.

4. Root Cause Investigation

Pull data on engagement metrics (open rates, click-throughs), survey feedback, and external factors (e.g., vaccine availability, news). Use qualitative and quantitative methods to identify why seniors responded negatively.

5. Re-test Design and Next Steps

Propose a follow-up experiment with modifications: different messaging for seniors, improved targeting, or a holdout to measure long-term effects. Define success metrics and power analysis.

Key Points to Mention

Sample Ratio Mismatch (SRM) check to detect randomization issues
Intent-to-treat vs. per-protocol analysis to handle non-compliance
Heterogeneous treatment effects and subgroup analysis with multiple testing correction
Instrumentation and logging validation (e.g., exposure logs, event tracking)
Qualitative data (user feedback, surveys) to complement quantitative findings
Power analysis and minimum detectable effect for re-test design

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.