Tubi·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

Tubi data scientist interview with a meaty causal inference question that honestly felt more like a take-home problem than a live screen. The question was well-constructed but I spent way too long on the bias directions before getting to the fixes.

Questions Asked (1)

Users can opt into seeing a discount banner (treatment), and treated users show higher conversion. Define selection bias, identify at least three mechanisms causing it and the direction of each bias, propose two methods that produce less biased causal estimates along with their assumptions and how you'd test those assumptions, and walk through a small numerical example where the naive estimate is wrong but the corrected method gets you closer to the truth.

A/B Testing & ExperimentationProduct Analytics & MetricsRoot Cause Analysis

Author's notes

This is the kind of question that sounds manageable until you're actually in it.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining selection bias as a distortion in the estimated treatment effect due to non-random assignment, then systematically explain three mechanisms (self-selection, confounding, and differential attrition) with their bias directions. Next, propose two methods like instrumental variables and propensity score matching, detailing their assumptions and tests, and finish with a numerical example contrasting naive and corrected estimates.

Pro tip: Emphasize that opt-in experiments often violate SUTVA and positivity, so always check overlap and consider sensitivity analysis to unmeasured confounding; this shows depth beyond textbook answers.

1. Define selection bias and its relevance

Explain selection bias as a systematic difference between treated and control groups due to non-random assignment, leading to biased causal estimates. Highlight why it matters in opt-in settings like discount banners.

2. Identify mechanisms and bias directions

Describe at least three mechanisms: self-selection (users with higher intent opt in, biasing estimate upward), confounding (unobserved factors like user engagement affect both opt-in and conversion, direction depends on correlation), and differential attrition (treated users more likely to drop out, biasing estimate downward if dropouts have lower conversion).

3. Propose methods for less biased estimates

Suggest two methods: instrumental variables (IV) and propensity score matching (PSM). For IV, assumption is instrument affects treatment but not outcome except through treatment; test via exclusion restriction and relevance. For PSM, assumption is no unmeasured confounders; test via balance checks and sensitivity analysis.

4. Construct a numerical example

Create a small dataset where naive estimate shows a large effect, but after applying IV or PSM, the estimate shrinks toward the true effect. Show calculations step-by-step to illustrate the correction.

5. Summarize and conclude

Recap the mechanisms, methods, and example, emphasizing the importance of assumptions and testing in causal inference for opt-in experiments.

Key Points to Mention

Selection bias definition and its impact on causal inference
Mechanisms: self-selection, confounding, differential attrition with bias directions
Instrumental variables: assumptions (relevance, exclusion restriction) and tests (first-stage F-stat, overidentification tests)
Propensity score matching: assumption of no unmeasured confounders, balance checks, sensitivity analysis
Numerical example with naive vs. corrected estimates
Importance of testing assumptions and sensitivity analysis in observational studies

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.