Roblox·Data Scientist·Technical Phone Screen·Senior

Senior

Apr 2026

Summary

A Roblox data science interview that was essentially one massive observational study design question. No A/B test allowed, no easy out, just a full end-to-end causal inference problem about local creators. Dense and exhausting.

Questions Asked (1)

You can't run an A/B test. Define 'local creators' as creators whose games match a player's native language. Design a full observational study to estimate whether players prefer local creators, covering metric choice, unit of analysis, sessionization, causal identification strategy, covariate matching, instrument selection, robustness checks, and the SQL schema you'd need.

A/B Testing & ExperimentationProduct Analytics & MetricsData Modeling

Author's notes

This was basically seven questions stapled together and handed to me as one.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the causal question: do players prefer local creators? Then outline an observational study using a natural experiment or instrument (e.g., language-specific game launches) to identify the effect. Structure your answer around metric definition, unit of analysis, sessionization, causal identification, matching, robustness, and SQL schema.

Pro tip: Acknowledge the endogeneity of creator language choice and propose a falsification test (e.g., placebo language) to strengthen credibility. Also, emphasize that 'local' is player-specific, so the same creator can be local for one player and non-local for another, requiring careful unit alignment.

1. Define metrics and unit of analysis

Choose engagement metrics (e.g., session length, retention, playtime) and define the unit as player-session or player-day. Ensure 'local' is determined by the player's native language and the creator's game language.

2. Sessionization and data aggregation

Define a session with a timeout (e.g., 30 minutes of inactivity). Aggregate metrics per player per session, and compute exposure to local creators within each session.

3. Causal identification strategy

Use an instrumental variable: e.g., the release timing of games by local creators in the player's language. Alternatively, exploit a natural experiment like a language-specific game launch. Discuss assumptions (exclusion restriction, relevance).

4. Covariate matching and robustness checks

Match players on covariates (e.g., past playtime, language proficiency, device) using propensity scores. Run robustness checks: placebo tests, sensitivity analysis, alternative metrics, and different session definitions.

5. SQL schema design

Outline tables: players (player_id, native_language), creators (creator_id, language), games (game_id, creator_id, language), sessions (session_id, player_id, start_time, end_time), and plays (play_id, session_id, game_id, duration). Include a view to flag local matches.

Key Points to Mention

Metric choice: engagement (session length, retention) and preference (repeat plays, likes).
Unit of analysis: player-session or player-day, with local exposure defined per player.
Sessionization: inactivity timeout (e.g., 30 min) and handling of overlapping sessions.
Causal identification: instrumental variable (e.g., game release timing) or natural experiment.
Covariate matching: propensity score matching on player and creator characteristics.
Robustness: placebo tests, sensitivity analysis, and alternative specifications.
SQL schema: tables for players, creators, games, sessions, plays, and a local_match view.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.