← Roblox Interview Insights

Roblox·Data Scientist·Technical Phone Screen·Intermediate

Intermediate
May 2026

Summary

Technical screen for a DS role at Roblox covering two coding problems, both stats-flavored. Nothing too wild but you need to actually remember your z-test formula and not just hand-wave through it.

Questions Asked (2)

Q1

Write a function that takes a historical array of observations, a significance level, a desired power, and a minimum detectable effect, then returns the required per-group sample size for a two-sided two-sample z-test.

A/B Testing & ExperimentationAlgorithms & Data Structures
Author's notes

I knew the formula conceptually but blanked on which z-values to pull for alpha and power.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the inputs and assumptions: historical observations provide an estimate of variance, and the function should compute sample size using the standard formula for a two-sided two-sample z-test. Derive the formula step-by-step, then implement it efficiently, handling edge cases like zero variance or invalid parameters.

Pro tip: Mention that in practice, you'd use the pooled variance from historical data but also consider using a t-test for small samples; however, for large-scale A/B tests at Roblox, the z-test approximation is standard. Also, always round up to ensure sufficient power.

1. Clarify inputs and assumptions

Confirm that historical observations are used to estimate the variance (or standard deviation) of the metric, and that the test is two-sided with equal group sizes. Assume normality or large sample sizes for the z-test.

2. Derive the sample size formula

Use the formula for two-sample z-test: n = ( (z_{1-α/2} + z_{1-β})^2 * (σ1^2 + σ2^2) ) / Δ^2, where σ1^2 and σ2^2 are variances of the two groups (often assumed equal), Δ is the minimum detectable effect, α is significance level, and 1-β is power.

3. Estimate variance from historical data

Compute the sample variance from the historical array. If the two groups are assumed to have equal variance, use the pooled variance; otherwise, use the historical variance for both groups if no other information is available.

4. Compute critical z-values

Calculate z_{1-α/2} for the two-sided test and z_{1-β} for the desired power. Use a standard normal quantile function (e.g., scipy.stats.norm.ppf) to obtain these values.

5. Calculate and return sample size

Plug the values into the formula, round up to the next integer to ensure the required power, and return the per-group sample size. Handle edge cases such as zero variance or non-positive MDE by raising an error or returning infinity.

Key Points to Mention

  • Two-sided two-sample z-test formula for sample size calculation
  • Use of historical data to estimate variance (pooled or individual)
  • Critical values from standard normal distribution for significance level and power
  • Minimum detectable effect (MDE) as the absolute difference to detect
  • Rounding up to ensure sufficient sample size
  • Edge cases: zero variance, invalid parameters, and assumptions of normality

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

Implement a function that computes P(A|B) given P(A), P(B|A), and P(B|not A) using Bayes' rule.

Product Analytics & Metrics
Author's notes

Pretty mechanical once you write out the law of total probability for P(B).

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

First, restate Bayes' rule and identify the given probabilities. Then, compute P(B) using the law of total probability: P(B) = P(B|A)P(A) + P(B|not A)P(not A). Finally, plug into Bayes' rule to get P(A|B).

Pro tip: Mention edge cases like P(B)=0 and how to handle them, and relate the formula to a real-world example (e.g., A/B testing at Roblox) to show practical understanding.

1. Restate Bayes' Rule

Write down Bayes' theorem: P(A|B) = P(B|A) * P(A) / P(B). This sets the foundation.

2. Compute P(B)

Use the law of total probability: P(B) = P(B|A)P(A) + P(B|not A)P(not A), where P(not A) = 1 - P(A).

3. Plug into Bayes' Rule

Substitute the computed P(B) and given values into the formula to calculate P(A|B).

4. Handle Edge Cases

Check if P(B) = 0; if so, P(A|B) is undefined. Mention this to show robustness.

5. Validate with Example

Optionally, test with a simple numerical example to ensure the implementation works as expected.

Key Points to Mention

  • Bayes' theorem formula
  • Law of total probability
  • Complement rule: P(not A) = 1 - P(A)
  • Edge case: P(B) = 0
  • Numerical stability (e.g., avoiding division by zero)
  • Real-world application (e.g., A/B testing, spam filtering)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.