Thumbtack·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say

Jun 2026Remote

Summary

Thumbtack DS interview that went deep into marketplace experimentation design. The question was essentially one long, multi-part case about running an A/B test on a pro ranking algorithm, and they expected you to hold the whole thing together across metric definition, randomization tradeoffs, power math, and guardrails. Dense but fair.

Questions Asked (1)

Thumbtack is updating the algorithm that ranks pros in search results for customer requests. Design an experiment to evaluate the new ranker, covering your primary metric, guardrails with formulas, randomization strategy, power calculations, bias controls, sequential monitoring, and how you'd read out results including cannibalization effects.

A/B Testing & ExperimentationProduct Analytics & MetricsSystem Design

Author's notes

This was basically six questions stapled together and they wanted you to move through all of them without losing the thread.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer around a clear experimental design: define the primary metric (e.g., request-to-quote rate) and guardrails (e.g., customer satisfaction, pro retention), then detail randomization, power analysis, bias controls, and sequential monitoring. Emphasize how you'd detect cannibalization and interpret results for a two-sided marketplace.

Pro tip: In marketplace experiments, always consider interference between the two sides (customers and pros) and use cluster randomization if needed; also pre-register your analysis plan to avoid p-hacking.

1. Define Metrics and Hypotheses

Specify the primary metric (e.g., request-to-quote rate) and guardrail metrics (e.g., customer satisfaction, pro retention, revenue). Formulate clear hypotheses for the new ranker's impact.

2. Design Randomization and Power

Choose randomization unit (e.g., customer request or pro) and strategy (e.g., cluster randomization if interference). Calculate required sample size using power analysis for the primary metric.

3. Control for Bias and Monitor Sequentially

Implement bias controls like stratification and pre-registration. Set up sequential monitoring with alpha spending to allow early stopping for efficacy or futility.

4. Analyze Results and Cannibalization

Compare primary and guardrail metrics between control and treatment. Assess cannibalization by examining shifts in pro engagement or customer behavior across segments.

Key Points to Mention

Primary metric: request-to-quote rate or booking rate; guardrails: customer satisfaction (CSAT), pro retention, revenue.
Randomization unit: customer request or pro; consider cluster randomization to handle interference.
Power analysis: determine sample size using baseline rate, minimum detectable effect, alpha, and power.
Bias controls: stratification, pre-registration, and avoiding peeking.
Sequential monitoring: alpha spending functions (e.g., O'Brien-Fleming) for interim analyses.
Cannibalization: measure impact on pros' overall engagement and customer repeat rates; use holdout groups if needed.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.