← Thumbtack Interview Insights

Thumbtack·Data Scientist·Technical Phone Screen·Senior

Senior
May 2026

Summary

Thumbtack data science interview that went deep on experimentation design for a marketplace matching model. The question was essentially one giant multi-part case and it covered pretty much every corner of A/B testing you can think of.

Questions Asked (1)

Q1

You have a new matching model ready to ship. Design a full A/B test for it, covering primary metrics and guardrails for both sides of the marketplace, randomization strategy, power analysis, variance reduction and validity checks, a pre-registration document with stop/go criteria, and how you'd analyze heterogeneous lift without p-hacking.

A/B Testing & ExperimentationProduct Analytics & MetricsProduct Strategy
Author's notes

This was a lot.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the experiment around the two-sided marketplace, defining success metrics for both customers and professionals, then walk through the design choices (randomization, power, variance reduction) and operational rigor (pre-registration, stop/go criteria). Finally, explain how you'd analyze heterogeneous treatment effects with pre-specified subgroups and appropriate corrections to avoid p-hacking.

Pro tip: Emphasize that in a two-sided marketplace, you must randomize at the level that minimizes interference (e.g., by customer or professional, not by request) and consider cluster randomization if spillovers are likely. Also, pre-register not just the primary metric but also the exact subgroup analyses and multiple-testing correction to build trust.

1. Define metrics and hypotheses

Specify the primary metric (e.g., match rate or bookings) and guardrails for both sides (e.g., customer satisfaction, professional utilization, cancellation rates). Clearly state the null and alternative hypotheses for the overall effect and key subgroups.

2. Design randomization and power

Choose the randomization unit (e.g., customer, professional, or geographic cluster) to minimize interference. Conduct power analysis to determine sample size and duration, accounting for expected effect size, baseline variance, and intra-cluster correlation if applicable.

3. Apply variance reduction and validity checks

Use techniques like CUPED or stratification to reduce variance and increase sensitivity. Implement validity checks: sample ratio mismatch (SRM), pre-experiment covariate balance, and checks for novelty effects or seasonality.

4. Pre-register and set stop/go criteria

Write a pre-registration document detailing the experiment design, metrics, analysis plan, and decision rules (e.g., ship if primary metric improves by X% with p<0.05 and guardrails not degraded). Include sequential testing or alpha spending if interim looks are planned.

5. Analyze heterogeneous lift without p-hacking

Pre-specify subgroups (e.g., new vs. returning customers, professional tenure) and use interaction tests or causal forests with multiple-testing correction (e.g., Benjamini-Hochberg). Report effect sizes with confidence intervals and avoid data dredging by sticking to the pre-registered plan.

Key Points to Mention

  • Two-sided marketplace metrics: define primary and guardrail metrics for both customers (e.g., match rate, booking rate, satisfaction) and professionals (e.g., lead quality, utilization, retention).
  • Randomization strategy: choose unit (customer, professional, or cluster) to avoid interference; consider switchback or geo-based randomization if spillovers are severe.
  • Power analysis: calculate sample size based on minimum detectable effect (MDE), baseline conversion, and variance; account for clustering and multiple comparisons.
  • Variance reduction: use CUPED, stratification, or regression adjustment with pre-experiment covariates to increase power.
  • Validity checks: SRM, covariate balance, A/A tests, and monitoring for novelty/primacy effects; ensure no peeking without correction.
  • Pre-registration and stop/go: document hypotheses, metrics, analysis plan, and decision criteria (e.g., ship if primary metric lifts ≥2% with p<0.05 and guardrails non-inferior); include futility stopping rules.
  • Heterogeneous lift analysis: pre-specify subgroups, use interaction tests or causal forests, correct for multiple testing (e.g., Benjamini-Hochberg), and report confidence intervals; avoid post-hoc subgroup fishing.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.