Thumbtack·Data Scientist·Technical Phone Screen
Jun 2026
Thumbtack DS interview that went deep into marketplace experimentation design. The question was essentially one long, multi-part case about running an A/B test on a pro ranking algorithm, and they expected you to hold the whole thing together across metric definition, randomization tradeoffs, power math, and guardrails. Dense but fair.
- Thumbtack is updating the algorithm that ranks pros in search results for customer requests. Design an experiment to evaluate the new ranker, covering your primary metric, guardrails with formulas, randomization strategy, power calculations, bias controls, sequential monitoring, and how you'd read out results including cannibalization effects.
“This was basically six questions stapled together and they wanted you to move through all of them without losing the thread.” The rest of the author's notes on Data Scientist interview at Thumbtack, Technical Phone Screen round, covers how they worked through the question, what the panel pushed back on, and what they would do differently.
View Post