Thumbtack·Data Scientist·Technical Phone Screen·Senior

Senior

Jul 2026

Summary

Thumbtack data scientist interview with a deep project walkthrough question that covers basically every angle you can imagine. Felt like one question but was really ten questions stapled together.

Questions Asked (1)

Walk me through your most impactful project from start to finish: how did you define the problem and success metric, what approaches did you consider and rule out, how did you handle feature selection and data leakage, what was your validation strategy offline vs. online, what trade-offs did you make under constraints, and what would you do differently now?

Product Analytics & MetricsTechnical Trade-offsA/B Testing & Experimentation

Author's notes

This is a beast of a question and I did not pace myself well.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Choose a project where you can clearly articulate the business problem, your technical decisions, and the measurable impact. Structure your answer as a narrative that walks through each phase of the project, emphasizing the trade-offs you made and the lessons learned. Highlight how you balanced technical rigor with business constraints, and be honest about what you would improve.

Pro tip: Quantify the impact of your project with specific metrics (e.g., 'increased conversion by 5%' or 'reduced churn by 10%') and tie it back to the company's key performance indicators. Also, mention how you collaborated with cross-functional teams to ensure alignment and adoption.

1. Define the Problem and Success Metrics

Start by explaining the business context and the specific problem you aimed to solve. Clearly state the success metric(s) you defined, how you chose them, and how they aligned with broader business goals.

2. Explore Approaches and Rule Out Options

Describe the different modeling approaches or solutions you considered. Explain why you ruled out certain options based on constraints like data availability, interpretability, or time, and why you chose the final approach.

3. Handle Feature Selection and Data Leakage

Discuss how you selected features, including any domain knowledge or automated methods used. Explain how you identified and prevented data leakage to ensure your model would generalize to unseen data.

4. Validation Strategy: Offline and Online

Detail your offline validation approach (e.g., cross-validation, holdout set) and how you designed the online experiment (e.g., A/B test). Mention how you ensured the offline metrics correlated with online performance.

5. Trade-offs, Constraints, and Reflections

Explain the trade-offs you made under constraints such as time, resources, or technical limitations. Conclude with what you would do differently now, showing growth and learning.

Key Points to Mention

Alignment of success metrics with business KPIs (e.g., conversion rate, customer acquisition cost)
Consideration of multiple modeling approaches and rationale for the chosen one
Techniques for feature selection (e.g., regularization, feature importance) and data leakage prevention (e.g., time-based splits)
Offline validation methods (e.g., cross-validation, holdout) and online experimentation (e.g., A/B test design, power analysis)
Trade-offs made due to constraints (e.g., model simplicity vs. accuracy, speed vs. performance)
Lessons learned and what you would do differently (e.g., better data collection, different model, earlier stakeholder involvement)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.