← Intuit Interview Insights

Intuit·Software Engineer·Technical Phone Screen·Intermediate

Intermediate
May 2026

Summary

Intuit product analyst interview with a meaty experimentation question built around their onboarding flow. One question, but it had a lot of layers and I definitely underestimated how much structure they expected.

Questions Asked (1)

Q1

The product team added a progress bar to a 4-step onboarding flow (onboard, send payment, set up bank account, file taxes). How would you evaluate whether it's actually helping? Walk through your hypothesis, the metrics you'd track, how you'd design the experiment, and any pitfalls you'd watch out for.

A/B Testing & ExperimentationProduct Analytics & MetricsProduct Sense & Ideation
Author's notes

I started okay with the hypothesis part, said something like 'helps means more users completing all four steps without dropping off midway.' But then I got a bit tangled on which metric to call primary vs diagnostic.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the goal of the progress bar—likely reducing drop-off and increasing completion rates—then structure your answer around a clear hypothesis, metrics, experiment design, and pitfalls. Emphasize a user-centric approach by considering how the progress bar affects motivation and perceived effort, and tie your evaluation to business outcomes like conversion and retention.

Pro tip: Acknowledge that the progress bar might have heterogeneous effects—some users may be motivated while others feel overwhelmed—so plan for segmented analysis and consider qualitative feedback to complement quantitative results.

1. Define the hypothesis and success criteria

Articulate a clear, testable hypothesis about how the progress bar will impact user behavior, such as reducing drop-off at each step and increasing overall completion rate. Define what success looks like in measurable terms.

2. Identify key metrics

Select primary metrics (e.g., step completion rate, overall funnel completion) and secondary metrics (e.g., time per step, error rates, user satisfaction) that capture both the intended and unintended effects of the progress bar.

3. Design the experiment

Propose an A/B test with a control group (no progress bar) and treatment group (with progress bar), ensuring random assignment, sufficient sample size, and a duration that captures the full onboarding cycle. Consider using a holdout group for long-term effects.

4. Analyze results and segment

Compare metrics between groups using statistical tests, and segment by user characteristics (e.g., new vs. returning, device type) to uncover heterogeneous effects. Look for statistically significant improvements and potential regressions.

5. Watch for pitfalls and iterate

Identify common pitfalls such as novelty effects, sample ratio mismatch, and confounding variables. Also consider qualitative feedback and potential negative impacts like increased anxiety or abandonment due to perceived length.

Key Points to Mention

  • Primary metric: overall onboarding completion rate; secondary metrics: step-by-step drop-off, time to complete, error rates.
  • A/B test design: random assignment, control vs. treatment, sample size calculation, and test duration covering full onboarding cycle.
  • Segmentation: analyze by user cohorts (e.g., new users, business size) to detect differential effects.
  • Pitfalls: novelty effect, sample ratio mismatch, seasonality, and unintended consequences like increased drop-off at later steps.
  • Qualitative methods: user interviews or surveys to understand why the progress bar helps or hinders.
  • Business impact: tie improvements to downstream metrics like tax filing completion and retention.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.