Microsoft·Technical Product Manager·Technical Phone Screen·Senior

Senior

May 2026

Summary

TPM interview at Microsoft focused on experimentation design for Bing. Just one question but it had real depth to it, more of a product thinking exercise than a technical one.

Questions Asked (1)

You're designing an A/B test for Bing. When would you run a user-tied experiment (where users are consistently in treatment or control) versus an untied experiment (where each individual search is randomly assigned)? Walk through the tradeoffs of each approach.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

I knew the basic definitions but fumbled a bit articulating the actual tradeoffs.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the core difference: user-tied experiments assign the same user to a consistent variant, while untied experiments randomize each search independently. Then, discuss when each is appropriate based on the metric of interest, the potential for interference or carryover effects, and the tradeoff between statistical power and user experience consistency. Finally, recommend a decision framework that balances these factors for Bing's search context.

Pro tip: Emphasize that the choice hinges on whether the treatment effect is expected to persist across sessions or searches; for metrics like user retention or long-term engagement, user-tied is essential, but for immediate relevance metrics, untied can provide faster insights with higher power.

1. Clarify the objective and metrics

Identify whether the experiment aims to measure short-term per-search metrics (e.g., click-through rate) or long-term user-level metrics (e.g., retention, lifetime value). This determines the appropriate randomization unit.

2. Assess interference and carryover effects

Consider if the treatment effect can spill over between searches or sessions (e.g., user learning, fatigue). If so, user-tied avoids contamination; otherwise, untied may be acceptable.

3. Evaluate statistical power and sample size

Untied experiments often have more independent observations, increasing power and reducing variance. User-tied experiments may require larger sample sizes or longer durations due to clustering.

4. Consider user experience and consistency

User-tied ensures a consistent experience, which is crucial for features that alter the UI or interaction model. Untied can lead to jarring inconsistencies if the treatment changes the user interface.

5. Make a recommendation and plan for analysis

Choose the approach that best aligns with the primary metric and business goal, and specify the analysis method (e.g., cluster-robust standard errors for user-tied) to ensure valid inference.

Key Points to Mention

Randomization unit: user vs. search (or session).
Interference and carryover effects: how treatment in one search can affect subsequent searches.
Statistical power and variance: tradeoff between independent observations and clustering.
User experience consistency: avoiding mixed experiences within a session.
Metric sensitivity: short-term vs. long-term metrics and their alignment with randomization unit.
Analysis considerations: need for cluster-robust standard errors or mixed-effects models in user-tied experiments.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.