Start by clarifying the product change and defining success metrics for both sides of the marketplace. Then outline a randomized experiment design with proper power analysis, and finish with a decision rule that translates metric lifts into dollar value.
Pro tip: Emphasize that in a two-sided marketplace, you must consider spillover effects and network effects; randomize at the guest level but analyze at the listing level to avoid contamination.
State the hypothesis that the High Wi-Fi badge/filter improves booking conversion for remote workers. Define primary metric (e.g., booking conversion rate for remote-worker segment) and guardrail metrics (e.g., overall conversion, host satisfaction).
Randomize at the guest level to avoid spillover, but ensure listings are not double-counted by using a cluster randomized design if needed. Consider stratifying by guest type (remote worker vs. leisure) and listing characteristics.
Calculate required sample size based on minimum detectable effect (MDE) for the primary metric, using historical variance and desired power (80%) and significance level (5%). Account for two-sided marketplace by powering for both guest and host outcomes.
Pre-register analysis: use intent-to-treat, compare primary metric via t-test or regression with covariates. Check for novelty effects, segment by remote-worker status, and monitor guardrail metrics for degradation.
Estimate incremental revenue from lift in bookings, subtract implementation and maintenance costs, and compute net present value. Launch if NPV > 0 and guardrails are not violated, with a threshold (e.g., ROI > 20%).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.