The 45-minute order lifecycle plus 30-minute carryover horizon was the key anchor.
Start by defining the trade-off between slot length and statistical power, then propose a block-randomization scheme that balances day-of-week and peak-hour effects while preventing contamination. Emphasize the need to validate assumptions with historical data and pre-register the analysis plan.
Pro tip: Use a balanced incomplete block design (BIBD) to ensure each treatment appears equally often in each day-of-week and peak-hour stratum, which increases precision without requiring all combinations. Also, consider using a washout period between slots to mitigate carryover effects.
Clarify the primary metric (cold-food incidents per delivery) and constraints like operational feasibility, sample size, and potential contamination sources. Determine the minimum detectable effect (MDE) and desired power.
Balance statistical power and bias: shorter slots increase sample size but may introduce carryover effects; longer slots reduce contamination but may be confounded with time trends. Use historical data to simulate power under different L and select the shortest L that meets power requirements while allowing for washout.
Stratify by day-of-week and peak-hour periods (e.g., lunch, dinner) to create homogeneous blocks. Within each block, randomly assign treatment/control to time slots, ensuring balance across strata. Consider a switchback design where each city switches between treatment and control multiple times.
Implement a washout period between slots to reduce carryover effects. Ensure couriers and customers are not exposed to both conditions simultaneously by using city-level randomization and avoiding overlapping delivery zones. Monitor for spillover via geolocation data.
Pre-register the analysis plan, including how to handle time trends and autocorrelation. Use mixed-effects models with random effects for city and time block. Validate assumptions with placebo tests and sensitivity analyses.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
First clarify that slot-level assignment (intention-to-treat) analyzes outcomes based on the courier's originally assigned slot, while realized exposure analyzes outcomes based on the slots where couriers actually delivered. Then explain how spillover demand from adjacent ON slots contaminates the OFF group's realized exposure, biasing the cold-food rate numerator (e.g., more cold food due to longer distances) and denominator (e.g., different order mix), and discuss the trade-offs between ITT and treatment-on-treated estimates.
Pro tip: Emphasize that ITT preserves randomization and avoids selection bias, but may underestimate the true effect if spillover is present; consider using instrumental variables or exposure-based analysis to estimate the causal effect of being in an ON slot, while acknowledging the assumptions.
Explain that slot-level assignment randomly assigns couriers to ON or OFF slots, while realized exposure is the actual slot state when a delivery is made. Highlight that spillover occurs when OFF-slot couriers pick up orders from adjacent ON slots.
Discuss how spillover increases the numerator of the cold-food rate in the OFF group because those orders may travel longer distances or experience delays, leading to more cold food incidents. This contaminates the OFF group's realized exposure.
Explain that the denominator (total orders) for the OFF group may also change because spillover orders are included, altering the order mix and potentially the baseline cold-food rate. This can bias the comparison if not accounted for.
Contrast the ITT effect (based on assigned slots) with the realized exposure effect (based on actual slots). ITT provides an unbiased but diluted estimate of the treatment effect, while realized exposure may be biased due to selection and spillover.
Suggest methods to handle spillover, such as using ITT as the primary analysis, employing instrumental variables or CACE (Complier Average Causal Effect) to estimate the effect of actual exposure, or adjusting for spillover via modeling.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
cold_food_rate ~ treatment + city_fe + slot_of_week_fe + weather + surge_index + courier_mix_pct, clustered SEs at city x slot.
Start by writing the regression equation with city and slot-of-week fixed effects, and cluster-robust standard errors at the city-by-slot level. Then explain how to incorporate pre-period baselines and time-varying covariates like weather, surge, and courier mix, ensuring they are measured consistently and included appropriately. Emphasize the importance of clustering at the level of treatment assignment to account for within-cluster correlation.
Pro tip: When clustering, ensure the number of clusters is sufficiently large (typically >50) to avoid biased standard errors; if not, consider wild cluster bootstrap. Also, always check for balance in covariates pre-experiment to justify their inclusion.
Write the model with the outcome variable, treatment indicator, city fixed effects, and slot-of-week fixed effects. Include an intercept and error term.
State that standard errors will be clustered at the city-by-slot level to account for correlation within these groups.
Include pre-period averages of the outcome (or other relevant metrics) as covariates to control for baseline differences and increase power.
Add covariates such as weather, surge pricing, and courier mix, ensuring they are measured at the appropriate level (e.g., city-by-slot-by-time) and are not affected by treatment.
Check for multicollinearity, assess model fit, and interpret coefficients, especially the treatment effect, while considering the fixed effects and clustering.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The DEFF from clustering is 1 + (n-1)*ICC = 1 + 119*0.02 which is around 3.38.
First, clarify that this is a switchback (cluster randomized) design where each city is a cluster and the unit of analysis is the slot. Compute the number of slots needed using the standard formula for cluster randomized trials, adjusting for ICC and cluster size. Then convert slots to ON/OFF switchbacks per city by dividing the total slots by the number of cities and accounting for the fact that each switchback period includes both ON and OFF conditions.
Pro tip: Always state your assumptions explicitly (e.g., equal cluster sizes, no time trends) and mention that in practice you would simulate the design to verify power, especially with small numbers of clusters.
Recognize that this is a cluster randomized switchback: each city is a cluster, each slot is an observation. The baseline rate is 6%, MDE is 0.6 percentage points (relative reduction of 10%), ICC = 0.02, cluster size = 120 orders per slot, and we have 14 days of data.
Calculate the number of observations needed per arm if randomization were at the order level. Use the standard formula for two proportions: n = (Z_{α/2} + Z_β)^2 * (p1(1-p1) + p2(1-p2)) / (p1-p2)^2, with p1=0.06, p2=0.054, α=0.05, power=0.80.
Multiply the individual sample size by the design effect (DE = 1 + (m-1)*ICC), where m is the cluster size (120 orders per slot). This gives the required number of slots per arm.
Determine the total number of slots needed across all cities. Since each switchback period consists of one ON and one OFF slot, the number of switchbacks per city is (total slots per arm) / (number of cities * 2). If the number of cities is not given, express the answer as a function of the number of cities.
Verify that the required number of switchbacks is feasible within 14 days (e.g., if each switchback is 1 day, you can have at most 14 per city). Discuss potential limitations such as time trends, carryover effects, and the need for simulation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Balance checks on pre-period cold-food rate, order volume, courier mix, and weather across ON vs OFF slots.
Start by outlining standard randomization checks (SRM, covariate balance, pre-period metrics) and then focus on the specific challenge of carryover effects in switchback designs. Explain how you would test for carryover by comparing outcomes in periods immediately following a switch versus later periods, and by using washout periods or modeling approaches.
Pro tip: Emphasize that carryover effects are a form of interference and can bias treatment estimates; propose a data-driven test like comparing the first day after switch to subsequent days, and consider using a washout period if effects persist.
Verify that the observed assignment ratio matches the intended ratio using a chi-squared test. Check for any anomalies in assignment over time or geographies.
Compare key metrics (e.g., order volume, delivery times) between treatment and control during the pre-experiment period to ensure no systematic differences. Use t-tests or non-parametric tests.
Segment the post-switch periods into immediate (e.g., first hour/day) and later periods. Compare treatment effects in immediate vs. later periods to detect decay or persistence of effects.
If carryover is suspected, exclude the initial period after each switch (washout) and re-estimate effects. Alternatively, model carryover explicitly using time-series or mixed-effects models.
Consider using a switchback design with longer blocks or a crossover design with sufficient washout. Validate findings with placebo tests or by simulating carryover.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Partial compliance I'd handle with an IV approach using slot assignment as the instrument for actual exposure.
Start by emphasizing that the integrity of the experiment and the validity of its causal inference are paramount. Then, walk through a structured decision framework that covers detection, diagnosis, and action, using specific examples from your experience. Conclude by highlighting the importance of pre-registered decision rules and stakeholder communication.
Pro tip: Frame your answer around minimizing bias and maximizing learning: 'The goal isn't to save the experiment, it's to save the truth.' Show that you prioritize long-term trust in experimentation over short-term wins.
Monitor for anomalies in real-time (e.g., sample ratio mismatch, missing telemetry, external events). Quickly diagnose the root cause and assess the severity and potential impact on the experiment's validity.
Determine whether the issue introduces bias, affects statistical power, or compromises the experiment's ability to answer the original question. Quantify the risk of false positives/negatives.
Apply pre-defined decision rules: stop if validity is compromised, extend if power is reduced but bias is absent, or rerun if the issue is systemic. Consider practical constraints like time and resources.
Transparently communicate the decision and rationale to stakeholders. Document the incident and learnings to improve future experiments and build trust.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.