I started talking about running a chi-square test on the group sizes and checking for balance across key user attributes, which felt right.
Start by defining what 'random' means in the context of A/B testing—each unit has an equal and independent chance of being assigned to any group. Then outline a multi-faceted verification strategy that includes pre-experiment checks (e.g., sample ratio mismatch, covariate balance) and post-hoc diagnostics (e.g., distribution of p-values, A/A tests). Emphasize that verification should be automated and integrated into the experimentation pipeline.
Pro tip: Mention that even with a perfect random assignment algorithm, implementation bugs (e.g., logging errors, bot traffic) can break randomness, so you should also check for data quality issues and consider using a holdout group for long-term validation.
Compare the observed group sizes to the expected ratio using a chi-squared test. A significant deviation (p < 0.001) indicates a potential assignment or logging issue.
For key pre-experiment covariates (e.g., demographics, past behavior), compare distributions across groups using standardized mean differences or statistical tests. Randomization should yield balanced groups.
Run an experiment where both groups receive the same experience. The distribution of p-values should be uniform, and the false positive rate should match the significance level.
Review the code or system that assigns units to groups. Ensure it uses a proper random seed, has no deterministic patterns, and is resistant to manipulation (e.g., user ID hashing).
Check for unexpected differences in metrics that should not be affected by the treatment (e.g., pre-experiment metrics). Also, look for anomalies in assignment over time or across segments.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.