Sample Ratio Mismatch (SRM)

Sample Ratio Mismatch (SRM) is a data quality problem in A/B testing that occurs when the actual distribution of visitors between test variants is statistically different from the intended distribution.

Example: A 50/50 test intended to split traffic equally between control and variant. After 10,000 sessions, control received 5,400 visitors and variant received 4,600 — a 54/46 split instead of 50/50. This mismatch indicates that the randomization mechanism is broken, and the test results cannot be trusted.

SRM is one of the most insidious A/B testing problems because the conversion data can look perfectly plausible — you might see a confident 95% significant result that is entirely an artifact of population bias, not a real effect.

Why SRM Invalidates Test Results

The entire validity of an A/B test rests on the assumption that variants receive equivalent, randomly selected visitor populations. If the split is uneven, the populations in each variant are not equivalent — systematic differences may exist between them that have nothing to do with the variant change.

Example of how SRM creates false results: A redirect-based test sends users to a new URL for the variant. If the variant page loads 500ms slower (triggering the redirect), visitors with slow connections may disproportionately abandon before being assigned to the variant — leaving only fast-connection visitors in the variant sample. The variant appears to convert better, but only because it systematically excluded slow-connection visitors.

This type of SRM produces a false positive: a statistically significant “win” that will not replicate in production.

Detecting SRM: The Chi-Squared Test

A chi-squared (χ²) goodness-of-fit test measures whether the observed traffic distribution differs significantly from expected:

For a 50/50 test with 10,000 total sessions:

• Expected: 5,000 control, 5,000 variant
• Observed: 5,400 control, 4,600 variant
• Chi-squared statistic: (5,400−5,000)²/5,000 + (4,600−5,000)²/5,000 = 32 + 32 = 64
• At 1 degree of freedom, χ² = 64 is far beyond the p < 0.05 threshold (critical value: 3.84)

SRM is confirmed.

Most enterprise A/B testing platforms (Optimizely, VWO, AB Tasty) run this check automatically and flag SRM in the dashboard. If your platform doesn’t flag it, run the calculation manually at the end of every test.

Quick rule of thumb: Any deviation greater than ±2% from the intended split on a large sample (10,000+ sessions) warrants investigation, even if not statistically significant.

Common SRM Causes and How to Diagnose Them

Cause	How it creates SRM	Detection method
Bot traffic asymmetry	Bots may consistently hit one variant URL more	Compare bot-filtered vs unfiltered sessions per variant
Redirect timing / latency	Slow variant redirect causes visitor dropout before assignment	Compare variant load time vs control; check time-to-first-byte
CDN caching	Cached pages bypass randomization	Check cache-control headers; verify test pages are cache-excluded
JavaScript error in variant	Variant code fails to load for some browsers/devices	Check browser console errors; compare pre/post variant JS load
Mobile rendering failure	Variant CSS/JS breaks on mobile	Segment traffic split by device type separately
Cookie setting failure	Assignment cookie not set on some devices	Check cookie presence rate per variant across device types
Filtering inconsistency	Exclusion rules differ between variants	Audit segment filters; ensure identical targeting on both sides
Third-party script conflicts	Tag manager or analytics script fires differently	Audit all third-party scripts triggered by variant code

SRM Investigation Protocol

When SRM is detected, follow this sequence:

1. Confirm the SRM — run chi-squared test on raw session counts per variant
2. Segment by device — check split on mobile vs desktop separately; mobile-only SRM often indicates a rendering issue
3. Segment by browser — SRM in a specific browser points to JS or CSS conflicts
4. Check page load time per variant — any meaningful latency difference (>200ms) in a redirect test is a likely cause
5. Audit bot filtering — compare filtered vs unfiltered counts
6. Review targeting rules — confirm identical audience segments on both variants
7. Fix the root cause — do not proceed without resolving
8. Re-run the test from scratch — SRM-affected data cannot be corrected

SRM vs Normal Statistical Variation

Small deviations from the target split (50.5/49.5) are normal and expected — this is just random variation. SRM is a statistically significant deviation from the intended split.

Important timing note: Run your SRM check at the end of the test, not daily. Chi-squared tests on small interim samples frequently flag false SRMs — especially in the first 24–48 hours of a test when traffic is still accumulating. Checking SRM mid-test can cause unnecessary test terminations.

SRM in Multi-Variant Tests

In A/B/C or A/B/n tests with more than two variants, SRM detection uses the same chi-squared test with additional degrees of freedom. For a three-variant 33/33/33 split:

• Degrees of freedom = 2
• Each variant’s expected count = total sessions ÷ 3
• Critical chi-squared value at p < 0.05: 5.99

Any variant deviating meaningfully from equal share warrants the same investigation protocol.

For a full checklist of what can go wrong in A/B tests, see 7 A/B Testing Mistakes That Invalidate Your Results and A/B Testing Best Practices.

See also: Statistical Significance, P-Value, A/B Testing.