# Sample ratio mismatch (SRM): what it is and which tools check for it

> A sample ratio mismatch (SRM) is when the versions of an A/B test get a different share of visitors than you configured, such as 10,000 against 10,600 on a 50/50 split. It means something other than randomisation decides who is counted where, so the result can't be trusted. [Optimizely](https://duelpilot.com/compare/duelpilot-vs-optimizely), [GrowthBook](https://duelpilot.com/compare/duelpilot-vs-growthbook), [Statsig](https://duelpilot.com/compare/duelpilot-vs-statsig), Spotify's Confidence and Duelpilot check for it automatically; for any other tool, an SRM checker tests your counts.

Last reviewed: 2026-10-29 · HTML version: https://duelpilot.com/guides/sample-ratio-mismatch

> **Definition:** A sample ratio mismatch is a statistically significant difference between the share of visitors each version received and the share it was meant to receive. Small random differences are normal; an SRM is a difference too large to be chance.

## Why an SRM invalidates the result

An A/B test works because randomisation makes the groups alike in everything except the change. If one version loses or gains visitors for another reason, the groups are no longer alike, and the difference in conversion can come from who was counted rather than from the change. The mismatch says nothing about which version is better; it says the comparison is broken.

It is not rare. The SRM Checker FAQ cites about 6% of experiments at Microsoft and about 10% of triggered experiments at LinkedIn, and notes that both companies had worked on root causes for years ([SRM Checker FAQ](https://www.lukasvermeer.nl/srm/docs/faq/)). Drawing on experience at four software companies, Fabijan and colleagues wrote a taxonomy of causes and rules of thumb for diagnosing them ([KDD 2019 paper](https://www.microsoft.com/en-us/research/?p=651990)).

## How to test for SRM

You need two things: the split you intended and the number of visitors each version actually got. A chi-squared goodness-of-fit test then gives a p-value for the observed counts under the intended split. Because you check every test, and often more than once, use a strict threshold: GrowthBook warns below p = 0.001 by default, and Spotify's Confidence flags below about 0.001.

```python
from scipy.stats import chisquare

# 50/50 split, 10,000 vs 10,600 visitors
chisquare([10000, 10600])
# statistic ≈ 17.5, p ≈ 0.00003  -> sample ratio mismatch
```

No code needed: Lukas Vermeer's SRM Checker runs the same test on summary counts from any tool, since it only needs the expected proportions and the observed counts ([SRM Checker](https://www.lukasvermeer.nl/srm/)).

### Checking repeatedly

A chi-squared test assumes one look. Checking it every day turns a strict threshold into a looser one, the same peeking problem as with conversion results (see [sequential testing](https://duelpilot.com/guides/sequential-testing)). Sequential SRM tests stay valid however often they run. Lindon and Malek describe anytime-valid sequential tests for multinomial counts ([arXiv](https://arxiv.org/abs/2011.03567)), and Optimizely's sequential SRM detection checks traffic counts continuously during an experiment ([Optimizely docs](https://docs.optimizely.com/experimentation-strategy/docs/optimizely-s-automatic-sample-ratio-mismatch-detection)).

## Common causes

- Redirect tests: the variant page loads slower or fails, or blockers stop the target, so visitors leave before they are counted.
- A variant that breaks or loads slowly before exposure is recorded, so exposures fire less often in that version.
- A trigger that depends on the variant: counting on "element visible" when the variant moves the element higher counts more visitors in the variant.
- Targeting that differs per version, for example a variant that changes a URL, query parameter or cookie that targeting reads.
- Bots or internal traffic concentrated in one version.
- Traffic allocation changed in the middle of a test.

## What to do when you find one

1. Don't ship on the result, in either direction.
2. Don't reweight or patch the data: the FAQ notes that correcting an SRM needs extra assumptions, and the bounds that avoid them are often too wide to help.
3. Find the cause: break the counts down by device, browser, day and page, and look for the version or segment where visitors go missing. Statsig, for example, automatically analyses the dimensions its SDK logs to point at likely drivers ([Statsig docs](https://docs.statsig.com/stats-engine/methodologies/srm-checks)).
4. Fix it and restart counting from zero. The old data was never a test of the change.

## Which tools check for SRM

Automatic SRM checks, as each vendor's documentation describes them (checked 29 October 2026). Other tools may check too; we list what we could verify.

| Tool | What its docs say |
|---|---|
| [Optimizely](https://duelpilot.com/compare/duelpilot-vs-optimizely) | Sequential SRM detection that checks continuously and runs daily, for Stats Engine A/B tests with manual traffic distribution, at least 1,000 visitors and up to 45 days of running time ([source](https://docs.optimizely.com/experimentation-strategy/docs/optimizely-s-automatic-sample-ratio-mismatch-detection)) |
| [GrowthBook](https://duelpilot.com/compare/duelpilot-vs-growthbook) | Every experiment is checked with a chi-squared test; a warning appears below p = 0.001, adjustable in the organisation settings ([source](https://docs.growthbook.io/app/experiment-results)) |
| [Statsig](https://duelpilot.com/compare/duelpilot-vs-statsig) | SRM checks on all experiments and feature gates as part of Health Checks, with a chi-squared test ([source](https://docs.statsig.com/stats-engine/methodologies/srm-checks)) |
| Spotify Confidence | An automatic SRM check on every experiment, chi-squared, flagged below about 0.001 ([source](https://confidence.spotify.com/glossary/sample-ratio-mismatch)) |
| Duelpilot | An always-valid sequential test on every check of a running test; below p = 0.001 the result is invalid and the test pauses (details below) |

## How Duelpilot checks it

- The check is an always-valid sequential test (a Dirichlet-multinomial mixture against the intended split, in the family Lindon and Malek describe), because the server re-checks running tests every minute. A one-off chi-squared test checked that often would raise false alarms, and each one would pause a test; the chi-squared value is still shown, for information.
- Below p = 0.001 (once a test has 200 visitors), the result is marked invalid with the server's reason and the test is paused automatically. Agents report the cause candidates; they never reinterpret or correct the data.
- The trade-off is power: a small mismatch needs more visitors to be flagged than a single chi-squared look. A 60/40 split shows by about 1,000 visitors, a 51/49 split by about 100,000.
- It tests against the configured weights. When a test follows your site's own random visitor groups instead of Duelpilot's split, the expected shares come from the group weights fixed at approval, and the results name the likely cause of a mismatch, such as a group missing on some pages.
- After the cause is fixed, a person resets the results: a new run that counts from zero, logged with a reason.

## Try Duelpilot with your own agent

Give your agent one line: [set up Duelpilot for my site using /agents.md](https://duelpilot.com/agents.md). It opens a workspace, installs and verifies the script, and drafts the first test; you approve it, then switch on autopilot if you like. Or [see pricing](https://duelpilot.com/#pricing).

**[Start for €0](https://duelpilot.com/login) · [Give it to your agent](https://duelpilot.com/#start)** (€0 until you launch your first test)

## FAQ

### What p-value threshold should I use for an SRM check?

A strict one, because you check every test and often repeatedly: GrowthBook and Spotify's Confidence use about 0.001. If you check daily, a sequential SRM test keeps that threshold honest however often you look.

### Does a sample ratio mismatch mean my variant lost?

No. It says nothing about which version is better. It means the groups are no longer comparable, so neither a win nor a loss can be trusted until the cause is found and the test restarted.

### Can I correct the data after an SRM?

Not without extra assumptions. Reweighting the arms ignores why visitors went missing, which is the problem. Find the cause, fix it and restart the test.

### Why do redirect tests show SRM so often?

The variant is a second page load: some visitors leave, are blocked or fail to load it before they are counted, so the variant ends up with fewer visitors than the original.

## Sources

- [Home | SRM Checker (Lukas Vermeer)](https://www.lukasvermeer.nl/srm/) (checked 2026-10-29)
- [Frequently Asked Questions | SRM Checker](https://www.lukasvermeer.nl/srm/docs/faq/) (checked 2026-10-29)
- [Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners (Fabijan et al., KDD 2019)](https://www.microsoft.com/en-us/research/?p=651990) (checked 2026-10-29)
- [Anytime-Valid Inference for Multinomial Count Data (Lindon, Malek)](https://arxiv.org/abs/2011.03567) (checked 2026-10-29)
- [Optimizely's automatic sample ratio mismatch detection](https://docs.optimizely.com/experimentation-strategy/docs/optimizely-s-automatic-sample-ratio-mismatch-detection) (checked 2026-10-29)
- [Experiment Results | GrowthBook Docs](https://docs.growthbook.io/app/experiment-results) (checked 2026-10-29)
- [SRM Checks | Statsig Docs](https://docs.statsig.com/stats-engine/methodologies/srm-checks) (checked 2026-10-29)
- [What is a Sample Ratio Mismatch? | Confidence](https://confidence.spotify.com/glossary/sample-ratio-mismatch) (checked 2026-10-29)

## Related

- [Peeking and sequential testing](https://duelpilot.com/guides/sequential-testing)
- [Reading A/B test results in GA4](https://duelpilot.com/guides/ga4-ab-test-results)
- [Best A/B testing tools](https://duelpilot.com/best-ab-testing-tools)
- [Agentic A/B testing explained](https://duelpilot.com/agentic-ab-testing)