Can you check A/B test results every day? Peeking and sequential testing explained

Why peeking breaks a classic A/B test

A classic (fixed-horizon) p-value assumes you chose the sample size in advance and look once, at the end. Every extra look is another chance for random noise to cross the significance line. If you stop the first time it does, the share of tests that declare a winner when nothing changed grows with every look.

Evan Miller's well-known example: a 50% conversion rate, a new logo that does nothing, a check after every visitor and a stop at 5% significance or at 150 visitors. About 26.1% of those experiments wrongly find a significant result, more than five times the 5% the analyst thinks they have (How Not To Run an A/B Test). Checking after every visitor is the worst case, but a daily dashboard over a month is a lot of looks too. His remedy for fixed-horizon tests: fix the sample size in advance and don't act before you reach it.

What sequential testing is

A sequential test controls the error rate over all the looks you take, not just one. Some designs allow a fixed number of planned interim looks. The more flexible kind, always-valid inference, allows any number of looks at any moment, so a person or an agent can open the dashboard daily, or every minute, without changing what the numbers mean.

Always-valid p-values and confidence sequences

Johari, Pekelis and Walsh define always-valid p-values and confidence intervals that give valid inference whenever you decide to stop, even if you continuously monitor the test, built on the mixture sequential probability ratio test (mSPRT). The authors worked with Optimizely on it, and the paper says the method was implemented in a large commercial A/B testing platform (Always Valid Inference). Two ideas carry it:

• An always-valid p-value only goes down over time (it is a running minimum), and the chance that it ever drops below 5% when there is no effect stays at most 5%, however often you look.
• A confidence sequence is an interval that holds at every look at once. It is wider than a fixed-horizon interval at the same sample size; that width is the price of being allowed to look.

The cost: more visitors

For the same power, a sequential test needs a larger sample than a fixed-horizon test that is looked at once. In return it can stop earlier when the effect is large, and you never have to pretend you didn't look. Plan the sample size with the inflation included.

What sequential testing does not fix

• Changing the primary metric after seeing the data, or picking the best-looking segment afterwards. That is still cherry-picking; segment results are exploratory unless you named the segment before launch.
• Weekday and novelty effects. A test stopped on a Thursday measured a different population from one that ran a full week. Valid statistics don't make a short test representative, so decide on whole weeks.
• Several variants. Each comparison with control is another chance of a false winner; correct for it (Holm, Bonferroni or a false-discovery-rate method).
• A broken split. If the arms don't get the share of visitors you configured, no statistics can save the result. See sample ratio mismatch.

How to look every day safely

1. Find out which statistics your tool uses. Its documentation should say fixed-horizon, sequential or Bayesian, and whether sequential is the default or an option.
2. With a fixed-horizon test, compute the sample size before launch and decide only when you reach it. Looking in between is fine as long as nothing you see changes when you stop.
3. With a sequential test, look as often as you like, but still decide on whole weeks and with a minimum sample.
4. Write down the primary metric, the guardrails and any segment you expect to react before launch.
5. Stop early only for harm or breakage, never because a variant is winning.

Tools that document sequential statistics

How Duelpilot does it

Duelpilot is run by your own AI agent, and agents check results every day, so it has one engine and no switch: an always-valid sequential test (mSPRT) per variant against control, on the cumulative daily totals, for conversion rate and for revenue per visitor.

• Always-valid p-values and confidence sequences on the lift, with Holm correction across variants.
• A sample ratio mismatch check that is itself sequential, so the server can check every minute without false alarms piling up; a mismatch marks the result invalid and pauses the test.
• Harm detection at 99% on the primary metric and every guardrail, which can pause a test early. Winning never ends a test early.
• Readiness rules: a verdict only at a full-week boundary, after at least 7 days, with at least 1,000 visitors in the smallest arm and 100 control conversions; at the maximum duration (42 days by default) without a significant difference, the result is inconclusive.
• Sample-size planning inflates the fixed-horizon estimate by 1.4 for the sequential test, so the plan your agent shows already includes the cost.
• A Bayesian chance to beat control is shown for people who like probabilities, never used to decide. Agents are told not to compute or reinterpret any statistic: they report the server's verdict.

What it doesn't have: a choice of engine, fixed-horizon or Bayesian decisions, or variance reduction such as CUPED. If you want to choose your statistics per test, Convert, GrowthBook, Statsig and PostHog document that choice.

Try Duelpilot with your own agent