Multiple Comparisons Calculator
If I run 20 tests, how many will be significant by chance?
About one. At the usual 5% threshold each test has a one in twenty chance of a false positive, so twenty tests produce 1.00 expected false winners, and the chance of getting at least one is 64.15%. A single significant result out of twenty is therefore exactly what nothing looks like. To keep the overall false-positive rate at 5% you would need a threshold of 0.25% per test, not 5%.
Test twenty things at the usual 5% threshold and one of them is expected to win on chance alone, even if every one of them does nothing.
That is not a criticism of any single test. It is what happens to a 5% error rate when you spend it twenty times.
Built by Muhammad Shahzeb Anwar, MSc Industrial and Organizational Psychology, who also builds the paid tools in this shop. The correction is well known in the sciences and almost never applied to a quarter's worth of marketing tests.
How many winners chance would have produced
Count every test you ran, not just the ones you wrote up. That includes tests you stopped early and segments you sliced afterwards.
Why twenty tests produce a winner on their own
A 5% significance threshold means you accept a one-in-twenty chance of calling a result real when it is not. That is a sensible bargain for one test.
Run twenty tests and you have spent that allowance twenty times. The expected number of false winners is simply the number of tests times the threshold: 20 × 0.05 = 1. The chance of getting at least one is 1 − 0.9520, which is 64%.
So a quarter in which you tried twenty things and one came out significant is indistinguishable from a quarter in which nothing worked at all. The result may still be real. It is simply not evidence yet.
The tests you forgot to count
The number that goes into this calculator is not the number of tests you wrote up. It is every comparison you made.
That includes the tests you abandoned when they looked flat. It includes each variant in a four-way test, which is three comparisons and not one. And it includes every slice you looked at afterwards — by device, by country, by new against returning — because each of those is another chance for noise to clear the bar.
A single A/B test cut six ways is seven comparisons, and the chance of at least one false winner is 30%, not 5%. This is the most common way a team ends up confidently acting on nothing.
What to do about it
The blunt fix is the Bonferroni correction: divide the threshold by the number of tests. Twenty tests at an overall 5% means judging each one at 0.25%. It is conservative, and it is easy to explain, which in a business setting counts for a lot.
The better fix costs nothing statistically: pick the most promising result and run that one test again, on its own, with the sample size fixed before it starts. A finding that survives a clean replication needs no correction at all.
What does not work is deciding after the fact that the winner was the one you expected all along. Twenty tests give twenty chances to find a story, and the story is always available afterwards.
What this will not tell you
This assumes the tests are independent and that each was judged once, at a threshold fixed in advance. Tests on overlapping traffic are not independent, and a test that was watched and stopped when it looked good has a false-positive rate well above its stated threshold before any of this arithmetic starts. Correcting for multiple comparisons does not repair either of those.
Questions people actually ask
How many false positives should I expect from 20 tests?
About one. Expected false winners are the number of tests times the threshold, so 20 tests at 5% gives 1.00, and the chance of at least one is 1 - 0.95^20, or 64.15%.
What is the Bonferroni correction?
Divide the significance threshold by the number of tests. For 20 tests at an overall 5% rate, judge each test at 0.25%. It is conservative - it makes real effects harder to detect - but it is simple and it holds the overall false-positive rate where you wanted it.
Do subgroup analyses count as extra tests?
Yes. Every slice you look at is another comparison and another chance for noise to clear the threshold. One A/B test examined across six segments is seven comparisons, and the chance of a false winner is about 30% rather than 5%.
Does this mean my significant result is wrong?
No. It means the result is not yet evidence. The way to find out is to run that single test again on its own with the sample size fixed in advance; a finding that replicates needs no correction.
Cite this tool
Free to use and free to quote. If a figure here is useful in something you are writing, a link back is all that is asked.
Muhammad Shahzeb Anwar, MSc Industrial and Organizational Psychology. Multiple Comparisons Calculator. Shahzeb Speaks. https://www.shahzebspeaks.com/multiple-comparisons-calculator/
The other free tools
Each one answers a single question and refuses the ones it cannot. All free, none asks for an email address.
Was the sample size fixed before the test started?
The paid A/B Test Read-Out asks that question first, because a test that was watched and stopped when it looked good has an inflated false-positive rate before multiple comparisons are considered at all. It reads out a single test properly; this page handles the pile of them.
See the paid tool