A/B Test Read-Out

Almost every A/B calculator takes four numbers and prints a p-value. This one asks a question first, and it is the question that decides whether any of the numbers mean anything: was the sample size fixed before the test started?

If it was not — if you watched it and stopped when it looked good — a false positive is far more likely than the p-value suggests, and nothing done afterwards repairs it. Not a bigger sample, not a longer run, not this page. So it says so, plainly, and still shows you the figure, because hiding it would be its own kind of dishonesty.

It also checks whether the ordinary test is allowed to be used at all. With three conversions in one arm the normal approximation does not hold, and the p-value it would produce looks exactly like a real one. That case gets no number, and an explanation instead.

The other half is the next test rather than the last one: the sample each arm needs for the smallest lift worth finding, and how many weeks that is at your actual traffic. The table shows the fact that quietly ends most testing programmes — halving the effect you are looking for roughly quadruples the traffic you need to find it. If the honest answer is that you do not have the traffic, that is much better known before the test than after.

It says on the page what a p-value is: if the two versions were genuinely identical, how often a difference at least this large would turn up anyway. It is not the probability that the variant is better, it is not the size of the improvement, and 0.049 is not meaningfully different from 0.051.

There is no Bayesian posterior here, no chance-to-beat-baseline, and no correction for peeking. Each needs a choice this page cannot make on your behalf, and all three get printed by tools that have not earned them.

Save as many versions as you like and put any two side by side, figure against figure, with the better one on each line marked and the difference in its own column. That screen is the one that decides things, and it is the part a free calculator cannot do at all. Two tests side by side includes whether each was run in a way that allows a conclusion, which is usually the difference that matters.

One HTML file. No account, no subscription, no internet needed. Opens on Windows, Mac and Android; the readme explains the extra step on iPhone and iPad, where Apple blocks JavaScript in downloaded files and every tool of this kind is affected.

Everything lives in your browser on your device. Nothing is uploaded, which is why it works offline and why nobody else can see your figures — and it also means clearing your browsing data deletes it. The tool does not pretend otherwise: the header shows how long since your last export at all times, turns amber after three days and red after seven, and export is one click that produces a plain file you keep.