Statistical Significance: What It Means Before You Ship an A/B Test

A dashboard turns green: the new checkout is “statistically significant” at p = 0.04. That is useful information. It is not yet a shipping decision.

The result says that the experiment crossed a threshold chosen for a particular statistical test under a particular null model. It does not say there is a 96% chance the new checkout is better. It does not say the lift is large enough to matter, the experiment was valid, or the result will persist after rollout. Those are separate questions, and a business decision fails when one green badge is made to answer all of them.

The practical reading is narrower: first establish that the test was designed and run well; then read the estimated effect and its uncertainty; then decide whether the plausible benefit is worth the cost and risk. Statistical significance has a legitimate place in that sequence. It just does not get the last word.

Statistical significance answers a conditional question#

Most product experiments begin with a null hypothesis, often that the treatment and control have the same value on a predeclared primary metric. The analysis asks whether the observed difference would be unusual if that null model and the test’s other assumptions were used to generate the reference distribution.

That conditional wording matters. A significant result is not a freestanding property of the data. It depends on the metric, analysis unit, population, test statistic, sidedness, stopping rule, and treatment of multiple comparisons. Change those choices after looking at the results and the original p-value no longer describes the procedure that produced the claim.

The threshold also has to exist before the result arrives. If alpha is 0.05, the rejection rule is designed so that, under the model, a result lands in the rejection region 5% of the time when the null hypothesis is true. NIST explains the relationship between alpha, critical values, and p-values. This is a long-run error property of the procedure, not a 5% probability that the null is true in this particular experiment.

What a p-value actually measures#

A p-value is the probability, calculated under the null hypothesis, of obtaining the observed test statistic or one more extreme. It therefore measures compatibility between the data and a specified model. A small p-value says the observed statistic sits far into the model’s tail. It does not assign probabilities to the null and alternative hypotheses.

That is why “p = 0.04 means the treatment has a 96% chance of working” is wrong. The probability runs in the other direction: the calculation assumes the null model and then evaluates the data. To estimate the probability of a hypothesis, an analysis would need a framework that places probabilities on hypotheses and supplies the required prior assumptions. A conventional p-value does neither.

The American Statistical Association’s statement on p-values draws three boundaries that matter in a business review: a p-value does not state the probability that a hypothesis is true, statistical significance does not measure effect size or importance, and decisions should not rest only on whether a threshold was crossed.

The null model is not “random chance alone,” either. It contains assumptions about sampling and the statistic’s distribution. A low p-value can flag incompatibility somewhere in that bundle; it cannot identify whether the explanation is the intended treatment effect, a broken randomization, a logging defect, an unmodeled dependency, or another departure from the assumptions. Context supplies that diagnosis.

What crossing alpha changes—and what it leaves open#

When a valid, predeclared test returns p at or below alpha, the defined procedure rejects its null hypothesis. That is the defensible claim. “Proves the treatment works” adds conclusions the calculation did not produce.

The gap becomes clearer when the result is translated into questions a product or revenue team actually has to answer:

A non-significant result deserves particular care. “We did not reject zero” is not the same as “we proved zero.” The interval might be narrow enough to rule out any worthwhile gain, which can support stopping. Or it might span a harmful decline, no change, and a valuable improvement, which means the experiment did not resolve the decision. Power and precision determine which of those stories fits.

Nor does significance establish causality by itself. Random assignment, comparable exposure, reliable measurement, and analysis at the correct unit are what make the treatment contrast interpretable. NIST describes experimental design as a detailed plan laid out before the experiment and says that randomization protects results from extraneous factors. A perfect p-value cannot repair users switching variants, revenue recorded only for one arm, or thousands of events from one account being treated as thousands of independent customers.

One conversion test shows why the badge is not enough#

Consider an illustrative A/B test with 10,000 eligible visitors in each arm. Control records 500 conversions and treatment records 560. The observed conversion rates are 5.0% and 5.6%, so the absolute difference is 0.6 percentage points and the relative lift is 12%.

For a common large-sample test of two independent proportions, the null hypothesis says the two population conversion rates are equal. The pooled rate is calculated across both arms, and the observed difference is divided by its pooled standard error:

p_A = x_A / n_A

p_B = x_B / n_B

p_pool = (x_A + x_B) / (n_A + n_B)

z = (p_B − p_A) / sqrt[p_pool × (1 − p_pool) × (1/n_A + 1/n_B)]

NIST documents this pooled two-proportion test, including its use as a large-sample normal approximation. For the illustrative counts above, z is about 1.89 and the two-sided p-value is about 0.059. With a predeclared alpha of 0.05, the test does not reject equality.

The useful business statement is not “treatment failed.” The observed result favors treatment, but this fixed-horizon test did not cross its rejection threshold. A corresponding approximate 95% interval for the absolute difference runs from roughly −0.02 to 1.22 percentage points. The data remain compatible with a negligible loss, no effect, and gains that could be worthwhile. Whether to collect more data cannot be decided from p = 0.059 alone; it depends on the original stopping plan and whether the unresolved range contains effects that would change the action.

Now imagine the same observed rates measured with far more visitors. The point estimate and the likely per-user economics have not changed, but the standard error becomes smaller. The result may cross the significance threshold because it is more precise. That does not make the 0.6-point difference larger. Conversely, a strategically valuable difference can miss the threshold when a small sample leaves wide uncertainty.

This is why sample size belongs in planning, not in the post-result story. The minimum detectable effect should reflect the smallest change worth acting on, and the sample plan should provide adequate power to detect that change under stated assumptions. NIST’s sample-size treatment for proportion tests makes the relationship explicit: significance level, desired power, effect size, and sample size are linked. “Get significance” is not a defensible sizing objective when the detectable effect has no connection to the decision.

Do not extend a fixed-horizon test simply because its current p-value is close to 0.05; continue only under the predeclared plan or a valid sequential design. Repeatedly checking and stopping when the line turns green changes the false-positive behavior that the original threshold promised. Formal sequential methods address this with planned analyses and stopping boundaries; FDA guidance on adaptive designs notes that choosing interim-analysis timing from accumulating comparative results can inflate Type I error. The same problem appears when a team tries several outcome definitions, segments, exclusion rules, or time windows and reports only the one that passed.

That does not mean every analysis must be frozen forever. Exploratory analysis can generate the next hypothesis, and sequential methods can support planned interim decisions. The distinction is disclosure: exploration should not be presented as if it were the single confirmatory test declared before the data were seen.

Read the experiment in decision order#

A sound review makes the significance result one checkpoint rather than the headline that suppresses everything around it. The order below is useful because each step can invalidate or reframe the next one.

1. Recover the experiment contract. Identify the eligible population, assignment and analysis units, primary metric, minimum effect worth acting on, alpha, sidedness, planned sample or stopping rule, exclusions, and any correction for multiple testing. If those choices were not made in advance, label the result exploratory rather than inventing a cleaner history.

2. Test whether the comparison is trustworthy. Check variant allocation, exposure delivery, missing data, duplicate units, instrumentation changes, and whether both arms ran through comparable business conditions. A statistically unusual difference is still unhelpful when the treatment group saw a holiday campaign that control did not. Randomization reduces this risk only when assignment remains intact through exposure and measurement.

3. Read the estimate before the threshold. Report both arm values, the absolute difference, the relative difference where it aids interpretation, and an interval on the same effect scale. Relative lift alone can make a small baseline change sound dramatic; absolute change alone can hide its proportional importance. The denominator and time horizon belong beside both.

4. Compare the plausible effects with the decision threshold. If the entire interval lies above the minimum worthwhile gain, the case is stronger than merely clearing zero. If it excludes worthwhile upside, stopping can be reasonable even when the point estimate is positive. If it spans both meaningful harm and meaningful benefit, the honest conclusion is uncertainty—not victory and not “no effect.”

5. Apply economics and guardrails. Translate the effect into the unit the decision consumes: incremental qualified leads, retained accounts, gross profit, or another pre-agreed outcome. Then inspect latency, refunds, support demand, reliability, and other guardrails. A statistically significant conversion gain can be a bad release when it attracts low-quality demand or raises cost more than contribution.

6. Make only the action the design permits. Ship, reject, continue under a valid rule, or record the result as inconclusive. State deviations and exploratory findings beside the conclusion. The ASA’s later task-force statement notes that thresholds can be useful when action is required, but they should be defined from study goals and the consequences of wrong decisions; it also stresses that no single uncertainty measure serves every purpose.

This sequence prevents two symmetrical errors. One is shipping a trivial or invalid “winner” because p fell below 0.05. The other is abandoning a possibly valuable change because a noisy estimate landed just above it. Both mistakes come from treating a threshold as the decision rather than as evidence used by the decision.

The effect has to clear your bar, not merely zero#

Statistical significance is most useful when its job is kept narrow. It can tell you that a predeclared test found the data sufficiently incompatible with a null model to trigger a specified rule. It cannot set the minimum worthwhile effect, price implementation risk, validate the data pipeline, or choose the guardrails.

Before the next experiment launches, write down the smallest effect that would change the decision and build the analysis around that number. When the result arrives, ask whether the plausible effects clear that bar. The green badge comes later.

Frequently asked questions#