How to Read a Confidence Interval Before Acting on an A/B Test

A confidence interval is not a winner badge. It is a range that shows how precisely an A/B test has estimated its treatment effect. The useful question is not merely whether the interval crosses zero. It is whether the full range resolves the business decision: does it rule out unacceptable harm, establish a worthwhile benefit, or leave both possibilities open?

For many familiar methods, the interval has this structure:

estimated treatment effect ± critical value × standard error

In a large-sample, two-sided normal approximation, the critical value for a 95% interval is about 1.96. That number is not universal: the appropriate interval depends on the metric, sample size, randomization scheme, and analysis method.

The “95%” also has a precise meaning. If the same sampling and interval-building procedure were repeated many times under comparable conditions, about 95% of the resulting intervals would contain the true parameter. It does not mean there is a 95% probability that the fixed parameter lies inside this one observed interval. NIST explains both this long-run interpretation and the relationship between sample size, variability, and interval width in its guidance on confidence limits.

Read the interval against a decision#

Suppose a product team runs an A/B test and reports the following illustrative result, not company data:

• Estimated conversion lift: +1.8 percentage points
• 95% confidence interval: −0.4 to +4.0 percentage points
• Smallest benefit worth shipping: +2.0 percentage points
• Unacceptable-harm boundary: −1.0 percentage point

The point estimate favors B, but the interval still includes zero. Under the stated method, the test has not established that the direction is positive at the corresponding two-sided 5% significance level.

That is only the first conclusion. The upper end includes benefits large enough to justify shipping, while the lower end still permits a small loss. The interval excludes harm worse than the team’s −1.0-point boundary, but it does not exclude all harm. The result therefore supports neither “B is a winner” nor “B has no effect.” It says that material upside remains possible and modest downside remains possible.

If conversion is the deciding metric, a defensible action is to withhold rollout and ask whether reducing that uncertainty is worth the cost of more information. That is a business decision based on an unresolved interval—not a statistical declaration that B failed.

Zero answers only the direction question#

For a difference metric, zero usually represents no treatment effect. Relative to that value, a confidence interval can support four materially different readings:

• Entirely above zero: the result supports a positive direction at the stated confidence level, assuming the design and method are valid.
• Entirely below zero: the result supports a negative direction under the same conditions.
• Crosses zero and is wide: meaningful benefit and harm may both remain compatible with the data. The result is inconclusive.
• Narrow and close to zero: effects large enough to matter may have been excluded, but a practical no-effect claim requires prewritten equivalence bounds and the corresponding equivalence procedure.

Statistical significance does not establish commercial importance. An interval from +0.01 to +0.03 percentage points can provide clear directional evidence while describing a gain too small to pay for implementation. Conversely, “not statistically significant” does not mean “no difference”; a wide interval can simply mean that the test was too imprecise to settle the question.

The thresholds that govern action should therefore be defined before the result is seen. The smallest worthwhile benefit comes from economics, implementation cost, strategic value, and risk tolerance—not from whether an endpoint happens to land above zero.

Check what the interval is actually estimating#

An interval inherits the definition and weaknesses of the metric beneath it. Before using one in a ship decision, confirm four things.

Effect scale. State whether the result is absolute or relative. A move from 12% to 13% is an increase of 1 percentage point but approximately 8.3% relative. Mixing those scales can turn the same result into two very different business claims.

Unit of analysis. The standard error must reflect the randomization unit and dependence in the data. Ten sessions from one account do not automatically provide the information of ten independent accounts. Research on trustworthy A/B-test analysis shows that unjustified independence assumptions can underestimate variance in some experiment designs.

Metric construction. Eligibility, aggregation, and denominator choices determine the estimand. Revenue per visitor, revenue per purchaser, and conversion per session answer different questions even when they use the same events. Microsoft’s paper on metric-interpretation pitfalls documents how plausible metric movements can lead experimenters to incorrect conclusions.

Experiment validity. Check allocation, exposure, missing telemetry, repeated observations, stopping rules, and multiple comparisons before trusting the endpoints. A sample-ratio mismatch—when observed assignment counts differ unexpectedly from the planned ratio—is a warning of possible data or randomization problems, not a detail that a narrow interval can repair. Microsoft’s sample-ratio mismatch research describes how ignoring it can reverse a ship decision.

Do not substitute MDE for the observed interval#

Minimum detectable effect and confidence interval have different jobs. MDE is chosen during planning and depends on assumptions about variance, sample size, significance level, allocation, and power. The confidence interval is calculated after observing the data. NIST’s sample-size guidance shows how the effect to be detected, variance, significance level, and power enter planning calculations.

Neither quantity defines what is worth doing. A team may design a test to detect a 1-point change yet require a 2-point improvement to justify an expensive rollout. Record the detectable effect and the decision threshold separately.

Use a result-reading contract#

A decision-ready experiment summary should make the following fields explicit:

Population and time window:
Randomization unit:
Primary metric and effect scale:
Control result:
Treatment result:
Estimated effect:
Confidence level, interval, and method:
Smallest worthwhile benefit:
Unacceptable-harm boundary:
Validity checks and exceptions:
Decision supported by the full interval:

The final line should be written from the entire interval. Ship when the evidence clears the prewritten benefit and safety requirements. Reject when it establishes unacceptable harm. Treat a narrow interval inside the approved equivalence region as evidence of practical similarity only under the planned equivalence method. When the interval still spans consequences that would lead to different actions, the honest result is unresolved uncertainty.