AB Testing:
Turn a Split Test into a Sound Decision

A dashboard says version B is ahead. The change looks harmless, the conversion line is green, and the team wants to ship before the result disappears. That is exactly when an A/B test is easiest to misuse. A visible lead may be random noise, a tracking failure, the result of checking too early, or a real effect too small to repay the cost of the change.

A/B testing is a controlled experiment in which eligible units—such as visitors, signed-in users, or business accounts—are randomly assigned to a control experience or a changed experience. The groups run concurrently, and a predefined outcome is compared. With sound randomization and measurement, the difference can support a causal claim about the assigned change. Microsoft describes online controlled experiments as a way to assess how software changes affect customer behavior, not merely as a method for finding correlations in a dashboard.

The useful output is therefore not “B won.” It is a bounded decision: for this eligible population, over this period, version B changed a specified outcome by an estimated amount, with stated uncertainty, while remaining inside agreed guardrails. The design either makes that sentence defensible or it does not.

Suppose a SaaS company changes the copy and layout of its demo-request page. Visitors who meet the eligibility rule are assigned persistently to A, the current page, or B, the new page. If assignment is random, both versions run at the same time, tracking works equally in both arms, and enough independent visitors enter the experiment, a difference in the predefined conversion metric can be attributed to the assigned page change within the limits of sampling uncertainty.

That “within the limits” clause matters. The experiment estimates an effect for the population that was eligible, the treatment actually delivered, the metric actually measured, and the operating period represented by the data. It does not prove that the same effect will hold for every acquisition channel, country, future season, or customer segment. Nor does a lift in button clicks prove a lift in qualified pipeline. Randomization protects the comparison from many confounders; it does not turn a proxy into the business outcome it is supposed to predict.

An A/B test is most useful when four conditions hold. A meaningful unit can be randomized. The treatment can be delivered consistently. A decision-relevant outcome can be measured. Enough independent units can arrive while the commercial context remains reasonably stable. If any of those conditions fail, changing the software setting or waiting longer does not repair the underlying information problem.

The method is also narrower than many teams assume. It can tell you whether an assigned change caused a measurable difference on the chosen outcomes. It usually cannot tell you why visitors reacted that way. A negative result may reflect confusing copy, weak motivation, an irrelevant offer, or simply an effect smaller than the experiment could resolve. Interviews, usability observation, logs, and qualitative research chosen for the decision can explain mechanisms that the randomized comparison cannot see.

Likewise, an inconclusive result does not establish that A and B are identical. It says the experiment did not resolve a difference at the planned level of precision. If the confidence interval still includes both a worthwhile gain and a meaningful loss, the honest answer is uncertainty—not “no impact.”

Design the decision before anyone sees a result#

The strongest protection against post-result storytelling is a short decision specification written before launch. It should identify the proposed action, the one meaningful treatment difference, eligibility, randomization unit, primary metric, guardrails, minimum effect worth detecting, sample plan, analysis method, and stopping rule. Microsoft’s pre-experiment guidance similarly emphasizes a clear measurable hypothesis, suitable metrics, power planning, and engineering checks before an experiment begins.

A practical hypothesis can fit in one sentence: “For eligible audience X, changing Y is expected to move primary outcome Z by at least D without breaching guardrails G.” Each symbol forces a decision. X fixes who the result is about. Y prevents the treatment from becoming a bundle of unrelated changes. Z names the outcome that will decide the test. D separates a detectable curiosity from a commercially useful improvement. G makes clear which harms can veto rollout.

The specification also needs actions for three outcomes: a clear gain, a clear loss, and an unresolved result. Without those rules, teams often reinterpret whatever arrives. A small positive estimate becomes a “promising win,” a negative estimate becomes a reason to inspect favorable segments, and an imprecise estimate becomes permission to run the preferred design anyway.

Choose the unit that receives the experience#

The randomization unit should match the level at which the treatment acts and interference can occur. A browser cookie may work for an anonymous landing page if repeat cross-device visits are rare enough not to matter. A signed-in user is stronger when the same person returns across devices. In B2B software, however, colleagues in one account may collaborate, share outputs, or expect the same configuration. Assigning individual users to different experiences can then contaminate both groups.

Google’s work on A/B tests in a collaboration network uses Google Cloud Platform data and simulation to show that selecting a unit containing connected users can avoid inconsistent exposure and estimation bias. The consequence is not free: account- or cluster-level assignment usually produces fewer independent units than user-level assignment. Microsoft has reported substantial power loss in some tenant-randomized experiments, including cases where detectable effects were about ten times larger than in comparable user-randomized tests. That figure is an observed Microsoft context, not a universal multiplier, but it exposes the trade-off.

Count the units you randomized, not the events they generated. Ten thousand pageviews from 800 assigned users do not become 10,000 independent observations. Treating repeated activity as independent makes uncertainty look smaller than it is.

Define success as an effect worth shipping#

Choose one primary metric because the test needs one main decision rule. Supporting metrics can help explain the result, and guardrails can block a harmful launch, but they should not become a large field from which the most flattering movement is selected afterward.

The primary metric must sit close enough to the commercial decision. Demo-request completion may be frequent enough to test, while closed revenue may be too rare and delayed. Yet a treatment that increases low-intent form submissions can raise the upstream metric while wasting sales capacity. In that case, lead quality, invalid submissions, or downstream qualification belongs in the guardrails even if it cannot serve as the primary metric.

Now set the minimum detectable effect from the decision, not from the traffic available. Ask for the smallest change that would repay implementation, maintenance, sales, and opportunity costs. If a 0.2-percentage-point conversion gain would not change the rollout decision, powering a test to detect it produces precision without value. Conversely, choosing an implausibly large effect merely to make the sample requirement convenient creates an experiment designed to miss smaller gains that might matter.

Metric language must remain unambiguous. In an illustrative result where A converts at 5% and B at 6%, the absolute lift is 1 percentage point. The relative lift is 20%, because the 1-point difference is divided by the 5% baseline. A report that says only “conversion increased 20%” invites overinterpretation. Give both variant rates, the absolute difference, the relative difference, and an interval showing the estimate’s uncertainty.

Prove that the test can finish—and remain valid#

Calculate sample and duration before launch#

There is no credible universal rule such as “every test needs 10,000 visitors” or “run it for two weeks.” Sample size depends on the baseline outcome rate, the smallest effect worth detecting, the accepted false-positive rate, desired power, allocation ratio, metric distribution, and analysis method. Duration then depends on how quickly eligible independent units enter the experiment.

For two conversion proportions with equal allocation, a common normal-approximation planning formula is:

n per variant ≈ 2 × (z for significance + z for power)² × p̄(1 − p̄) ÷ (p₂ − p₁)²

Here, p₁ is the baseline rate, p₂ is the smallest rate worth detecting, and p̄ is their average. For a two-sided 5% significance level and 80% power, the z values are about 1.96 and 0.84. Penn State’s sample-size lesson for two independent proportions explains this structure and also notes that unequal allocation can raise the total sample requirement.

Take an illustrative planning case, not company data: baseline conversion is 10%, the smallest worthwhile result is 12%, allocation is 50/50, significance is two-sided at 5%, and power is 80%. Then p̄ is 0.11, the absolute effect is 0.02, and the approximation gives roughly 3,838 independent visitors per variant, or about 7,676 total. The exact requirement may differ under the team’s chosen test, continuity correction, clustering, attrition, variance reduction, or sequential method. The calculation’s job is to expose feasibility before launch.

The inverse-square relationship is the brutal part. Under the same assumptions, halving the absolute effect you want to detect requires roughly four times the sample. A low-traffic team cannot make that constraint disappear by selecting a more optimistic dashboard confidence setting.

Convert the sample requirement into time using eligible-unit flow, not total site traffic. If only 20% of visitors can encounter the changed component, the other 80% do not supply treatment information. A test that needs 7,676 eligible visitors and receives 550 a week needs about 14 weeks before allowances for missing data or business cycles. By then, a campaign, product release, or acquisition mix may have changed enough to weaken the relevance of the result.

Longer is not automatically safer. Research on long-term online experiments identifies threats including unstable cookie identifiers, survivorship bias, selection bias, and misleading trends. Short tests can miss weekly cycles or novelty effects; very long tests can drift into a different operating context. The defensible duration is the shortest period that reaches the planned information requirement while representing the cycles relevant to the decision.

Low-traffic teams have a few honest levers. They can reserve experiments for larger, decision-changing effects; test one treatment instead of spreading traffic over many arms; broaden eligibility only when the added audience truly shares the treatment and decision; or use a more frequent upstream metric when it is a defensible proxy and downstream quality remains a guardrail. Variance-reduction methods can improve precision when suitable pre-experiment data exist. None of these techniques manufactures independent customers or rare conversions.

A good preregistration can still produce unusable data. Assignment may not persist, the treatment may fail to render, one variant may log differently, or the analysis may silently exclude people affected by the change. The operational checks are part of the experiment, not cleanup after the “real” analysis.

Validate allocation, exposure, and telemetry first#

Begin with counts of randomized units. If a 50/50 experiment records a difference larger than normal sampling variation would explain, it may have a sample ratio mismatch, or SRM. Microsoft’s taxonomy of SRM causes treats the mismatch as a symptom that can arise from assignment, execution, logging, or analysis failures. Its examples show why the cause, not the cosmetic ratio, matters: missing units may be precisely those most affected by the treatment.

Do not interpret conversion lift until a sample ratio mismatch has been diagnosed and the analysis shown to be trustworthy.

After allocation, check whether each unit stayed in its assigned arm, whether the intended treatment actually rendered, and whether exposure was recorded with the same logic in A and B. Compare event loss, join rates, error rates, latency, and missing values by variant. If B slows the page enough to prevent its own conversion event from firing, the apparent business result is entangled with broken observation.

Guardrails need live safety monitoring even when the primary analysis is fixed at the horizon. A severe error or latency regression should be able to stop exposure early under a prewritten safety rule. That is different from repeatedly checking the primary metric and stopping when it looks favorable: one protects users from harm, while the other changes the false-positive behavior of the decision rule.

Match the stopping behavior to the statistical method#

A conventional fixed-horizon analysis assumes the team will collect the planned sample and evaluate the primary comparison at the specified endpoint. Looking every morning and stopping the first time an ordinary p-value crosses 0.05 creates repeated opportunities for noise to look decisive. Microsoft’s discussion of near-real-time experiment monitoring warns that frequent checking with fixed-horizon statistics can inflate false positives and points to sequential testing when continuous monitoring is required.

If the team needs the option to stop for success, futility, or harm, it must use a method designed for those decisions and follow that method’s boundaries.

Do not change the primary metric, minimum effect, eligible audience, preferred segment, or test duration because a partial result looks encouraging. Each change expands the search that produced the reported winner. Exploratory segment findings can generate a new hypothesis, but presenting the best discovered segment as if it were the planned confirmatory result hides the additional uncertainty.

Statistical significance also answers less than its name suggests. A p-value is not the probability that B is better, and crossing a threshold does not measure the size or commercial importance of the effect. The American Statistical Association’s statement on p-values explicitly says that statistical significance does not measure effect size or result importance and that decisions should not rest on a threshold alone.

That distinction cuts both ways. A huge sample can make a 0.1-point lift statistically detectable even when it cannot repay the change. A smaller sample can leave a commercially valuable estimate uncertain. The decision needs the effect estimate, uncertainty interval, guardrails, costs, and the action rule written before the result arrived.

Read the result in the order that prevents wishful thinking#

A trustworthy analysis moves from whether the experiment worked to what the treatment did. Reversing that order makes a green headline metric psychologically difficult to discard when a validity failure appears later.

1. Verify assignment and data quality. Compare observed allocation with the configured split, confirm persistent assignment, inspect treatment delivery, and check whether tracking and exclusions behaved equally across arms. A failed SRM or telemetry check pauses interpretation until its cause is resolved.

2. Report the population and exposure. State eligibility, dates, allocation, randomized-unit counts, exposed-unit counts when relevant, and the analysis population. This establishes who and what the estimate represents.

3. Estimate the primary effect. Show both arm rates or means, the absolute and relative difference where appropriate, and the uncertainty interval under the prespecified method. Avoid reducing the result to a winner label.

4. Inspect guardrails and known interactions. A gain in the primary metric does not compensate automatically for more errors, slower performance, lower lead quality, or harm elsewhere in the customer journey. Apply the veto thresholds as written.

5. Apply the decision rule. Ship only when the effect is sufficiently large and precise, guardrails remain acceptable, and implementation cost still supports the choice. Reject a clear loss. For an inconclusive result, decide whether another test could economically reduce the uncertainty or whether the rational action is to keep A.

6. State the boundary of the claim. Name the population, treatment, period, and outcomes covered. If future persistence matters, continue post-launch measurement or use a holdout appropriate to that question. Microsoft’s work on external validity in online experiments shows why even an internally valid short-term estimate can change with user learning, product interactions, or a shifting population.

An inconclusive result often creates the hardest conversation. If the interval includes both the minimum worthwhile gain and no effect, the experiment did not answer the shipping question. Extending it is defensible only if the original method permits it or a new design is established without pretending the added horizon was prespecified. Running until the preferred answer appears is not patience. It is a new and undisclosed decision rule.

When A/B testing is not the right method#

Randomization is powerful, but it is not a ceremonial step that every product change must pass. The method is a poor fit when the treatment cannot be isolated, independent units are too scarce, the outcome arrives too late, or every plausible result leads to the same action.

The choice of method should follow the unresolved question. If the team needs to know whether users can find a setting, watch them try. If it needs to estimate the causal effect of a shippable change and can maintain random assignment, test it. If it cannot act differently across the plausible results, collect no data at all. An experiment without a decision is analytics theater.

Write the sentence that can cancel the test#

Before building version B, write the hypothesis with X, Y, Z, D, and G filled in, then calculate the required independent units at the correct randomization level. If the audience, meaningful effect, guardrails, or feasible stopping point cannot be named, stop there. The most valuable A/B testing decision is sometimes the one that prevents an unanswerable experiment from launching.

Frequently asked questions#