A/B Tests: The Design Elements Behind Trustworthy Results: Hypotheses, metrics, sample size, and stopping rules explained

An A/B test is a controlled experiment that randomly assigns eligible units to a control experience and a treatment experience, then estimates how the assigned change affected predeclared outcomes. Trustworthy results depend on a falsifiable hypothesis, a stable assignment unit, valid telemetry, primary and guardrail metrics, sufficient precision, and a stopping rule chosen before the result is inspected.

The two letters are the least important part. Variant A is usually the current experience and variant B the proposed change, but a random split does not rescue an ambiguous treatment, a broken assignment, an opportunistic metric, or a test that stops when the dashboard looks favorable.

For a binary outcome, the point estimate is straightforward:

control rate = control conversions ÷ control assigned units
treatment rate = treatment conversions ÷ treatment assigned units
absolute effect = treatment rate − control rate
relative lift = (treatment rate − control rate) ÷ control rate

Here is illustrative arithmetic, not a company result. If 10,000 assigned control users produce 800 conversions, the control rate is 8.0%. If 10,000 assigned treatment users produce 860, the treatment rate is 8.6%. The point estimate is +0.6 percentage points, or 7.5% relative lift.

That calculation is not a ship decision. It still needs the planned uncertainty interval, sample-size target, metric and telemetry checks, guardrails, and stopping rule.

A hypothesis is a decision contract#

“Changing the page will increase conversion” is too loose. It does not name the eligible population, mechanism, treatment, primary outcome, time horizon, or action after the result.

A testable contract answers:

• Population: Which units can enter, and when?
• Assignment: What unit is randomized—person, account, device, session, geography, or time block?
• Treatment: What exactly differs, and is that difference stable during exposure?
• Mechanism: Why could the change move the outcome?
• Primary metric: Which one result governs the decision?
• Guardrails: Which harms can stop or block the change?
• Minimum effect: What improvement is large enough to matter?
• Decision rule: What will the team do for a positive, negative, inconclusive, or invalid result?

The mechanism matters because it predicts diagnostic movements. If a shorter form is expected to reduce friction, starts and completions may change in a particular sequence. If completion rises but qualified downstream outcomes fall, the mechanism did not produce the intended business result.

Randomize the unit that carries interference and memory#

Random assignment supports causal interpretation when treatment and control groups are comparable except for the assigned change. The unit must match how exposure persists.

Session-level assignment can contaminate a test when the same person sees both variants and remembers the experience. User-level assignment can fail when several users in one account influence one another. Account-level assignment reduces that interference but also reduces the number of independent units. Geography or time-block assignment introduces different dependence and seasonality concerns.

Write the assignment key, allocation ratio, eligibility event, exposure event, exclusion rules, persistence window, and re-entry behavior before launch. Log assignment even when the experience fails to render, or treatment-dependent data loss can remove precisely the units most affected by the change.

Microsoft’s sample-ratio-mismatch research describes SRM as an unexpected difference between observed and configured assignment proportions. It can arise from assignment, triggering, redirects, telemetry loss, or filtering. The SRM test is a warning; investigation still has to find the cause.

Metrics need different jobs#

Microsoft’s during-experiment patterns separate overall outcome, feature and diagnostic, guardrail, and data-quality metrics. That taxonomy prevents one crowded dashboard from treating every movement as an equal reason to ship.

Use a denominator that treatment cannot silently redefine. “Purchases per checkout starter” can mislead if treatment changes who starts checkout. “Purchases per assigned eligible user” preserves the randomized population, while the starter rate can remain a diagnostic.

Declare how repeated events, bots, refunds, late conversions, missing data, outliers, currency, and account changes are handled. Archive the query or code version. A metric name is not a definition.

Sample size is planned from the decision, not copied from a rule of thumb#

The A/B sample-size guide by Georgiev and colleagues explains that planning depends on baseline rate or variance, effect definition, significance, power, allocation, and the analysis. Correlated observations and absolute versus relative effects require additional care. NIST’s comparison reference likewise derives proportion-test requirements from the assumed rates and error probabilities.

Start with the minimum detectable effect, or MDE: the smallest change the experiment is designed to detect with the chosen error rates. The MDE should come from the decision. If a change smaller than 0.3 percentage points cannot repay implementation and operating cost, powering for 0.05 points wastes traffic. If a 0.3-point harm is unacceptable, the guardrail may need more precision than the primary upside metric.

No universal sample size or duration exists. Required independent units increase when the baseline is noisy, the worthwhile effect is smaller, higher power is required, the significance threshold is stricter, or allocation is uneven. Seasonality, novelty, learning, network effects, and conversion lag can require a longer calendar even after the numerical sample target is reached.

Plan both an information target and a calendar boundary. Reaching a user count in two hours does not observe a weekly retention outcome. Running for several weeks does not repair too few independent accounts.

Stopping rules determine whether repeated looks are valid#

A fixed-horizon design selects its sample or duration and analysis before launch, monitors only for integrity and safety, and evaluates the primary result at the planned end. Repeatedly checking a conventional fixed-horizon p-value and stopping on the first favorable look changes its error behavior.

Sequential methods can allow continuous monitoring when the analysis is explicitly built for it. The always-valid inference paper develops one such statistical contract. Its existence does not make any real-time dashboard sequentially valid; the estimator, boundaries, and decision rule must implement the method actually claimed.

Safety stopping is different from success stopping. A severe reliability regression can end exposure early under a predeclared guardrail even if the primary analysis is fixed-horizon. Preserve the reason as “stopped for harm,” not “treatment lost under the primary metric.”

Write four end states before launch:

1. Ship: primary benefit crosses the decision threshold and guardrails pass.
2. Do not ship: evidence shows material harm or insufficient value under the rule.
3. Inconclusive: the interval includes materially positive and negative outcomes.
4. Invalid: assignment, exposure, telemetry, interference, or analysis failed.

Review trustworthiness before reading the winner#

External validity remains bounded. A result for eligible users during one period establishes an effect for that tested population and implementation under the design assumptions. It does not guarantee the same effect for new users, other countries, another season, a later product state, or a permanently scaled rollout.