What Is A/B Testing? Validity and Low-Traffic Constraints

A/B testing is a controlled experiment that randomly splits eligible traffic between two versions of a page or product, A and B, and measures which performs better on a chosen metric. Random assignment isolates the change from other factors only when assignment, measurement, sample planning, and stopping rules remain valid.

For a SaaS website, version A is usually the current experience and version B is a deliberate change. Eligible visitors are assigned to one version, their outcomes are measured, and the groups are compared. The method is powerful because both versions run at the same time under broadly similar conditions. It is also easy to misuse: an experiment can produce polished charts and a precise-looking winner while answering the wrong question or violating the assumptions behind the analysis.

The practical starting point is not a page mockup. It is a written decision specification: what will change, who can enter, what unit gets randomized, which outcome matters, how large an improvement must be to matter, and what rule ends the test.

Plan sample size before looking for a winner#

For two conversion rates, one common normal-approximation formula estimates the required sample per variant:

n = (Zα/2 + Zβ)² × [p₁(1−p₁) + p₂(1−p₂)] ÷ (p₂−p₁)²

Here, p₁ is the baseline conversion rate, p₂ is the rate worth detecting, Zα/2 is 1.96 for a two-sided 95% significance level, and Zβ is 0.84 for 80% power. AB Tasty and MetricGate explain this structure.

Consider an illustrative example, not company data: baseline conversion is 10% and the smallest worthwhile variant result is 12%, with 95% significance and 80% power. (1.96 + 0.84)² = 7.84; [0.10 × 0.90 + 0.12 × 0.88] = 0.1956; and (0.12 − 0.10)² = 0.0004. The result is about 3,834 visitors per variant, or roughly 7,670 total. The formula is a planning approximation; the live analysis method, traffic structure, and metric distribution still govern the actual design.

There is no universal visitor threshold. Published rules of thumb vary by roughly an order of magnitude—from about 1,000 visitors or 50 conversions per week to 30,000 visitors and 3,000 conversions per variant. That disagreement is why the baseline rate and minimum detectable effect belong in the calculation instead of being replaced by a fixed benchmark.

Duration follows the plan as well. Common guidance is to cover at least one full business cycle—often one to two weeks—and many tests run two to six weeks, but the precomputed sample and operating context determine the endpoint. Do not stop the first time a conventional fixed-horizon result looks significant: repeated peeking inflates false positives. Either complete the prewritten plan or use a sequential method designed for early stopping.

Two neighboring terms also need clean boundaries. Split testing is often used as a synonym, but in a narrower technical usage A/B variants share one URL while split-URL testing redirects the visitor to another URL. Multivariate testing evaluates combinations of several elements at once rather than two whole-page versions, so it generally requires substantially more traffic. Kirro and VWO document those distinctions.

What an A/B test actually estimates#

In a controlled online experiment, random assignment creates treatment and control groups that can support a causal comparison when the design and data remain trustworthy. Microsoft’s account of online experimentation describes A/B tests as randomized controlled experiments used to assess how software changes affect customer behavior.

Suppose A converts at 5% and B at 6%. The absolute lift is one percentage point. The relative lift is 20% because the one-point difference is divided by the 5% baseline. Both are valid descriptions, but the absolute difference is what drives the added outcomes and much of the sample-size calculation. Always report the two rates, the absolute difference, the relative difference, and an interval showing the estimate’s uncertainty.

Write the decision specification before launch#

A usable experiment plan should resolve these fields before anyone sees results:

Microsoft’s pre-experiment patterns recommend a clear, measurable hypothesis, appropriate metrics, power planning, and a randomization unit suited to the product. They also note that power must be based on the units actually affected by a treatment, not all traffic nominally placed in the experiment.

Write one sentence that ties the plan together: “For eligible audience X, changing Y is expected to move primary outcome Z by at least D without breaching guardrails G.” If the team cannot fill in X, Z, D, and G, it is not ready to run an A/B test.

Four validity limits can erase the apparent winner#

Assignment and exposure must stay intact#

The observed allocation should match the configured allocation within normal sampling variation. A statistically meaningful mismatch is called a sample ratio mismatch, or SRM. It can reveal broken bucketing, missing telemetry, biased filtering, or a treatment that changes which users remain observable.

Check assignment counts before interpreting conversion differences. Also confirm that each unit receives a persistent variant, the treatment rendered as intended, and tracking behaved the same way in both variants.

The randomization unit must match the interaction#

Browser-level assignment may be acceptable for an anonymous landing page, but it can fail when one person visits on several devices. User-level assignment can also fail when members of the same SaaS account collaborate and expose one another to the treatment. Google’s collaboration-network experiment uses GCP data and simulation to show that selecting a randomization unit that contains connected users can prevent contamination and estimation bias.

For a B2B SaaS website, decide whether the treatment acts on a visit, a person, or an account. Count independent units at that level; do not inflate the sample by treating repeated pageviews as independent people.

The stopping rule must match the statistical method#

A fixed-horizon test does not permit checking an ordinary p-value every morning and stopping the first time it crosses a threshold. Each extra opportunity to stop changes the behavior of that decision procedure. If continuous monitoring matters operationally, use a method designed for sequential decisions and follow its stopping rule.

Changing the primary metric, audience segment, or test duration after seeing a promising result creates the same governance problem: the reported uncertainty no longer describes the search that produced the winner.

Statistical significance is not business significance#

A p-value does not tell you the probability that B is “the true winner,” and it does not measure effect size. The American Statistical Association’s statement says decisions should not rest only on whether a p-value passes a threshold and recommends full reporting and contextual interpretation.

A positive click result can still be a poor trade if qualified signups fall. A statistically detectable 0.1-point improvement can still be too small to repay engineering and maintenance costs. Define “worth shipping” before the result arrives.

Low traffic is an information constraint, not a tool setting#

For a binary conversion metric, sample size depends on the baseline rate, the smallest effect worth detecting, the chosen error rate, and statistical power. Penn State’s sample-size guidance for two independent proportions shows that the required sample grows with outcome variance and approximately with the inverse square of the absolute effect. Halving the effect you want to detect therefore requires roughly four times as much sample under the same assumptions.

The illustrative 10%-to-12% calculation near the start shows how quickly sample needs grow even for a two-point change. Repeat that calculation with the analysis method and baseline data actually used; its point is to expose the order of magnitude before launch, not to establish a traffic benchmark.

Low-traffic teams have only a few honest levers:

• Test fewer variants and reserve experiments for changes large enough to alter a decision.
• Measure a more frequent upstream event only when it is a defensible proxy, and keep downstream quality as a guardrail.
• Increase eligible traffic only by combining pages or audiences that truly share the same treatment and decision.
• Use validated variance-reduction methods when stable pre-experiment data exist. They can improve precision, but they cannot manufacture independent users or rare conversions.
• Stop planning an A/B test when the required duration makes the audience, product, or acquisition mix likely to become a different operating context.

Know when a different method answers the real question#

An A/B test is appropriate when the team can randomize a meaningful unit, expose enough eligible units, measure a decision-relevant outcome, and maintain the design through its stopping point. Use another method when those conditions do not hold.

Finally, read a completed test in a fixed order: verify instrumentation and allocation; report exposure and unit counts; inspect the primary effect and uncertainty interval; review guardrails; then apply the prewritten decision rule. An inconclusive result does not establish “no effect.” It says the data and design did not resolve effects at the precision required.