Multivariate Testing: How It Works and When to Use It

Multivariate testing (MVT) compares combinations of changes to multiple page elements against a defined outcome, helping identify both effective combinations and the influence of individual elements (Adobe’s MVT overview). Its central trade-off is traffic: as the number of combinations grows, less traffic reaches each combination and results take longer to establish (Optimizely’s MVT documentation).

multivariate testing: two equal unmarked page screens with different abstract layouts side by side, traffic tokens, variation swatches, calculator, notebook, desk lamp, coffee cup

What does a multivariate test measure?

An MVT separates the elements being changed from the alternatives available for each element: a factor is a page location or element, a level is one content option for that factor, and a combination is the complete experience containing one option from each factor (Adobe’s terminology guide).

The distinction matters when interpreting results: combination reports compare complete experiences, while section rollups aggregate results for an individual element across combinations (Optimizely’s explanation of MVT results).

How full factorial and fractional factorial tests work

A full factorial design includes every selected setting of every factor with every selected setting of the other factors; with two levels per factor, the number of combinations is two raised to the number of factors (NIST’s full factorial design reference).

Factors, each with two levelsFull factorial combinations
24
38
416
532

These counts are the standard design sizes in NIST’s factorial table, rather than visitor sample sizes.

For mixed level counts, multiply the number of alternatives for each factor to obtain the full combination count (Optimizely’s combination structure).

A fractional factorial design tests a deliberately selected subset of the full design to reduce the number of combinations (NIST’s fractional factorial reference). The reduction limits which effects can be separated: some effects are aliased, meaning their contributions cannot be distinguished without assumptions about the other effects (NIST’s guidance on modeling fractional designs). A smaller design therefore needs an analysis plan that specifies which interactions it can estimate.

Multivariate testing vs. A/B testing

An A/B test compares complete versions of an experience and can include substantial layout changes or several changed elements; it is suitable when a redesign does not divide naturally into separate component choices (Adobe’s A/B testing guide). Changing multiple elements in one A/B variant still provides a comparison between the complete versions, without separately estimating the effect of each change.

MVT organizes alternatives into factors and combinations so that component effects and interactions can be examined within the selected page design (Adobe’s MVT overview).

DecisionSuitable approach
Choose between complete layouts or redesignsA/B testing
Compare combinations of component alternativesMultivariate testing
Reduce combinations while estimating selected effectsA planned fractional factorial design

How much traffic does multivariate testing need?

Traffic requirements depend on the baseline outcome rate, the effect size being sought, the statistical error thresholds, and the analysis method; sample-size formulas also depend on assumptions about group sizes and metric variance (Optimizely’s explanation of sample-size calculations). Those dependencies make a fixed visitor threshold unsuitable as a general rule for MVT.

For planning, establish the following inputs:

  • Eligible daily visitors: traffic reaching the tested page and allocated to the experiment.
  • Baseline conversion rate: the historical rate for the chosen event.
  • Combination count: experiences and their planned allocation shares.
  • Duration: days available for data collection.

Adobe’s traffic estimator uses combination count, typical conversion rate, eligible daily visitors, and duration to assess feasibility (Adobe’s MVT traffic estimator documentation).

With equal allocation, the planning arithmetic is:

Expected visitors per combination = total visitors entering the experiment ÷ number of combinations

This estimate follows the equal-probability allocation described for Adobe’s full factorial MVT (Adobe’s MVT overview). Unequal allocation needs a separate estimate for each combination’s share; Optimizely calculates combination shares from the allocation percentages of its component variations (Optimizely’s allocation documentation).

Set the minimum detectable effect (MDE) and statistical power alongside the error threshold: power is the probability of detecting the specified real effect, and planning for a smaller MDE generally requires a larger sample when other assumptions stay fixed (Optimizely’s sample-size methodology).

Match the calculation to the intended analysis, whether it compares complete combinations or estimates selected factor effects and interactions; a two-group A/B sample-size estimate cannot automatically stand in for every MVT design (Optimizely’s guidance on matching sample size to the test).

Estimator defaults also matter: Adobe’s MVT estimator uses 95% confidence, 80% power, a 25% minimum reliably detectable lift, and a Bonferroni correction for multiple comparisons (Adobe’s documented estimator assumptions). These are that estimator’s settings, so an estimate made with them does not establish sensitivity to a smaller improvement.

How to set up a multivariate test

  1. Define the primary outcome. Separate the primary success metric from secondary and monitoring measures, as in Optimizely’s metric hierarchy.
  2. Select factors and prepare their alternatives. Focus on page locations relevant to the outcome, build the content in advance, and give factors and options identifiable names (Adobe’s MVT planning guidance).
  3. Configure eligibility and assignment. Define the audience and traffic allocation, and preserve consistent assignments during the run; Optimizely uses visitor identifiers to assign experiences and warns that reconfiguration can reassign visitors (Optimizely’s bucketing documentation).
  4. Preview every combination. Check the assembled experiences for conflicting content and incompatible designs, rather than reviewing each alternative only in isolation (Adobe’s experience-preview guidance).
  5. Keep the design unchanged during the run. Adobe warns that adding or changing locations and offers in a live MVT can reset reports (Adobe’s guidance on live-test changes).

How to interpret main effects and interactions

A main effect summarizes how the outcome changes across the levels of one factor, averaging across the other tested settings; it describes an overall pattern rather than a particular complete combination (NIST’s main-effects plotting reference).

An interaction effect describes dependence between factors: the effect of changing one element differs according to the setting of another (Optimizely’s explanation of interaction effects). This is why interpreting each element’s average separately can miss an important pattern in their combinations.

A full factorial model can include both main effects and interaction terms, while a fractional model must account for the design’s alias structure (NIST’s factorial modeling guidance). Keep the combination comparison, the overall element effects, and the interaction analysis distinct in the result.

The highest observed conversion rate alone does not establish a reliable winner: sampling uncertainty remains, and testing multiple variations or metrics increases the opportunity for false positives unless the analysis controls those comparisons (Optimizely’s multiple-comparison guidance).

Stopping rules depend on the method: fixed-horizon tests require a predetermined sample and analysis plan, whereas appropriately designed sequential tests support ongoing evaluation (Optimizely’s statistical-method comparison). Use the stopping rule belonging to the analysis rather than ending the experiment at the first favorable result.

When the estimated run length exceeds the available duration, reduce the combination count or extend the run (Adobe’s traffic planning guidance). A complete-page comparison can instead use an A/B design when that comparison answers the intended question (Adobe’s A/B guide).

One person. A whole marketing team.

Invite only