This Optimisation 101 series demystifies the art and science of conversion optimisation

What is multivariate testing?

Multivariate testing (often referred to as MVT) sounds intimidating but really it isn’t. Once you get your head around it you’ll wonder what all the fuss was about!

Multivariate testing is similar to A/B testing but takes things further and tries to figure out which combination of variations produces the best results.

How does it work?

A multivariate test typically takes a few different elements on a web page and changes them all in a single experiment.

For example, a multivariate test may change a website image, headline, and button, all in the same experiment.

Where the magic happens is that each variation of each element is combined to find the best performing combination of variations.

As it’s a combination of variations, you can quickly end up with a large number of total variations. This can be a problem as it will really slow down your experiments.

To work out the total number of variations, just multiply the number of variations for each element together.  

So for an MVT experiment with two images, two headlines, and two buttons, the total number of variations would be 2 x 2 x 2 = 8 variations.

Why run multivariate tests?

Multivariate testing is useful when multiple elements on a page can be changed together to improve a single conversion goal e.g. sign ups or form completions.

A multivariate test should eliminate the need to run a number of sequential A/B tests on the same page with the same goal – which means quicker results for you.

Multivariate testing gotchas

The biggest downside to multivariate testing is the amount of traffic that can be required. As mentioned above, lots of variations can be generated which means the traffic each variation will receive is less, and therefore more overall traffic is required.

Before starting a multivariate test use a traffic estimator to see how much traffic you’re going to need to get a statistically significant result.

Another issue with multivariate testing is if one of the elements being tested doesn’t actually have an effect on the conversion goal – if that’s the case it would have been better to run an A/B test.

So as you can see multivariate testing is a powerful tool if used correctly, and what’s more it’s not just for websites, you can run multivariate tests for email too.

Some further reading…

Optimizely – What is multivariate testing

Hubspot – The critical difference between an A/B test and multivariate testing

VWO – Difference between A/B testing and multivariate testing

WiderFunnel – A/B Split Testing vs. Multivariate: Pros & Cons

Mailchimp – Multivariate Testing for MailChimp Pro