Author: Thijs de Valk

We’ve written quite some posts on Conversion Rate Optimization and A/B testing on yoast.com in the last few years. However, we’ve never really touched the subject of multivariate testing. In this post I’ll explain what multivariate testing is, its pros and cons and when you should (and should not) be using it.

What is multivariate testing?

Explaining what multivariate testing is, seems quite easy. In fact, when you look up multivariate testing on Wikipedia, it says the following:

“In internet marketing, multivariate testing is a process by which more than one component of a website may be tested in a live environment. It can be thought of in simple terms as numerous A/B tests performed on one page at the same time.”

However, I find this to be oversimplifying things a bit. With A/B tests you’re testing different versions of a complete page. This is not what you do with a multivariate test (MVT). With an MVT you’re testing elements within a page.

So lets say you have 2 versions of a headline, 2 versions of an image and 2 versions of a block of text. With a multivariate test you will test every possible combination against the others to figure out which combination yields the best conversion rate. And this is where things get tricky, because in the setup I mentioned above, there are already 8 combinations (2 x 2 x 2 = 8):

1. Headline A + Image A + Textblock A
2. Headline A + Image A + Textblock B
3. Headline A + Image B + Textblock A
4. Headline A + Image B + Textblock B
5. Headline B + Image A + Textblock A
6. Headline B + Image A + Textblock B
7. Headline B + Image B + Textblock A
8. Headline B + Image B + Textblock B

Advantages

The first advantage of multivariate testing is that you can see the effect small changes on your site have. Of course, small changes can also be tested with an A/B test, but that’s suboptimal as you can only test one small change at a time. Generally, A/B tests are used for the big changes and multivariate testing is used for optimizing smaller elements.

Also, by using a multivariate setup, you’re actually able to not only test the effect of changing one element, but you’re also able to test the combined effect (interaction effect) of several elements. Will changing an element still have the same effect if you change another element on that same page? These are questions you wouldn’t be able to answer with an A/B test, but you can answer them with multivariate tests. With multivariate tests you can pretty accurately see what the effect of each element is in which setup or situation.

Limitations

The biggest drawback of multivariate testing is that you need an even bigger amount of traffic and especially conversions than you do for an A/B test. I always say you should have at least 100 conversions on each variation. So if your multivariate test has 3 different versions of 3 different elements, you’ll need at least 2700 conversions (3 x 3 x 3 = 27 combinations with each at least 100 conversions).

And that’s a lot for most websites, especially if we’re talking sales on a specific product. For most websites this also means that the page needs to have quite some traffic on it, since most conversion rates aren’t that high.

Lastly, the multivariate test setup looks at a lot more variables than an A/B test and also looks at how these variables interact. This means there’s a lot bigger chance that mistakes or errors can occur in the reporting. So you should check your multivariate test results even more than your A/B test results.

When to use multivariate testing

As I’ve already mentioned above, multivariate tests are used to test smaller elements on a page. You’re testing small variations of the same element, instead of overhauling the complete page as you would in an A/B test. So first and foremost, multivariate testing isn’t supposed to be used as a starting point.

Obviously, before starting to test at all, there are a few things you have to do. First of all, you should check whether there’s even enough traffic and conversions on the page you want to run a multivariate test on. Since an MVT setup increases your amount of variations really quickly, it’s important that you have an abundance of traffic and conversions. You usually already know whether your traffic and conversion rate are up to par, because you’ve already A/B tested the page in question.

What’s next?

The next step is to find out what kind of changes you think are needed for that page. To do this you first need to be clear on what the goal of the page is. What do you want people to do on your page? Write down what you think could be the cause of visitors not completing this goal. You can find this out by being critical yourself, doing user testing (or even just a survey) and looking at your analytics.

Now you know what your page’s goal is and what could be causing a lower conversion rate on this goal. The next step is that you need to find out what you could possibly change to make more people fulfill that goal. You have to think of different versions of the factors that could be hurting your conversion rate.

When your traffic and amount of conversions are acceptable, you’ve found your page’s goal and causes for it being suboptimally completed, you should formulate hypotheses. Write down what you’ll be changing, what effect you expect this to have and maybe even why. For instance:

• If we shorten the checkout form, more people will complete their purchase.

To my mind, hypotheses are even more important for multivariate testing, because it is so easy to just add a few more variations. Formulating hypotheses will prevent you from randomly adding new variations. Make sure you have a hypothesis for every single variation you’re creating.

Multivariate or A/B?

If you find your hypotheses are about changes that drastically change the page’s layout or look, then you’re better off choosing for an A/B test. As said, A/B tests give you the possibility to find out whether one (version of a) page performs better than the other.

If however, you find that your hypotheses are about small changes (f.i. text on your call-to-actions) then multivariate testing could be a good choice. But you have to be sure your page meets the criteria for multivariate testing that I mentioned above.

What do you think?

Multivariate testing isn’t as easy as it might seem. It’s something you should only get into if you know what you’re doing. So I’m wondering: have you ever tried multivariate testing? And did you follow my steps? Or do you have other steps you think should be followed? Let me know in the comments!

This post first appeared as Multivariate testing: what it is and when to use it on Yoast. Whoopity Doo!

We’ve written a few posts on how to set up your A/B tests. However, the setup of your A/B tests is only the first step, obviously. Once you’ve set up a test and you’re getting some results, what should you do then?

In this post I’ll be explaining how you can interpret your A/B test results and what you should look out for.

Test duration

Before you can actually start interpreting your A/B test results, you need to be sure that your test has been running for at least 7 days. This means that you’ll have corrected for the fact that some days get you more traffic, sales or anything else than others.

There are some guides out there telling you to run until your test is at least 95% significant. Marieke has explained to great detail why this is oftentimes not a smart thing to do. Please be aware that this is not a hard requirement to our minds.

Interpreting your results

Sometimes it can be hard to actually know what’s going on when looking at the results of your A/B test. So let me walk you through what I always look at in the results:

We’re using Convert for our A/B tests, and when a test is done, it will us a chart that looks something like this one. There’s a lot going on here, which can be quite daunting. First of all, it shows quite clearly that Variation 1 is the winner. However, I’m always very careful when it comes to this. Let me show you why:

This chart shows you that there’s a pretty big jump in the conversion rate, the improvement is 150% and a nice confidence level. However, it also shows us that all this has been calculated over just 18 conversions. Over 10,000 visits, and just 18 conversions. I prefer to have at least 100 conversions on each variation, so this is pretty slim. Now we know that while this result is pretty awesome, we have to be careful in using this data to support any future action.

So, what’s next?

The next thing I’ll check is the graph:

I always take a look at the trend of the test. This graph shows the conversion rate of the original version, the variation, the total visitors and the average conversion rate. The original version (blue line in this case) and the variation (purple line) should not be too close during the duration of the test. If there’s just a spike at the end that made the test variation the winner, you know the results aren’t trustworthy. In this case, it looks pretty good, as the variation outperformed the original pretty soon. But don’t get too ecstatic and stay aware of the low number of actual conversions, in this case.

Be sure you also look at the Y-axis (the vertical one) to make sure the differences aren’t really small. The Y-axis tends to change with results, so small differences can look like big ones.

Other statistics

Convert will give you quite some other statistics that could be interesting and important to look at:

The conversion rate and amount of conversions are obvious ones, but I’m mostly interested in the revenue stats. Which version got you the most total revenue and revenue per visitor? But maybe you’re looking for a higher adoption rate of your product (more people using your product) instead of more revenue. In this case you should look at the bottom one: average products sold per visitor.

These statistics will be available per variation and will give you some detailed information on what the difference between the variations actually was. So be sure to look at these and use them for your interpretation!

Implementing the winning variation

In this case, we have chosen to implement the test variation, but we’ve also kept a close eye on the amount of sales and the revenue. We have checked the amount of sales and revenue against the weeks before, to be sure it wasn’t a fluke during the tests. And I implore you to do the same. It will probably not go wrong a lot, but when it does, you’ll be sorry for it. So stay aware of the fact that these tests are not perfect.

Over to you

Have you ever run a test without knowing what to do with the results? Did you ever make the wrong choice? Or you simply have something to mention about this post? Let me know in the comments!

This post first appeared as Interpreting your A/B test results on Yoast. Whoopity Doo!

At Yoast we see a lot of websites on a daily basis. One thing we notice when looking at these websites is how poorly the website’s or page’s goal is communicated a lot of the time. A really good and fast way to convey your goal is by using headlines and taglines.

In this post I’ll tell you how to use your headlines and taglines to your advantage by having them communicate a clear message that people will understand. Because you want people to understand what your website or page is about, right?

What are headlines and taglines?

Before jumping into this post, it might be useful to explain what headlines and taglines actually are. Wikipedia explains a headline in the following way:

The headline is the text indicating the nature of the article below it.

So the headline is basically a page’s or post’s title. And as Wikipedia so eloquently states it; that title should indicate what that page or post is about.

And Wikipedia explains the tagline as follows:

In entertainment, a tagline (or tag line) is a small amount of text which serves to clarify a thought for, or designed with a form of, dramatic effect.

I don’t completely agree with a tagline being for dramatic effect, but I do agree with the fact that a tagline serves to clarify a thought. Usually, it serves to clarify or specify the headline. A tagline is also something that’s not as mandatory as the headline. A page has to have a headline, but it doesn’t have to have a tagline. Especially not if the headline is clear enough in itself.

Why use headlines and taglines?

As I mentioned above, headlines and taglines are a good and fast way to tell people what your website is about. And this matters, because people want to understand what your website is about, so they know if you have what they want. People don’t want to spend 30 minutes (or 5, for that matter) searching your website to see if you do. They’ll just go to a website with a clearer message.

Additionally, having to think about a decent headlines and/or tagline for your website and its pages, actually forces you to think about your core business. What is it that you’re trying to convey? If a page’s headline is getting too complicated, it might be a good idea to think about making multiple pages for what you’re trying to do, for instance.

Don’t assume people know you

When you’ve figured out what your core business or purpose for a page is, it can actually be quite hard to write this down in such a way that it’s understandable for everyone. As Michiel said in his post about clarity and focus:

We want clarity. It’s the experience we get when visiting that website for the very first time: do we get the website? Is it clear what the benefits are for us visitors? And where or how can I get these benefits?

Don’t think that people will understand what you’re talking about, just because they’re on your site. Try to put yourself in the shoes of a person visiting your website for the first time. Is it clear what you’re offering by just reading the headline and/or tagline? For this post the headline is “About headlines and taglines”. That’s simple, short and clear. People will know just by reading that headline what the post will be about.

Of course, for a single post it’s much easier to write what it’s about than for your entire website or business. So you should take some time in formulating your core business and goals. If you think you found a decent headline, it might be a good idea to have someone outside of your business to check it out and let them tell you if they understand. And keep doing that until they understand.

Headlines that work

There are quite a few ways you can make sure people will understand your headline and even fulfill the goal of your page. I’ll give you a few tips here:

Action oriented headlines

If possible, try to write your headlines in an action oriented way. You can do this by using verbs and sentences that imply an action for the visitor. For instance, we could have a headline saying: “Keep your site optimized with the WordPress SEO Premium plugin!”. This shows people one of the core values of the plugin, and making it active will motivate a lot more people to actually try it.

Combine your headline with a tagline

Combining your headline with a tagline will give people a very concise and clear explanation of your headline. Not all headlines need one per se, but usually the effect of having a tagline (or sub headline, whatever you want to call it) is a positive one. The tagline is another chance to make sure people will be intrigued and to clarify for your visitors.

Have a clear headline

This is probably the hardest to do, because how do you know what is clear to people? This is something you should definitely test. But in any case, you need to make sure it’s as specific and concrete as possible. Again, it all comes down to clarity.

Be clear!

By now you must get my main point: you have to be clear! If people can’t understand your website or what you’re selling/offering, they won’t stick around to figure it out. Especially when people are looking for something, they won’t do anything else than just scan the page they land on. So be sure that what they’re scanning is as clear and concise as possible.

What do you think? Have you tried some nice headline variations that worked out really well for you? Or have anything else to mention? Let me know in the comments!

This post first appeared as About headlines and taglines on Yoast. Whoopity Doo!

One of the things we notice are hardly ever optimized on most websites are forms. Forms are an integral part of just about every website. However, for some reason, people don’t (want to) spend much time thinking about these.

There’s a lot of different forms your website can have. However, I think most of them boil down to this: a contact form, a ‘request a quote’ form or a checkout form. These three all have different ways of how to optimize them, as the goal for them is different.

In this post I’ll share my views on how you should optimize each variety and how to optimize forms in general.

You need to have a form

This might sound silly, but the amount of websites we encounter during our Website Reviews that don’t have a contact form or any other kind of form where they should definitely have one, is worrisome. Some websites don’t even have a contact page.

Oh and just so we understand each other; a mailto link on a page is not the same as a contact form.

Focus your form

Just as with everything else on your website, you need to focus your forms. No matter what kind of form it is, make sure you break it down to the bare necessities. If you don’t need personal information, don’t ask for it. The fewer the fields, the shorter the form and the higher the response rate on that form.

These fields do need to be clearly visible. Don’t be tempted to use ‘slick’ form design that will basically make the fields transparent. It must be clear where people should enter the details.

And, of course, make sure your call-to-action (the send button, in this case) is visible and clear. We can’t stress this enough. If people can’t find that button, then all the form optimization in the world won’t help you.

Decent erroring and validation

One of the other things that can be implemented for any kind of form are inline erroring and validation. People don’t want to be told they’ve made a mistake filling out your form after they already hit the send button. Make sure these errors and validations appear right when the user is done filling out the field.

Not only can you be too late with your messages, you can also be too vague, or just completely wrong. For instance: don’t tell people a certain field is not filled out correctly, but rather tell them what’s wrong with how they filled it out, so they don’t unwittingly repeat the mistake (multiple times in a row…). Or, another great one: do not tell people their password for a login is wrong, when in fact they’re entering the wrong email address. I can go all day with this; my annoyances are many.

It’s so simple: make sure people understand how to fill out your form and, in the rare event of a mistake, make the errors so ridiculously clear, they won’t ever be able to reproduce it.

Contact forms

Contact forms are probably the most common kinds of forms you’ll find on the web. However, we see a lot of contact forms that look like no one ever thought about them.

Privacy

I always look at the contact form as the equivalent of someone coming up to a salesperson in a physical shop. In an actual store, you’d expect the salesperson to be open, friendly and helpful. Now consider what you’d think about that salesperson if he/she was telling you “Tell me your name, date of birth and occupation before you ask me any questions!”. That would probably result in you walking out the door, right? So why should your website be any different?

Of course, I’m exaggerating a bit (although I have seen contact forms asking this and more), but the point remains clear. When it’s possible to walk up to someone and ask a question right off the bat in a physical store, this should be possible on websites as well.

Tone of voice

The tone of voice is very important for people to feel safe enough to ask a question. And tone of voice is just as much determined by what’s said as by what’s not said. So not having any text near your contact form will not absolve you from needing a decent tone of voice. You need to invite visitors to ask you a question and explain the process of how you’ll handle their question. The last thing you want to do is confuse or piss off a customer right before they wanted to ask you a question. They’ll either leave your site or they’ll be angry while contacting you.

‘Request a quote’ form

I wanted to make a separate section of this kind of form, because it can definitely be helpful to ask for more information in a ‘request a quote’ form. It all depends on the setup of your business, your website and your form. The same premise still holds though: if you don’t really need certain information, don’t ask for it.

Pricing information

Half the time I see a ‘request a quote’ button or form, I feel kind of cheated. These kinds of forms really make me feel the website owner just doesn’t want to share their usual pricing, because they can see if they can make more money off of you. So you should be clear about the reason you’re not communicating your prices. Preferably even show people a price range they can expect. This will not only filter out people that aren’t interested in that price range, but it’ll also give people like me a better idea of what they can expect. And that alone can help increase your response rate and conversion rate.

Checkout form

I’ve done a post on how we optimized our checkout page in November 2013, and most of that design still holds. There are some things you should definitely implement though:

Progress bar

This actually goes for every form you have on your website. You should implement a progress bar that shows people how far along in your form they are. This gives them a lot of insight in the whole process and will most likely increase your form’s completion rate.

Clarity

Try to make it as clear as possible what all the options and details of the transaction are. What is the price, the product they’re buying, the payment options, etc. All of this should be very clear and quickly visible. The use of logos, product images, etc. can really help in this process.

How are you doing?

Are your website’s forms up to par? Or do they need some more optimization? If you’re not sure or want to have us take a look at them (and the rest of your site), take a look at our Website Reviews!

This post first appeared as Your website needs (better) forms on Yoast. Whoopity Doo!