[Printable] Split Testing and Statistical Significance 1-Pager

n test If you’ve had the chance to read The The 4-Hour Chef by Tim Ferriss then you know that he’s kind of hooked on the idea of one-pagers.

In essence, a one-pager is a very condensed resource about a given topic. It’s meant to list only the essential, and only the bits that will give you the most results while at the same time requiring the least of your input/effort.

I’ve liked the idea of one-pagers right away the minute I saw them. That’s probably because I like structured information. I like structure in general. Okay, I’m a Structure Nazi (like a Grammar Nazi only less mainstream).

Seeing that one-pagers are a great tool to convey somewhat complex ideas in a relatively easy to grasp manner, I’ve decided to use them on this blog.

My first target, as you can see in the headline – split testing and statistical significance.

Split testing and statistical significance 1-pager

Preliminary

Goal

Every split test needs a goal. This goal must be:

Precise.
Measurable.
Attainable.

Subject

The subject is the thing you are testing.

It can be an element of your website, email copy or whatever else that will help you achieve the goal.

Parameters

Defining the goal will allow you to shape your test’s parameters. Those parameters are the exact elements you are measuring or tracking.

You need two types of parameters:

Trials.
Actions/Conversions.

Examples

Goal:

Get more email subscribers to your list.

Subject:

Your email subscription form.

Parameters:

Number of impressions/views of the form.
Number of conversions (people who clicked the subscribe button).

Carved-in-stone rules

When running the test:

Test only two versions of your test subject.
Test each version for at least a month (both versions can be run in parallel).
Test each version for the same amount of time and in the same conditions.
Avoid data pollution – don’t test anything else on the pages where your split test is running.
Get to the critical mass – at least 300-500 trials for each version.

Check statistical significance

Checking statistical significance will let you know if the results you’re getting from your split test are by any chance accidental.

For that, you can use my statistical significance calculator >>.

If your results are significant, you can name the winner and the loser of your test.

Next step: scrap the loser, create another version of your test subject and run it against your winner in a new test. In other words, start over.

Don’t over-interpret

The main thing to keep in mind when split testing is not to over-interpret your results. Just because there is a winner and there is a loser, doesn’t meant that you will be able to tell exactly why the test has turned out the way it did. The reasons can be many.

It’s a lot safer to just take the results as they are, and not build theories trying to explain them. Most of the time they turn out to be false anyway. Your time is much better spent creating a new version and running the test again.

Your downloadable copy.

What’s your relationship with split testing? Do you even lift split test?