An Introduction to Website Split Testing

Jan 11 2010 by Alex Cleanthous | 36 Comments

It’s a fact of life that when people hire a web designer, they don’t just want a website, they want a website that does something! There can be a world of difference between these two things. The "action" they need the website to take for them can be one of several common things: selling products for their business (an e-commerce site), generating sales leads, and/or providing free information in the hope that the visitor will make a purchase from the company at a later date.

These aims differ slightly for some sites, like non-profits, blogs and communities; however business sites in general aim to generate revenue by one of the above methods.

Sometimes the client will split up these functions between two separate businesses – the website development will be taken care of by "Best Ever Web Designers Incorporated", while the marketing is done by "Your Ad Here Pty Ltd". However, if the push towards action (marketing) is created at the web design level, rather than externally, it is much more effective. The more effective your client’s website, the happier they will be; and the happier your clients, the better your web design business thrives!

More than that, you can create an additional ongoing revenue stream that requires a little effort each month at your end but can provide a significant benefit to your client in the form of increased sales or sales leads for their business.

Here we look at how to determine which aspects of your web design work is most effective to help the client’s website achieve its aims, by a process known as split testing.

What is Split Testing?

In website split testing (also known as A/B testing), a software application splits the traffic between two or more variants of the same web page, measuring whether there is a difference in the effectiveness of that page in achieving the website’s goals. Every time a visitor takes the desired action on a website, such as making a purchase or submitting a form, it is called a conversion.

Web designers might check whether:

  • A landing page with a graphic of a person encourages better conversion rates than one with a graphic of the product
  • A different positioning of the shopping cart button encourages more people to click it
  • A form with fewer elements, or different elements, encourages more people to finish filling it out.

Just about every aspect of a website can be split-tested.
Split testing involves a single element of a page being changed at one time. A different type of split testing is sometimes used to achieve faster improvements with a shorter number of tests; multivariate testing. In this type of testing, multiple variables are changed within a single web page. So, rather than having multiple versions of the same page, multivariate testing rotates different elements on the same page, for example, the headline, the call-to-action, the image and the opening paragraph.

The advantage with multivariate testing is that you can find the optimum combination of website elements in a shorter space of time and achieve significant improvements in the website’s performance; the disadvantage is that it takes a considerable amount of traffic to get accurate results.

For most websites, A/B split testing will be the optimum method of improving a website’s conversion rate over time.

Why Bother With Split Testing?

Split testing is what separates elite web designers from the rest because it provides them with the knowledge of which combination of website elements will produce the optimum results for the client and they can demand a higher rate as a result. Split testing can help:

  • Get visitors to stay longer on the website
  • Get visitors to interact more with the site
  • Get more visitors to request a consultation or join a newsletter
  • Get more visitors to buy from the site

It can also achieve these aims (depending on what the site is set up for) at a lower cost per action (CPA) than competing sites in the same niche. In some ultra-competitive online markets (home loans, insurance, accommodation, SEO, online dating, etc) split testing is the only way to guarantee that you will be heard above the noise of the competition.

Split testing also helps ensure that when it comes to advertising, you are maximising each and every dollar. If your website doubles its conversion rate from 1% to 2% using split testing techniques, you have just doubled your revenue using the same marketing budget. You will be able to spend more on advertising your site and earn a better return on investment (ROI) from each dollar you spend. Sounds like you’d be crazy not to, doesn’t it?

It becomes even more exciting when you start to examine the numbers. Many untested websites have conversion rates that run around 0.25%. This means that it takes 400 visitors to get one sale. With split testing, you don’t need to increase the number of visitors, just the rate at which they convert. Many success stories tell of conversion rates upwards of 10%. That would be the equivalent of a 40-fold increase in visitors, all of a sudden giving you 40 sales or sales leads out of 400 visitors, instead of 1.

Who Should Be Split Testing?

Ideally, the web designer should start split-testing as soon as the site launches, as they are the ones who have designed the site and have all of the source files on hand. Even if the owner feels that the website is performing well enough, the benchmark for "good" performance could be moved even higher through using split testing. Just keep in mind that split testing does require an established traffic flow because you need to people visiting your site to be able to split-test web pages.

Why is it ideal that the designer does it at the launch stage? Because it is much easier to keep the site as a work in progress after launch, not only visually and structurally, but in terms of its goals. It is infinitely harder for a third party to come along later and change things around, and in many cases, you wouldn’t want them to. That site’s your creation!

Doing the split testing yourself, as a designer, also gives you a real-world feel for what elements will work and what ones won’t. It adds an extra dimension to your sense of aesthetics, removing the guesswork that you necessarily engage in during the design phase. You are no longer doing what you think will create conversions, but implementing what you know produces measurable results. And this knowledge alone will increase the value you bring to your future projects.

Setting Up a Split Test with Google Website Optimizer

Google Website Optimizer is an A/B split testing and multivariate testing platform that allows you to easily see the effects that making changes to page elements has on your website conversion rate. It’s a free platform, and is robust and user friendly. One of the major advantages, though, is that you can use the Google Website Optimizer with existing web page URLs (e.g. yourdomain.com); some other split testing applications require you to use a custom URL (e.g. yourdomain.com/tracking.php) which makes it very difficult to split test non-paid website traffic, such as SEO traffic, referral traffic or direct traffic.

Google Website Optimizer serves different pages to different users, with hundreds of variations possible. In our experience, using the website optimizer tool does not affect your rankings and does not cause duplicate content issues. Here is the process for setting up an A/B test with Google Website Optimizer:

Identify the page or pages you want to test (usually the home page or pages that receive a lot of traffic).

Define which element you want to test. In A/B testing, you’ll only test element one at a time. Commonly tested elements are the headline, sub-headline, opening paragraph, image, call to action, and the submit button (look and/or placement).

Add tags to the original page, the variation page and the conversion pages to allow tracking.

Enter the website URLs you are testing into the Google Website Optimizer.

Preview how the pages will look before sending them live.

Google Website Optimizer will rotate the page that is displayed for every visitor, so that the first person sees version 1, the second person sees version 2, the third person sees version 1 and so on.

You can access the results and reports of various statistics within Google Website Optimizer to see which variant is the most effective.

What Should You Test with Google Optimizer?

These are the most commonly tested elements of a web page:

  • Headings
  • Sub-headings
  • Opening paragraph
  • Images
  • Call to action
  • Offers
  • Hyperlinks
  • Submit button
  • The location of different elements
  • Colors

The first six elements are the ones which generally have the most impact on conversion rates, and the ones you should test first. Google once tested 41 different shades of blue for their pages. They have the luxury of doing this because they get more traffic than any other website, and they have the drive to do this because performance is their life. They thrive on it! You don’t need to go to this extent but if Google places so much value on it, shouldn’t you?

Knowing When a Split Test is Finished

The number of conversions needed to garner a reliable indication of future traffic varies incredibly with different types of sites. If the business you are designing for is a high-customer volume, low-value spend store, you should test until you have 70 or so conversions for one variant. If it is a low-customer volume, high-value spend store, aim for 20 conversions.

If the set conversion is further from becoming an actual "profit" for the store (i.e. a conversion creates a lead from a visitor, rather than a sale from a lead), you may want to be more certain that one variant is working better than the other. You may also want to test more different variants of the same element, or do multivariate testing … within the limitations of your traffic volume.

Determining statistical significance at the conclusion of a test

For you to have a high confidence level that the test results aren’t just a fluke, you can apply a simple formula to determine whether the difference in results is statistically significant. This rule is:

The numerical difference between the two results must be greater than the square root of the sum of the two results.

Or

Y-X>√X+Y, where Y>X

Sounds like a nightmare from before your high school maths exam, doesn’t it? It is easier to understand with an example:

  • If the result of page A was 20 conversions, and the result of page B was 30 conversions, then the difference between the two is 10.
  • The sum of the two results is 50
  • The square root of that sum is 7 (rounded down)
  • The difference (10) is greater than the square root of the sum (7), so the result is statistically significant.

If the two results had been 22 and 28:

  • The difference between the two is 6
  • The sum of the two results is 50
  • The square root of that sum is 7 (rounded down)
  • The difference (6) is less than the square root of the sum (7) – the result is not statistically significant.

If the results of your split test are not statistically significant, you can either:

  • Keep testing and see if a more definite pattern emerges
  • Decide that the element doesn’t make a difference to conversion in this case, and test something else.

Tips for Ongoing Split Testing

Either you, as the designer, or the website owner, should be implementing split testing constantly, on all your high traffic web pages. It takes time to get test results, and in the meanwhile customer preferences and market sensibilities are changing, along with the season and your stock.

It is important to note that not all split tests will be successful. In fact, if you have a 20% success rate where one in five split tests improve the overall conversion rate of the site then you are doing well. The key point to remember with split-testing is that every success forms a new baseline, with considerable website performance improvements achieved after a number of different split tests.

The easiest way to manage all of the split tests is to set one day per month when you review results and then set up tests for the coming month based on those results. Every month you should track your results in a spreadsheet to avoid accidentally re-testing the same elements over and over again. This also provides a point of reference for when you are starting new projects – you can see which placements and copy types are most effective in which industries.

If your website grows to the point where you have upwards of 1 million visitors per month, you can use multivariate testing.

And don’t forget to check what competing sites in the same niche are doing, and test the same elements on your site!

Once you start split testing you’ll find that what you thought would perform better doesn’t and you’ll be very surprised with some of the results. The best part about split-testing is that the confidence you will gain from split testing will help you with your future web development projects because you will have an increased knowledge of what produces results and what doesn’t. Consumer behavior is a specialized field and web designers can get on a steep but exhilarating learning curve for it by engaging in split testing on their own and their clients’ websites.

Related Content

About the Author

Alex Cleanthous is the Head Online Strategist of Web Profits, an Australian-based online marketing firm servicing Australasia, the US and Europe. He keeps Web Profits on the cutting edge in all aspects of online marketing. Alex writes an industry blog about success online. Follow Alex on Twitter: @AlexCleanthous.

36 Comments

Visual Website Optimizer

January 11th, 2010

Hi Alex,

You may also want to check Visual Website Optimizer, our hassle-free A/B split testing tool which lets you design split tests by selecting sections on your page in a visual designer and creating variations in a WYSIWYG editor. Also, unlike GWO, it doesn’t require page tagging and other JavaScript code hassles. You just need to add a small snippet of code once to your website and then you can create unlimited number of A/B tests from the visual website optimizer interface, without needing to touch your website code ever again.

I just thought of mentioning our new tool here because it is a viable alternative to GWO.

The tool is currently in private beta, but if you want to try it out, please use invite code “vwo-user” (without quotes) while signing up at http://vso.com

Thanks!
-Paras Chopra
Founder, Visual Website Optimizer

James Costa

January 11th, 2010

Excellent article, Alex! Thanks for providing such a thorough introduction to A/B testing. :-)

Dimi Arhontidis

January 11th, 2010

Very well written, thanks for sharing!

Webby freebies

January 11th, 2010

That is a really interesting article!
Thanks for sharing Alex.

Doris Moller

January 11th, 2010

Thank you for your most helpful article on split testing.

Doris Moller

RJ McCollam

January 11th, 2010

Really cool article. I worked for a company where I was always making changes to their landing pages, but I never got involved int he split testing. Some really good stuff.

Pravin

January 11th, 2010

Nice article…Thanks…keep writing…

Antone Roundy

January 11th, 2010

The formula for checking statistical significance in this post is an easy rule of thumb, but depending on your sample size (ie. number of impressions), it may overstate the significance of the results.

For example, if you had 20 and 28 conversions, the difference (8) is greater than the square root of the sum (48 — square root is just less than 7). But let’s say for example you had 400 impressions of each (5% and 7% conversion rates), your “confidence interval” (which is essentially the probability that the difference between the two conversion rates was NOT simply random) would only be about 77%.

Statisticians generally want the confidence interval to be at least 90% (or better yet 95%) before calling a result statistically significant. To get a 90% confidence interval with 400 impressions of each and 20 conversions of one, the other would need 32 conversions. For 95%, the other would need 34 conversions.

If, on the other had, you only had 50 impressions of each (40% and 56% conversion rates for 20 and 28 conversions), the confidence interval would be 89.5%, which you might be safe rounding up to 90%.

In my own split tests, I’ve OFTEN seen tests where the results seemed to be going in favor of one candidate with confidence intervals up into the 80′s only to have it turn around and the other candidate end up clearly winning after more data was collected. It pays to let your tests run until you really have strong statistical significance.

BTW, if you click my name, it’ll take you to a page on my website that calculates statistical significance more precisely.

…but if you want to use the rule of thumb given above, just be aware that it may be overstating the significance of your results, so make sure the difference is more than just barely greater than the square root of the sum.

Smashing Share

January 12th, 2010

Very informative and in-depth. Thanks Alex for sharing your knowledge with us

Noel Wiggins

January 12th, 2010

What a tremendous article, I have had the desire for trying google optimizer longer than I wish to admit. I am convinced that this is the only way to go. and I think today may be the actual day that this process will start.

Convincing clients of this concept, I don’t do split testing as of yet for clients but I do a weekly “stats analysis” where we go through the stats and based on the numbers make tweaks to the design to see if those tweaks helped or hurt.

Some clients get it while others in the wooing stage think I am just soaking them for money, they try and educate me on how they just want a website and thats it. I guess those are classified as red flag clients. But in this recession I can afford to be picky with who I decide to work for.

So here is to hoping that by me split testing my site I might be able to attract better leads that will be more open to the process.

Helio

January 12th, 2010

I think Google Website Optimizer is best for doing this. I had test it once on my website. It helps…………Thanx

Shawn Hooghkirk

January 12th, 2010

Great article Alex. I completely agree with split testing expanding your knowledge base. Just knowing certain information like the placement of a call to action button, or even the color of that button is crucial for new projects.

Another method is speaking to others that are split testing and finding out their results — While also sharing your own.

Nick Monkhouse

January 12th, 2010

Thanks Alex – great article. Helpful to have a formula to tell if feedback is statistically valid or not.

Tutorijali HDonWEB

January 12th, 2010

Great tips Alex :-) … think I’ll try to do split testing on my blog

Leb

January 13th, 2010

Great article. Split testing helps eliminate the guess work and improve the conversion rate of websites. Still many website owners ignore this.

alisha

January 13th, 2010

nice and informative artical ragarding testing point of view.

Eric Di Bari

January 13th, 2010

I know google has an interface to do this through analytics. Good idea.

lon

January 14th, 2010

I think website testing is as important as the design. The problem is alot of designers neglect this. They are too focused on the design that they forget people apart from them will have to use it. You need to design with the end user in mind. User testing helps facilitate that.

Matt Trevino

January 14th, 2010

Great article! Thanks for the sharing such in-depth information.

uttam

January 19th, 2010

Nice article! Thanks for sharing

Justin Moore-Brown

January 26th, 2010

Great article!

I had no idea about split testing but will definitely try out the Google Web Optimizer!

Richard

April 5th, 2010

You can measure statistical significant using most common A/B testing calculators: http://mixpanel.com/labs/split-test-calculator

John Hyde

April 19th, 2010

Great coverage of the subject.

We’ve had 43% 38% and 29% boosts this month just from changing words on pages.

We’ve also run 6 tests that made no difference and one that bombed.

And April isn’t over yet…

Divya

April 20th, 2010

Nice article ..

jatt

May 26th, 2010

oh great article. this give a wounderful teasure of knowledge. split testing in google optimizer is unstopable may it give comfortune to the optimizer

jatt

May 26th, 2010

i will go with dear divya and again say that its nice article.

Craig

September 15th, 2010

Wow! Great article on website split testing, yet another aspect I will need to put some time into, the calculation for statistically and non statistically is another golden nugget to remember from Six revisions, Thanks!

Sagar S. Ranpise

October 4th, 2010

A Very indepth information and easy to understand. Thanks a lot for sharing your knowledge.

Kamal Patel

November 20th, 2010

WoW…What a Great article
Thanks for sharing, really loved to read many times.
I had used before but i can use it with more depth idea on that and hope increasing the leads as well.

Robert Steers

February 19th, 2011

Excellent introduction to split testing! Great tip too for ongoing split testing!

Lisa Noble

March 2nd, 2011

Great information! I am going to implement this into my process on the project I am currently working on and going forward.

Julie

April 8th, 2011

Nice overview. It is certainly a useful tool when it comes to conversion rate optimization.

BerkshireKate88

June 21st, 2011

Very useful post, I’m considering some split testing at the moment. Thanks.

Amara

June 27th, 2011

Amazing article! Thank you! :)

C. Payne

October 19th, 2011

Wonderful article! I plan to implement this process immediately for my company’s site.

Leave a Comment

Subscribe to the comments on this article.