kauai thumbIt is a crime against our customers if we don’t have a robust web Experimentation & Testing program in our respective companies. That is a bold statement but a good testing program is truly game changing in multiple ways.

(If you have not read the Experimentation and Testing Primer post I recommend that as foundational material for this post.)

The good news is that Experimentation and Testing is increasingly being accepted as something any decent web program should constantly be engaged in. The wonderful thing is that now technology makes it increasingly easy to test your ideas and insights, cheaply and quickly.

We are not limited by the long release cycles on our websites or “IT” or “Marketing” or other such usual hurdles. In the simplest model a simple javascript tag is placed on the site and then using your vendor’s system you can run tests you want and measure success without needing extra work by your developers or IT staff.

Yet in many companies testing is not quite ingrained in the culture or we are stuck in the simplistic A/B or bare bones Multivariate tests (see the link above for details of what these are).

So what does it take? Here, IMHO, are seven recommendations to build a great experimentation & testing program:

# 7 First get over your own opinions:

It really sucks but this is of paramount importance. If you are running the program it is important that first you get over yourself. If you are going to convince everyone else that testing and validating opinions  should be a way of life, you should first truly drink the kool-aid. (IMHO the more entrenched the opinion in a company the more likely it is that it is wrong.)

When I go out and talk about testing I make the case of it by sharing the most recent three or four times testing has proven me to be disastrously wrong. I share the worst stories, the big losers. The point I am making is that we don’t really represent the customers and validating opinions is great. 

Bottom-line: You will get a receptive audience and change minds much faster, because you are willing to “open the kimono.”

# 6 State a hypothesis, not test scenario:

Most often people will come to you and say, I want to run a test different box shots, can you swap this image with text, we should try different promotions etc. 

The golden rule is: Always start with a hypothesis, not test details or test scenario. Turn to the person and say “what is your hypothesis.”

It is amazing how many times people are taken aback by that. Mostly because we as humans don’t want to put that much thought into anything.

The magic of this question is that it forces people to take a step back and think. They might come back to you and say “my hypothesis is that images of people are much more powerful at making a connection than current box shots hence we will have a higher engagement score (or sales or whatever)” or “my hypothesis is that visitors to the site are more interested in user generated content than our company propaganda”.

Bottom-line: Two great outcomes: 1) You can now contribute to the creation of the test, rather than just starting with a “I want you to do this” 2) In every well crafted hypothesis is a clear success measurement (how we’ll know which test version wins). If you don’t see a success measurement in the hypothesis then you don’t have a well thought out hypothesis. 

# 5 Create goals/decisions before hand:

Another big mistake that is often made is that even if the success metric is known we don’t bother to set parameters to judge the “victory” by.

Decide what the success metrics for the test are before you launch and don’t forget to create a goal for those metrics. So you are launching a test to improve conversion rate. Great. By how much do you think you’ll improve the conversion rate?

Frequently that thought has not been put in up front but it is extremely critical for these two reasons:

  • It forces you to think, to do some research as to what the current trends in those success metrics are and go through a goal creation exercise for your test.
  • The awesomely cool outcome of this is that you’ll be able to judge if you should do the test in the first place.
    So if testing dancing monkeys on your home page will only improve conversion rate by 0.001% (your goal) then maybe that test is not worthwhile, you should think of something more powerful. If you do $10 billion in sales on your website then clearly a 0.001% lift will endear you to your company leader / VP /CEO / guy-gal with bigger title than yours.

Bottom-line: This will push the thought envelope and at the same time encourage creation of tests that will yield more powerful customer experience improvements on your site.

# 4 Always test and validate for “multi-goal”:

Almost all testing is “single goal based”, especially the current swath of multivariate testing companies. Put this javascript box on this and that page, put this javascript tag on the goal page, crank the leaver, sing happy birthday, eat cake and here are the results.

Life and customer experiences are significantly more complex. Visitors come to your website for multiple purposes and if you use the current multivariate or other such testing tools then work to integrate other tools that will allow you to measure the impact of your test on all those other purposes.

Simple example of multiple purposes: Visitors come to your home page to buy, to find jobs, to print product information, to read your founder’s bio, to find your tech support phone number etc. If you only solve for conversion rate you might be majorly and negatively impacting your customers. Do you know if you are for tests you are running?

Simple example of tools integration: You’ll get statistical significance and single goal success from your testing vendor. But with small smarts you can integrate your testing parameters with your survey tool – say ForeSee (so you can measure conversion rate and Customer Satisfaction and get open ended customer feedback in each test version) or you can integrate testing parameters with your clickstream tool – say ClickTracks (so you can measure conversion rate and Customer Satisfaction and Click Density and Funnel Analysis for each test version). [See Disclaimers & Disclosures.]

Bottom-line: In the first few months measuring “single goal” will work and happiness will prevail. Only by moving to truly multi-goal will you be able to make the most optimal decisions, and in turn create a program in your company that will sustain and be a long term competitive advantage.

 # 3 Accept “simple” / “silly” initial tests, but remember “if you pay peanuts you’ll get monkeys” :

Testing programs are run by people who are really really smart, you : ). Usually though they have not truly followed #7 above. When business users come to them with simple tests those are scoffed at and scenarios of deeply complex “Albert Einstein” tests are provided. This is the wrong thing to do in early stages of the program.

For the first little while we want to win over fans, we want to achieve mindset shifts. The best way to accomplish this is to accept and run the first few simple tests that originated from the minds of your users. You will thus get them involved in testing their ideas and win or lose you have shown them the power of the tool / methodology.

There is a important thing to remember though, your end goal should to have the mindset shift proceed towards the direction of doing complicated “big” tests, ones that put a lot bigger things on this line and not just play with the hero image on the home page.

Bottom-line: In early stages test your the ideas of your users, not matter how small or big, but drive your program to doing more complex and fundamental tests over time. 

# 2 Create a fun environment:

We often forget that this is supposed to be fun. What other situations can you think of where you can “play” with your customers, and they never find out? That is awesome. Testing should be fun, it should be enjoyable, it should stretch our brains.

One really simple recommendation is to get everyone involved in the test to bet on the outcome (only in US states where betting is legal). Everyone loves betting and since these might be their tests they might really like the odds of winning. Stay with something small, one dollar for every prediction of the success metric or which version of the test will win.

What happens in reality is throughout the test duration people will keep checking which version is winning, thus learning complex measurements. After the test winner is declared you can bet the “losers” will want to pour over every success metric and its definition and computation until they realize that they really did lose, but they just learned so much more than you could have taught them in any other way. : )

 Bottom-line: Learning should be fun. (Now where else have I read that. : )

# 1 Two absolutes you need: Evangelism & Expertise:

The number one recommendation is that you need two key people in any successful program, one Testing / Experimentation Evangelist and one Testing Expert.

Most people don’t yet get the testing religion, to convert them you will need a evangelist. Not just someone who “gets it” but someone through their communication skills, pure love, business understanding, position can go out there and preach and articulate the value proposition. If this person does not know what “r squared” is that is ok.

To run your program and actually execute much of the above you need s Testing Expert. Someone who is seeped in metrics and data and complex computations and has enough business expertise to look at tests and provide good feedback and even help generate great value add new ideas (and push back politely on the non value add ones over time). This person should meet #2, #4 & #5 Criteria from the Great Analyst post.

This position is important because the testing world is young on the Web and increasingly even if the vendors try to convince you of how great their tool is over the other guys the biggest challenge is great ideas to test and accurate success measurement.

There are many good vendors (we use Offermatica for multivariate testing). Differences between tools will not be your limitation for quite some time (testing ideas, culture, program sophistication, implementation etc will be). So if you like Dave Morgan because he posted on your blog then go with SiteSpect : ). But don’t forget your Expert.

Agree? Disagree? Have alternative recommendations? Got questions? Please share your feedback / critique via comments.

[Like this post? For more posts like this please click here.]

Social Bookmarks:

  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite