We all wish that our key internal partners, business decision makers, would use Web Analytics data a lot more to make effective decisions. How do we make recommendations / decisions with confidence? How can we drive action rather than pushing data? The challenge is how to separate Signal from Noise and make it easy to communicate that distinction.

This is where Excellent Analytics Tip #1, a recurring series, comes in. Leverage the power of Statistics.

Consider this scenario (A):

You do send out two offers to potential customer. Here is how the outcomes look:

• Offer One Responses: 5,300. Order: 46. Hence Conversion Rate: 0.87%
• Offer Two Responses: 5,200. Order: 55. Hence Conversion Rate: 1.06%

Is Offer Two better than Offer One? It does have "better" conversion rate, by 0.19%. Can you decide which one of the two is better with just 40 to 50 responses? We got 9 more orders from 100 fewer visitors.

Applying statistics tells us that the results, the two conversion rates, are just 0.995 standard deviations apart and not statistically significant. This would mean that it is quite likely that it is noise causing the difference in conversion rates.

Consider this scenario (B):

You do send out two offers to potential customer. Here is how the outcomes look:

• Offer One Responses: 5,300. Order: 46. Hence Conversion Rate: 0.87%
• Offer Two Responses: 5,200. Order: 63. Hence Conversion Rate: 1.21%

Applying statistics will now tell us that the two numbers are 1.74 standard deviations apart and the results rate 95% statistically significant. 95% significance is a very strong signal. Based on this, and only a sample of 5k and sixty odd responses, we can confidently predict success.

Is this really hard to do? No! Simply use this spreadsheet: StatCalc.xls. (While we have a "enhanced" version of this spreadsheet this is the original file we found on the web and the file contains credit to the original author Brian Teasley.)

All you do is simply punch in your numbers in blue highlighted cells and you are on your way. This methodology can be easily apply to all facets of your insights analysis, including:

• Search Engine Marketing Campaigns
• Various Direct Marketing Campaigns and Offers
• Any kind of % metric (% of traffic that reaches a goal from Entry Point 1 or Entry Point 2)
• Differences between results for you A/B or Multivariate tests

You can easily adapt the spreadsheet, as we have, to compute statistical difference between absolute numbers (say you want to know if the difference Page Views Per Visitor or Average Time on Site between segment One and Two is Significant)

Powerful benefits to presenting Statistical Significance rather than simply Conversion Rate:

1. You are taking yourself out of the equation, it is awesome to say "according to the God's of Statistics here are the results…"
2. Focusing on quality of Signal means that we appear smarter than people give us Analysts credit for.
3. You take then thinking and questions out of the equation. Either something is Statistically Significant, and we take action, or we say it is not Significant and let's try something else. No reporting, just actionable insights.

Here is one more great resources for tools / spreadsheets that I would like to point out if you want to get deeper into this way of thinking:

Two small tips:

1. This is a best practice but aim for 95% or higher Confidence. That is not always required but it is recommended.
2. "Statistics are like a bikini. What they reveal is suggestive, but what they conceal is vital." –Aaron Levenstein