February 2009


18 Feb 2009 01:51 am

above average 1Yes. I noticed the slightest hint of sarcasm in the title of this post.

This post covers four commonly used measurement techniques that 9 times out of 10 work against the evolution of Reporting Squirrels into Analysis Ninjas.

I'll also admit that most of the times when I encounter them I might think slightly less of you (especially if you present the aggregate version to me rather, presenting the segmented view atleast gets you time to explain :)).

If I am being slightly tough minded here it is only because I am hugely upset by the fact that analytics on the web is deeply under leveraged, though the good lord knows we try and pump out KPI's by the minute.

One root cause of this under leveraging it our dashboards that are crammed full of metrics that use these four measurement techniques. The end results: Data pukeing and not insights revelation.

So who are the four amigos?

Averages.

Percentage.

Ratios.

Compound Metrics (aka Calculated Metrics).

Each a technique that when used "as normal" actively hinder your ability to communicate effectively the insights that your data contains.

Only one caveat: I am not saying these techniques are evil. What I am saying is don't be "default" when using them, be smart (or don't).

Before we get going here's my definition of what a Key Performance Indicator is:

Measures that help you understand how you are doing against your objectives.

Note the stress on Measures. And Objectives. It it doesn't meet Both criteria its not a KPI.

With that out of the way lets understand why Averages, Percentages, Ratios and Compound Metrics are four usually disappointing measurement techniques.

#1. Averages.

Raise your hand if you are average? Ok just Ray? No one else?

Raise your had if your visit on any website reflects an average visit? Just you Kristen?

No one is "average" and no user experience is "average". But Averages are everywhere because: 1) well they are everywhere, which feeds the cycle and 2) they are an easy way to aggregate (roll up) information so that others can see it more easily.

Sadly seeing it more easily does not mean we actually understand and can identify insights.

average time on site clicktracks

Take a look at the number above.

51 seconds.

Ok you know something.

Now what?

Are you any wiser? Do you know any better what to do next? Any brilliant insights?

No.

It is likely that the Average Time on Site number for your website has been essentially unchanged for a year (and yet, yes sirrie bob, it is still on your "Global Senior Website Management Health Dashboard"!).

Averages have an astonishing capacity to give your "average" data, they have a great capacity to lie, and they hinder decision making. [You are going to disagree, quite ok, please share feedback via comments.]

What can you do?

I have two recommendations for you to consider.

Uno. Segment the data.

Identify your most important / interesting segments for your business and report those along with the Overall averages.

segmented time on site

You have more context. Social Media boo! Paid Search booer! Organic yea! Email yea! Etc Etc Etc. : )

While this is not the most optimal outcome, it will at the very minimum give your Decision Makers context within which to ask questions, to think more clearly (and mostly wonderfully ignore the overall average number).

So on your dashboards and email reports make sure that the Key Performance Indicators that use Averages as the measurement technique are shown segmented. It will prod questions. A good thing, as Martha would say.

Dos. Distributions baby, distributions!

If averages often (*not always*) stink then distributions rock.

They are a wonderful way to dissect what makes up the average and look at the numbers in a much more manageable way.

Here's how I like looking at time on site. . . .

average time on site

So delightful.

I can understand the short visits (most!) and decide what to do (ignore 'em, focus hard core, etc).

I can see there is deep loyalty, about 30%, I can decide what these people like, what they don't like, where they come from, what else I can do. [Would you have imagined from the Average Time on Site that you have fanatics on your site who are spending more than 10% on each visit!!]

I can try to take care of the midriff, what is up with that any way.

See what I mean? The difference between the two: Reporting Squirrel vs. Analysis Ninja!

#2. Percentages.

Nothing, really nothing, is perhaps more ubiquitous in our world of Web Analytics than percentages.

You can't take a step without bumping into one.

Some percentages are ok, but very very rarely are they good at answering the "so what" or the "now what" questions.

The problem with percentages is that they gloss over what's really important and also tend to oversell or under sell the opportunity.

Let's compare two pictures. In the first one we just report conversion rates, see what you can understand in terms of insights fro this one. . . . .

blog conversion rates

Now try to answer the question: So What?

Any answers?

Yes some conversions are lower and others are higher? Anything else? Nope?

Ok try this one. . . .

blog conversion rates with raw values

Better right?

You get context. The raw numbers give you key context around performance.

[Update: I use this plugin to get raw conversion rate numbers into Google Analytics: Better Google Analytics Firefox Extension. I highly recommend it, you get the above and a bunch more really cool stuff. Must have for GA users.]

Also notice another thing, I'll touch on this in a bit as well. If you only report overall conversion rate (as we all do in our dashboards) your use of a percentage KPI is much worse. You get nothing.

By showing the various "segments" of conversions I am actually telling the story much better to the Sr. Management. What's working, what needs work.

Here's another constant problem with conversion rates. . . .

I am looking at a table of data (in any tool really) and it looks like there's something here.

bounce rates keywords

Ok well I want to fix things. I want to know where I can improve bounce rates, so I sort. . . .

bounce rates keywords sorted up

Data yes. Totally useless. I can't possibly waste my time with things that bring one visit.

So I re sort to see if I can find where its totally working for me. . . .

bounce rates keywords sorted down

Strike three, again not very useful, just take a peek at the Visits column.

What I really want is not where the percents are high or low. I want to take action.

What I really really want is some way of identifying statistically significant data, where bounce rates are "meaningfully up" or "meaningfully down" so that I can take action confidently.

I can't do that in Google Analytics. Quite sad.

Some other tools like Coremetrics (by default) and WebTrends (in some places by default or with a external "plugin" you can buy from external consultants) will compute a %delta (difference between two numbers) and color it red or green.

That's not what I am taking about.

That is equally useless because that percentage difference make you take action where there is no significance in the two numbers. Don't fall for that.

It is truly a crying shame that the Google Analytics does not have something like the Google Website Optimizer does. . . .

google website optimizer results

. . . . a trigger for me to know when results are statistically significant, and by how much I should jump for joy or how many hairs I need to pull out of my hair in frustration. See those sweet colored bars in the middle? See the second column after that? Minorly orgasmic right?

Isn't it amazing that after 15 years web analytics tools are still not smart, even though they have so much data and computations. Ironic if you think about it.

What can you do?

I have three recommendations for you to consider.

Uno. Segment the data.

Wait, did I not say that already? : )

Do it.

Useless. . . .

overall conversion rates

Useful. . . .

segmented conversion rates

Show opportunities, show failures, let the questions comes.

Dos. Always show raw numbers.

Often conversion rates mask the opportunity available.

Conversion rate from Live is 15% and conversion rates for Yahoo! are 3%.

Misleading.

We all know that Yahoo! has significantly more inventory than Live and even if you had all the money in the world you can't make use of that 15% conversion rate from Live.

Show raw Visits. It will look something like this:

conversion rate comparison

See what difference that would make on a dashboard? No false alarms.

You overcome the limitation of just showing the percentage.

In the example above I am using Visits, because I want to show the HiPPO's where the constraints are (without them having to think, thus earning my Ninja credentials!). But I am most fond of using Outcomes when I pair up raw numbers (Orders, Average Order Value, Distribution of Time, Task Completion Rates, etc etc) because HiPPO's love Outcomes.

Tres. Don't use % delta! User Statistical Significance et al.

When you use percentages it is often very hard to discern what is important, what is attention worthy, what is noise and what is completely insignificant.

Be very aware of it and use sophisticated analysis to identify for your Sr. Management (and yourself!) what is worthy.

Use Statistical Significance, it truly is your BFF!

Use Statistical Control Limits, they help you identify when you should jump and when you should stay still (so vital!).

This is all truly sexy cool fun, trust me.

#3. Ratios.

Can I be honest with you?
[Ok so I can hear your sarcastic voice saying: "Why stop now?" ;)]

Ratios have a incredible capacity to make you look silly (or even "dumb").

I say that with love.

What's a ratio?

"The relative magnitudes of two quantities (usually expressed as a quotient)." (Wordnetweb, Princeton.)

That was easy. : )

In real life you have see ratio's expressed as 1.4 or as 4:2 or other such variations.

You are comparing two numbers with the desire to provide insights.

So let's say the ratio between new and returning visitors. Or the ratio of friend requests sent on Facebook to friend request received. Or the ratio of articles submitted on a tech support websites to the articles read. Or… make your own.

They abound in our life. But they come with challenges.

The first challenge to be careful of is that the two underlying numbers could shift dramatically without any impact on your ratio (then you my friend are in a, shall we say, pickle). . . .

key performance indicator ratios

I have put my "brilliant" excel skills to demonstrate that point. In your dashboard you'll how the ratio (all "green" for four months). Yet the fundamentals, which is really what your Sr. Management is trying to get at, have changed dramatically, perhaps worth an investigation, yet they'll get overlooked.

I hear you protesting all the way from Spain, "aw come one, you have got to be kidding me!". I kid you not.

Think of all the effort you have put into automating the dashboard and cramming all the data into it. Ahh… you've stuffed it with percentages and ratios to make it fit. And you've automated it to boot.

ratioCasualty? Insights. Actionability.

The second problem with ratios is a nuance on the above. It is perhaps more insidious. It occurs when you compare two campaigns or sources or people or other such uniquely valuable things.

I see it manifested by a HiPPO / Consultant / Vendor Serviceman foisting upon you that 1.2 is a "good ratio".

Then you start measuring and people start gaming the system. Because you see 12/10 gets you that ratio as does 12,000/10,000. Yet they both get "rated" the same and that as you'll agree is dumb.

What can you do?

I have two recommendations for you to consider.

Uno. Resist just showing the ratio.

Throw in a raw number, throw in some other type of context and you are on your way to sharing something that will highlight a important facet, prod good questions.

Enough said.

Dos. Resist the temptation to set "golden" rules of thumb.

This is very hard to pull off, we all want to take the easy way out.

But doing this will mean you'll incent the wrong behavior, hinder any thought about what's actually good or bad.

You can a ratio as a KPI, but incent the underlying thing of value. For example Reach and not the ratio of Visits to Subscribers (!!).

#4. Compound Metrics (aka Calculated Metrics).

Here's a visual for you:

compound

Its a compound metric. : )

A unrecognizable paste you produce after mixing a bunch of, perhaps perfectly good, things.

All kidding aside compound metrics are all around us. Most Government data tends to be compound metrics (is it a wonder that we understand nothing that the government does?).

A compound metric is a metric whose sub components are other metrics (or it is defined in terms of other computations).

Here's an example:

(% of New Visits) times (Average Page Views per Visit) equals, making something up here, Visit Depth Index.

What?

Yes what indeed.

The environments where compound metrics thrive are ones where things are really really hard to measure (so we react by adding and multiplying lots of things) or when confidence in our ability to drive action overtakes reality.

Honestly no matter what the outcome is here (or how much of a "god's gift to humanity" it is) how can you possibly do anything with this:

Website Awesomeness= (RT*G)+(T/Q)+((z^x)-(a/k)*100)

(If you don't know what those alphabets stand for just make something up.)

Compound metrics might be important, after all the Government users them, but they have two corrosive problems:

confusion1) When you spit a number out, say 9 or 58 or 1346, no one,except you has any idea what it means (so a huge anti actionability bias) and worse

2) You have no way of knowing if it is good or bad or if you should do something. You can easily see how a raise in some numbers and fall in others could cause nothing to happen. Or all hell could break loose and yet you still get 9. Or 58. Or 1346.

What can you do?

I have three recommendations for you to consider.

Uno. Take them with a grain of salt (or a truck full of salt).

Really.

Regardless of if it comes from me or President Obama or [insert the name of your favorite religious deity here].

Stress test how you'll overcome the two challenges above. If your compound metric passes those tests you are all set.

Dos. Degrade to key "critical few" components.

Grinding RT and G and T and Q and z and x and a and k into a mush is the problem. Not RT or G or T or Q or z or x or a or k themselves.

Spend some time with your HiPPO's and Marketers and people who pay your salary. Try to understand what is the business really trying to solve for. Put the nose to the grind stone and so some hard work.

At the end of this process, as you decompose the individual components, what you'll realize is that all you need is RT and Q and G. Report them.

No not as a weird married "couple". As individuals.

Everyone will know what you are doing, you help the business and your dashboard focus, drive action.

Tres. Revisit and revalidate.

If you must use compound metrics please revisit them from time to time to see if they are adding value. Also check that they are adding value in all the applicable scenarios

If you are using weights, as many compound metrics tend to do, then please please stress test to ensure the weights are relevant to you. Also revalidate the weights over time to ensure you don't have to compensate for seasonality or other important business nuances.

End of story.

I'll close There are two schools of thought about Analtyics.

One is that math is easy so let's go add, subtract, multiply and divide because calculators, computers and data are easily available.

This is the "Reporting Squirrel" mental model, data above all else.

The other is that your entire existence is geared towards driving action. So think, stress test, be smart about the math you do. Computers and calculators are cheap but it does not excuse doing the things outlined above.

This is the "Analysis Ninja" mental model, insights above all else.

Good luck!

Ok now its your turn. What do You think of these four measurement techniques? Agree with my point of view? Why? Why not? Care to share your own bruises from the wonderful world of Web Analytics Key Performance Indicators? Got questions?

Please share your feedback. Thank you.

PS:
Couple other related posts you might find interesting:

10 Feb 2009 01:50 am

green symmetry 1One of the fun part of my professional life is all the email I get from you. Yes it is an insignificant amount of work but it means I learn a lot about what's on your mind, what you find hard, what you find easy, what challenges bedevil you.

At the moment the "could you help me" / "what do you think I should do" emails are around 50 a day.

[My one small request is that you do a search on www.google.com before you email me, sometimes that also works pretty well.]

Here are some questions from the last couple weeks that were interesting enough to share with you.

    #1: Bounces on a "non-bounceable" Page. How Come?

    #2: Bounce Rate in Omniture. Why Not?

    #3: Decent Funnel Abandonment Rate. How Much?

    #4: Rock Steady New vs. Returning Visitors Ratio. Why?

    #5: Sudden Data Drop! Help Me!?!

The questions and answers are below, I hope you find them to be of value.

#1: Bounces on a "non-bounceable" Page. How Come?

How can an a page that is not accessible without coming from a previous page can have a "bounce rate"? It should only have an exit rate, isn't it? That page is not a landing page.

This is a good one, don't cha love riddles?

Many people wonder how their page deep in the site can have bounce rates. Or say their shopping cart page, that's usually not even indexed by the search engine!

Here are some reasons I have found for this problem in my experience:

It turns out that people will bookmark pages on your site during their visit and then visit during bookmarks and then bounce! The session already started so you get a "single page view session" and you have a bounce.

More than once the culprit was that people in the company had bookmarked pages and were visiting it as there were changes happening to see if the changes went live or looked good. They see one page and bounce (damn!).

Another time it was that someone had bookmarked it and sent it around on email, all those clickthrus showed up as "direct" and bounce. Ditto for this tweet crazy world (if they short url the wrong page!).

Of course other times you think the page is "unindexable" but not all robots will respect your robots.txt file instructions.

trampoline

Update: Long sessions can cause you to see Bounces on "non-bounceable" pages. Your visitor comes to your site. Browses. And, say, adds to cart. Goes to gossip for 30 mins. Comes back. Reloads the page. Leaves. Boom! Just caused a bounce. : ) Won't happen a lot, but certainly does happen.

And once, just once for me, a delightful person had set up a program (javascript executing program!) to go ping a deep page in the site to make sure the site was up! Of course that showed up as bounces, but by segmenting that page we found a nice clean pattern and then the culprit.

Those are some reasons in my humble experience. How about you? Have you run into this? What did you find?

BTW if you want to understand the reasons go to your page level analysis reports (in any tool) and look at the Entrance Sources & Keywords, Navigation Summary for clues. Or send a email around your company and ask if someone's been visiting it or had included it in a irrelevant email blast to customers.

#2: Bounce Rate in Omniture. Why Not?

I can't find Bounce Rate in Site Catalyst, can you please tell me how to create the reports you had on your blog / presentations?

It is not even funny how often I get this question.

Omniture does not have the metric Bounce Rate in the set of metrics that it provides by default in the application. Perhaps in the next release it will make the cut.

omnitureThough as you know Omniture can compute anything you want, so you can absolutely create your own custom metric and apply it all the many many reports available to you in Site Catalyst. I know it can be done because I have done it myself.

But I am not an Omniture "expert" so I'll request my peer blogger Adam Greco to write a post on the official Omniture blog that covers two things:

1) How to create the bounce rate metric in Site Catalyst.

2) How to create two reports I constantly present in this context: Top Landing Pages (Visits & Bounce Rates for each). Keywords – Paid & Organic splits (Visits, Bounce Rates, Conversions).

I'll link to Adam's post when he writes it, and then I'll have a link handy to send to all of you ask me how to do this (so often!).

[If you are a Omniture Customer you should regularly be reading Adam's Inside SiteCatalyst blog, it is very good.]

Update: Adam has very kindly responded to my request, here's his post: Bounce Rates [Inside Omniture SiteCatalyst]. Thanks Adam!

#3: Decent Funnel Abandonment Rate. How Much?

What is the best way to tell when you are performing well with your funnel reports?

I understand that funnel reports can be used to find leaks. Obviously you will never get 100%, but how can you justify when your pages are optimized good enough?

No complaints? There has to be a better quantifiable way.

I love this question because we as Analysts (/Marketers) tend to obsess about things and don't often think of the law of diminishing returns.

It is important to know that how low can you go often depends in the type of business, and it also depends on the type of funnel (for some you want to get to 5% if you possibly can).

If you are doing this in a dedicated way then I encourage you to plot the funnel abandonment over time.

You'll start to see that initially, as you move from 80% abandonment to 60% you can probably do that quickly. Low cost.

It will be a bit hard to move to 40%.

law of diminishing returns

Then you'll start to note that with time (and effort) your graph is flattening out, say at 30%.

That's probably your point of diminishing returns. At this point you can do a cost benefit analysis and see if its worth it.

If you want to do more then you should "segment like hell" (different traffic types, offers, campaigns what not) and try to find improvements you can make for specific sources (for us it was affiliate traffic!) and move away from the aggregate.

Once you apply this type of thinking you can optimize for your own internal maxima and move on to other bigger things (remember there is no shortage of good work).

#4: Rock Steady New vs. Returning Visitors Ratio. Why?

I'm stuck in one area. For our major media site we have a consistent ratio of:

New Visits: 25%
Return Visits: 75%

We want to grow, but also maintain a healthy growth cycle. Our overall visits, uniques and pageviews have been increasing by about 10% month over month, but we are still maintaining the same new vs. return ratio.

Do you have any insight on what a healthy new vs. return ratio is?

Another riddle wrapped in a enigma. : )

The healthy ratio for any site will depend on so many different factors. I have often stressed that what works for best buy does not work for circuit city (well at least not any more!).

Hence it is imprudent to think that there is optimal ratio to shoot for.

steady balance

But in the case of this media site my suspicion is that while the overall ratio stays the same (while page views are increasing) the ratio of different visitor segments might be shifting quite a bit.

So step #1 would be to look for the ratio of New vs Returning Visits for the search traffic (paid or organic) first, because for so many media sites search dominates acquisition strategy.

Then also look at the metrics for top referring urls, same idea.

Take the investigation a step further and look at the content consumption for different parts of the website. Do more of the New people read the Op Ed and for some reason all returning people read entertainment news?

Something like this…

content vs visit distribution1

[See this post for the story behind the above picture: Deliberate, Dig, Understand, Throw A Feast!]

I would really drill down in various sections of the web analytics reports to see if I can find segments that might be attracting and retaining new (or returning) visitors.

As we have frequently harped on here, looking at the Visitor Loyalty and Recency reports can be very actionable.

Net net at the end of the exercise:

You'll get a understanding of if the overall is telling the truth of there are pockets where the ratio is very different (and why?).

Looking at content consumption will help you understand what the patterns are for various types of content.

Fun!

#5: Sudden Data Drop! Help Me!?!

I have had a web analytics tool on my site for some time. I am curious because my numbers go down rather than up even though I get a reasonable amount of views to website.

The numbers went up to about 800 and then started to go down again. Now its at 450 ish.

Also countries that once showed having had views no longer do. Can you explain this to me please? Thank you.

I share this with you to impress upon you the enormity of the tasks that get trust on me.

As you can imagine there is no way that I can answer this question (and above is 100% of the information I have!).

But I did try. Here's my reply. . . .

There is not enough information in your email to explain what might be happening. My recommendation would be that you reach out to your web analytics vendor and share your website and other details and perhaps they can shed some light.

But when I look at things like this I start with the referring urls report for the current month and the one before to see if any referrers have unusual drops in traffic, ditto for search engines and keywords.

heading in the wrong direction

If any campaigns running before have expired now – that can also cause your traffic to drop by lots very fast.

Finally if the javascript tags have disappeared off the site or are missing from any newly launched pages that could also be an issue. Check with your web master / tech gods if any major launches have happened (or they got rid of your tag because the footer is gone).

See I tried.

What would your advice be to this kind soul?

I would very much love to have your feedback on all five issues. What did you think of my guidance? Do you have a different answer?

Ok here's a challenge: What's the toughest web metrics / data / analytics question you have ever gotten?

If I get a bunch maybe I'll pitch and see if I can answer some. If. : )

Thanks so much.

PS:
Couple other related posts you might find interesting:

Next Page »