April 2008

30 Apr 2008 01:05 am

old newKnow the difference between a Reporting Squirrel and a Analysis Ninja?

One is in the business of providing data.

One is in the business of providing, to use a old fashioned word, information.

This one of the core reasons why most dashboards are "crappy", i.e. they are data pukes that provide little in terms of context and even less in terms of actionable value.

Here are some examples of sub optimal dashboards, sub optimal in my mind from a actionable perspective. . . .

sub optimal dashboard 2

Perhaps the most common type is above. Lots of data, even drill downs included, but you can't look at it and go: "Wow we need to do . . . ". No sirrie bob you can't.

sub optimal dashboard 1

I wanted to point the above out purely because of a common feature of 80% of Web Analytics Dashboards, in excel with a billion tabs to look through. This is not a dashboard, it is the result of a massive sum of money paid to a Consultant who is trying to impress you with his / her excel skills – without actually telling you anything.

sub optimal dashboard 3

You are walking down the street. You look at someone from behind and you think "hmmm she's / he's pretty". So you speed up and overtake them and in the process you sneak a glance at them (yes you are married but looking is still ok :), and you are hugely disappointed. Not pretty. That's the dashboard above. Very sexy and Web 2.0'fied and a ton of data there, but a lot less actionable than you might have hoped.

Why is this so? All the above efforts are well intentioned, took lots of honest work and probably took months to put together. So why?

Here are some hidden (corrosive) reasons why most dashboards tend to stink when it comes to helping the Executive make any decisions:

  1. They leave the interpretation to the Executive (/ customer / requestor / other Squirrels). This is a fatal flaw because most dashboards are highly aggregated views of any KPI and are missing all the nuance and analysis (that only you as Ms. Ninja have, and you don't go with dashboard).
  2. Most Executives actually want insights / action recommendations but they don't trust the Squirrels / Ninjas / VP's / Data Providers. So they ask for numbers. We dutifully cram as many of them on to a A4 size paper in 3 size font and send it along with a magnifying glass.
  3. Most Squirrels / Ninjas live in a silo. Going out to collect enough tribal knowledge to actually know what is going on to then make recommendations from the data is not something that we do, nor are we encouraged by our Executives or our organization structures. This incentivizes data pukeing.
  4. Often dashboard creators tend to be "outsiders" (Consultants, Experts etc) and they often don't have deep practitioner experience that would allow them to understand the human / "below the surface" issues like the above three. That leads non-Practitioners to make the common mistakes like creating the above three dashboards.

If you want your Executives / Customers to take action, you have to give them information and not data. It takes effort to get there, it will take all your charms (though no violation of any HR intimacy policies), and it will take some time.

Step one as always is to become aware of the above three problems.

Step two is to get a possible solution from the Occam's Razor blog. :)

My attempt at solving this problem was to try and attack it from a human psychology perspective: How can I create a "dashboard" that will incent the right behavior from the Squirrels / Ninjas while giving Executives the information they need to make decisions (rather than engaging in a bitchfest which is the typical outcome).

Recommendation #1 was to move to a Critical Few philosophy for Executive reporting: Only report the three or five (at most!) metrics that define success for the whole business. Kill all the ancillary metrics that were nice to know (and my kill I mean let lower levels worry about it).

Recommendation #2 was my humble, admittedly ugly, attempt at a "Action Dashboard":

executive management dashboard

4Q. (Sorry Jonathan! :)

Each quadrant representing a solution to a human problem that lead to crappy dashboards.
(Apologies for having to redact some of the data above, to protect the innocent.)

Let me walk you through it.

First very up top a clear identification of what the Critical Few metric was, who was responsible for that metric from a business perspective (translate into "head on the line") and who was responsible for the analysis.

Also note the little red dot. That here indicated trouble. It can have two other colors, yellow for don't fire anyone yet but get ready and green for send someone a big hug and a box of chocolates. Next. . . .

kpi trend

The first quadrant (the graphic) shows the trend for the metric. Ideally segmented (as is the case here, cart abandonment is illustrated for four key customer segments).

This quadrant is to satiate Executive curiosity that you know what you are doing, it will be glossed over (and that's good!).

insights from analysis 1

The second quadrant (Key Trends & Insights) is to add value by interpreting the trends and adding context. It says there that some things are up or down (in english :), and it also warns which data might be bad etc. You are starting to do your job here.

This quadrant is the one that Executives will read a lot initially, over time they will gain confidence in you, they will love that you share context (hello Ninja!), over time they will gloss over it (a good thing).


The third quadrant, clockwise, (Actions / Steps To Take) is force the shy Web Analyst to get out and talk to Marketers, Website Owners, VP's, Whomever it takes to get all the tribal knowledge, identify root cause for the trends in the metric and recommend solid action to take. The Analyst will rarely be able to do this by themselves. It will require human contact with others, it will require conversations, it will mean identifying solutions collaboratively. It is a fantastic opportunity to become smart about the business.

This quadrant is key to driving action. No longer do you leave things to interpretation or let's blame people etc. You are recommending what actually needs to get done. Your Executives will kiss you and over time this is the only quadrant they'll read. It will also mean that monthly meetings will move from bitch fests to deciding who does what. Amen!

impact crater barringer arizona

The fourth quadrant, (Impact on Company/Customer) exists in case it is not clear to the Executives why they need to take action (listen to poor old you the lowly Analyst). I feel it is the key thing missing from any dashboard, they are normally missing the kick in the rear end and this quadrant delivers it. It is the answer to this question: "As a result of this trend (up or down) what was the impact on the company and its customers". It also forces you, Marketer / Analyst, to do hard work to estimate the impact and put it on paper.

This is the killer quadrant, if nothing else drives action this will, knowing exactly how much money was lost, how many customers were pissed, how much opportunity was wasted. Now when they ignore you they do that at their own peril and with their butt on the line. Trust me action you recommend will be taken.

See how simple it is?

You fix the human problems, you address the flaws in the system today and you actually become much smarter about the whole business (thanks to q3 and q4).

Win – win – win.

Over time you'll gain a lot more trust from your Executives and all the crappy dashboards can die and be replaced with one that looks like this one. . . .

executive management dashboard nirvana

Now you are asking your Executives to simply layer their own judgment on the recommendations and help the company take action. Who needs to see the numbers? They pay you and I to deliver actionable insights.

I stress that it won't happen overnight, but shoot for that nirvana state.

May the force be with you.

Ok now your turn. Care to share your own learnings and battle scars? Your success stories? Perhaps critique my "Action Dashboard" (sorry could not think of a better name, do you have suggestions?). Your perspectives are most welcome and would be greatly appreciated. Thank you.

Couple other related posts you might find interesting:

21 Apr 2008 01:04 am

new Here is a dirty little secret:

Unless you are a exceptionally content heavy website And your content changes every day, it is quite unlikely that your Top 25 of anything changes all that much.

So why do we insist on stuffing our dashboards with top x or this and top y of that and top z of whatever? Let me explain with an example.

Here are the keywords driving traffic for one month:

top search keywords month one 1

And here are the keywords driving traffic to the website in the second month:

top search keywords month two

Notice something amazing? While traffic is very different in those two months the actual keywords that sent traffic from search engines barely changed. In fact even the order in which the keywords sent traffic is astonishingly similar!

I bet most of you (ok atleast you Analysts) are not actually surprised at this because you get dashboards, reports, data pukes and it all seems to be the same all of the time.

The reason for this is also not very surprising. Most websites are in stable businesses, they tend to do the same things when it comes to their acquisition strategy and there is a heavy concentration of the few things visitors want from them.

Yes there are some things happening "under the water" but they can be hard to find (take time and effort!) and our web analytics tools don't make it easy for us to identify those elements.

This leads to a few very sad outcomes:

1) We, Analysts, dutifully keep shoveling data over the fence (and that in our hearts of heart does not make us happy).

2) The Marketers and Decision Makers are frustrated because they think web analytics means stuffing their Inboxes with things they don't want to read.

3) Takes a really long time to take any action because the core things that matter are not apparent. Unhappy bosses.

So what's the solution?

whats changed report

Focusing your analytical and reporting efforts on "What's Changed".

ClickTracks, mentioned often here and in my book :), is a great web analytics tool and one of my all time favorite reports is the What's Changed Report.

It is quite brilliant in its execution.

You can choose any two time periods and ClickTracks will churn through all your website data, apply intelligent mathematical algorithms (statistical significance etc) and present a report to you that shows the most significant changes along all your data dimensions. It's Dr. Stephen Turner and his team at their best (IMHO :).

Awesome, because rather than looking at the top x of whatever you look at every day now you can look at just the data that matters.

Search key words worth paying attention to. . . .

Here is how the What's Changed report looks if I compare two time periods:

whats changed report search keywords sm 1

[Click on the image above for a higher resolution picture.]

Isn't that fantastic!

You can separate the wheat from the chaff by pressing just one button and now you know, instantly, which key words / key phrases need your attention because they have started sending a lot more traffic in the current time period. Maybe these need to go directly into your AdWords or AdCenter campaigns.

You also know which keyword are "falling", becoming less important in your data (perhaps you should drop 'em, perhaps you spent a lot of money to no effect, perhaps customers are moving on). You are SEO'ing your poor heart out, are your efforts working?

Both sides of the report very actionable because they uncover things "below the surface", things that web analytics tools are typically bad at highlighting.

Once you pick your time period you can get this view for all your data.

Referrers worth paying attention to. . . .

clicktracks whats changed report referrers sm

[Click on the image above for a higher resolution picture.]

Which campaigns are working, which are not. What websites are new surprises, on the left, and which ones are on their way to becoming disappointments, most likely on the right. Who should I start relationships with (say new affiliates) and which ones I need just let go. You get the idea.

See what I mean when I say it is absolutely actionable?

[Special Note: Please do not type in all the url's above and check 'em out. Some of them are from websites of an adult nature and might not be office worthy. The data is real for my blog, and no you can't ask me why they link to a web analytics blog. Your guesses are welcome though. :)]

Content consumption, pages, worth paying attention to. . . .

clicktracks whats changed report visited pages sm

[Click on the image above for a higher resolution picture.]

Again all of the above questions and techniques apply in making this valuable.

To me there were two surprises in the above picture.

On the "Rising" side one post, on Personas and Non-Line marketing, became so much more popular. It was written months and months ago. Suddenly it popped. I immediately rushed to check out Entrance Keywords and Entrance Sources and found the reason and I am taking action so that I can get other posts get that same bang!

On the falling side the surprise was that I did not realize in the last month three of those posts (besides the first) were that popular, all were written months and months ago. So in this case I am realizing what I should have known a month ago! If I were doing any sales or marketing or had key topical (important now) calls to action I would go add to these post.

Other reports / data. . . .

You can get this report along many other dimensions telling you what campaigns are working better or worse, what products are selling more or less (statistically) and other such great trends.

Very useful, very actionable, access to trends and insights that are normally hidden. Access to this kind of report helps you do the 90/10 that Tiger Woods is advocating. . . .

tiger woods focus on what to do next accenture

Today sadly with our Web Analytics tools we spend 90% of the time reporting on what you/company did and send masses of reports out, and 10% on what we / you / Marketers should do next. [My thanks to the Accenture corporation for creating such a great ad.]

What's Changed changes that.

Recently our good friends from Juice Analytics have taken a step to help the Users of Google Analytics create their own What's Changed report for Referrers. You'll find all the details at: http://is.gd/5Kw

You go to the site , spend nine seconds install GreaseMonkey in your firefox browser and then spend another nine seconds installing the plugin, two seconds restarting your browser and boom! You get this. . . .

google analytics juice analytics whats changed plugin

A new button in your Top Referrers report.

Now go ahead and press that button, it will change to a button called Loading…. and after a few seconds (please be patient) you get two additional reports on that page.

#1: Sites with more than 50% growth in traffic in last 3 days:

google analytics whats changed report referrers

Sweet! Now compare that to my Top Referrers report. . . .

google analytics top referrers report

You can easily see how the first report is more valuable than the default report, for one thing top referrers stays the same most of the time.

I can look at the first report and go identify who my real friends are :), why they are sending me traffic, what can I do to get them to my top referrers etc.

#2: Sites with more than 50% growth in traffic in last 3 days:

google analytics whats changed report new referrers

I won't repeat the message again, you can see how immediately valuable this type of information it (not the least to realize that my friends at Zaaz might not be telling the complete truth when they tell me they read my blog every day! Chris, Jason, Shane I don't believe you! :)).

Go run and install the plugin from Juice Analytics right now and see how quickly you can spot new things in your reports.

The "what's changed" functionality in Google Analytics is not quite as robust or pervasive or flexible as it is in ClickTracks. Hopefully someone from Juice Analytics or the Google Analytics team is reading this post :) and perhaps there will be more in the future. But for now I am thrilled that users of Google Analytics can atleast take a small step forward in understanding their data. For that I am grateful to the folks at Juice.

[Sal: I would love to have this "button" on the Search Keywords reports! Please.]

Update: 04/23: Sal kindly obliged and we have a very useful button now in the search keywords report as well to look at "What's Changed". Please see: Keyword Trends in Google Analytics. Here is now it looks (KW that sent 20% more traffic). . . .

google analytics whats changed report organic search keywords

Web Analytics is actionable, I hope that all the Vendors will take a pause from simply helping us Puke data out and consider implementing mathematical and statistical techniques that help us all become the Tiger Woods!

So here's my question to you: When you do reporting of your web analytics data are you on the What's Changed train?

Ok now your turn. Would you care to share some of your insights? Big or small? What has worked for you? Are there other techniques that you have tried that work for you? Any templates or pictures you could like to share with us? Please share your own strategies and success stories. Thank you.

Couple other related posts you might find interesting:

Next Page »