complex beautifulWhat would make you cry of happiness in a Web Analytics report?

What would make you cry of happiness in any report / presentation that you got from a analytics practitioner or consultant or your mom?

This post attempts to sort through the good, the bad and the ugly and answer that question (except that Mom bit, that will require therapy!).

It will also help you win contracts, prizes, company bonuses, and generally give you Superwoman (/man) like powers to impress people with your awesomeness in presenting complex insights that simply drive actionability.

Some context first.

grand prizeI had the honor of helping judge the winners for the WAA Championship (and the SEM Scholarship Contest ). That made me think a lot about what makes great analysis.

When looking at so many wonderful entries, how does one decide the winner? Are there specific traits? How do you know who deserves to sit at the kids dinner table and not the adult one?

Thus this post was born. It was my attempt, before I judged the contest, to create a framework that would help me identify real analysis and separate the Squirrels from the Ninjas.

Daniel said it would make a great post, and so this one's for him.

[UPDATE: Thanks to permission from the WAA I was able to add the top four winners to this post. Please see links at the bottom of the post, the contain great learnings.]

My hope is that it will help you identify what makes for magnificent analysis and in your day to day job (as Marketers, Usability Professionals, Consultants, Analysis Ninjas, Reporting Squirrels, …) present your thoughts on a set of data and have the maximum impact in terms of insights and action.

Seven Filters That Help Identify Great Analysis:

After you are done with any analysis, and before you present it to your client / peers, apply these 7 filters to ensure that what you are sending out is real gold. . . . .

1) No data pukes.

A summary of the data from the tool is not enough (no matter how pretty). Period.

Often "analysis" that is submitted is essentially a small table of data, which is essentially a "mini me" of the large table from which it came. This is not analysis, it is just a smaller report.

no data pukes pleaseHere is another thing that people consider analysis: x,xxx visits to the championship page almost a xx% of the x,xxx visits for the period of 06/01 to 06/14.

That's the "table" in English. It has the additional disadvantage of forcing me to do math two or three times and try to even graph it in my head. Too much work for anyone to do from a "analysis".

This might be a bit harsh but as I read any "analysis" here is what's going on in my mind:

A] "What's your point?" Give me value, not data.
B] Based on your point, "what do you want me to do?"
C] If relevant, and usually only if asked, give me the data (and please please please don't make me think or have to compute 19% of 8,296 Visits!).

Remember its the first one that I want the most. A drives action.

Bonus points: If you did a good job with the graph, you should not have to repeat in English underneath the graph what it is showing.

If there are any data rivers in your data, please consider redoing your analysis.

2) Hard tie to business outcomes. Always.

If you have learned anything on this blog then it is probably my insane obsession with Outcomes (see Trinity, Web Analytics 2.0).

At the end of the day every analysis needs to solve for the business outcomes. So you have to have some understanding of the goals going in (this is much harder than you imagine).

Many people just jump into the data, find interesting trends and patterns, convert those into "insights" and off it goes. The problem? You are sending things out you think matter, rather than what the business actually cares about.

business outcomes

Revenue, leads, increased customer satisfaction, brand value, friend invites, loyalty, bounces, website engagement :^), job applications, ads clicked, task completion rate and …. and … and .. You get the idea.

Awesome analyses always have hard tie to outcomes.

It was interesting that for the WAA championship there were no real business outcomes provided to the Analysts. Just loose guidance: give us stuff. :)

Yet that did not stop the superstars of web analytics. They simply assumed what the site's desired business outcomes were.

Many of them opened with their interpretation of the three goals of the website (or the bold ones even said something like: "you should actually be driving xxx outcomes but you the WAA are focused on silly things" – now that is chutzpah I can admire!).

Is your analysis focused on clearly established business outcomes? If not by your boss / client, then by you?

3) Usage of other tools (True Analytics 2.0).

Another little thing I obsess about, trying to always advocate the use of more than one source of data to ensure people understand more than just the "what". . . .

web analytics 20 demystified1

I think it was the fifth analysis that I saw that used something other that he web analytics tool the Analysts were provided (Google Analytics).

Such a shame.

Google Analytics (or for that matter Omniture or WebTrends etc) are great tools. You need the What. But it is so limiting. You need the Why and the What Else and more. True Web Analytics 2.0 to get robust answers that have deeper customer and competitive insights.

Sure the WAA does not do surveys to understand how to serve their customers better, and does not have "direct" competitors and WA 2.0 is hard work.

But the enterprising Analysts (and the winning team) went out of the box. They multiple hatsused the AdWords Keyword Tool, they did searches on other search engines, investigated Web Analyst's search behavior on Google Trends, checked how many corporate WAA members members link to the WAA site (a measly one!), one of them did their own survey directly to members (!!), checked the DMOZ, compared the site to IAB etc.

Now that is almost orgasmic. They did not take the lame excuse that the client did not give them data sources. They used all the tools at their disposal and executed a 2.0 analysis.

Do you?

Bonus: Remember you don't have despair about what the client as. Use free services like Compete and mine Google for press releases by various organizations (like that contain relevant info and more.

4) Not boring. Please.

Ok this blog is a exception (!), but let's admit it: Web analytics is boring.

Analytics of all sorts is boring. To lots of people (not you and I of course!).

Most web analyst report outs, glossing consultant analysis, make you want dramatically hurt yourself, rather than read them. They are all the same, data pukes, pretty graphs that tell you nothing, no tie to outcomes and descriptions and summaries that would make the IRS proud.

Let that not be you.

Look that these nice folks, they read all the analysis (check out the thick stack!), and just look at how excited they are!

excited old couple

I have to read lots of reports and summaries and briefs. I am always looking for people who made it interesting to read their submissions. Do they have a interesting way of framing the analysis?

In the real world this quality stands over all other "consultant" / "analyst" reports.

The analysis of the winner of the WAA Championship was essentially a series of email exchanges between people, with each email they revealed their ideas, insights and methodologies. No graphs. No tables. Just a quirky sense of humor (and deeply delightful analysis).

They stood out from the polished nicely templated graphics rich submissions of everyone else.

They made web analytics unboring. They made it fun.

Sexy wins. :)

5) Connect insights with actual data.

This might sound absolutely surprising but in many of the analysis it is really hard to see what the connection is between data and the insights derived from that data.

It seems along the way we have all developed "best practices" and preferences and "what works" and what does not and so on and so forth. Hence as we look at websites and data we sometimes simply jump to making recommendations based on what we know and think and feel rather than staying grounded in data.

connectOften I read something like: "Redesign the navigation", and my first thought is why? based on what?. Or "Internal search should be every where" – why? surely a best practice, but why for this site?

Lots and lots of people did this in the WAA Championship, especially those that were from decent sized agencies or consulting companies. They have the curse of knowing lots.

Me? I always put that secondary. My recommendation: Tie your recommendations to the data on hand. Include your feelings in a appendix, but in the main body, tie to data.

6) Meet the "expectations of scale".

This is perhaps a personal bias (especially in competitions). I am not going to, sorry, have the same set of expectations from Michelle Chin as I do from Jaume Clotet as I do from Zaaz.

tall and shortEach of those comes with massively different set of experiences and resources. The bigger you are the more I expect (and please remember not more data pukes, more analysis!).

More in terms of insights, more in terms of rigor, more in terms of everything.

If you are "big" or you have written a book (!!) then you are playing the game at a different level when it comes to expectations. Michelle has to be just so good to beat the bejesus out of you (and I know Michelle, she can!).

Look at your size. Do your analysis reflect the depth that your size should? It better.

7) Have something unique. Enough said.

Remember that if you are going for a RFP or a contract that 99% of what you will have access will be the same, 70% of the analysis that you will end up doing will be the same as your competitors, you might have read the same books and attended the same conferences.

Do something that makes you stand out.

And I'll let you into a secret, it is not the formatting of the text you deliver or 3d charts. That has been done to death.

And its not that hard.

Here's a example, everyone will report that a metric (say conversion) was 53% for keyword z and it was 56% for keyword q. Why don't you compute statistical significance between the two? Rather than reporting those two numbers out you can show how much confidence there can be in those numbers.

See how easy it was to stand out?

unique 2

Or here's another one. One of the Analyst started by stating that they were leaving out a time period that could distort the data. Interesting that they thought of that.

You could likewise eliminate from your analysis sources that reflect "one time only not repeatable events". Why bother?

Or try this, measure offline impact of the online activity! It is hard to do and you'll stand out!!

Business life can be a contact sport (competitors certainly are) and if you want to win then you have to have a UVP – a unique value proposition.

Never let a analysis leave your computer without making sure that there is something unique in it that will stand out.

See that was not hard?

Here is a summary of the "Avinash Filters for Awesome Analysis Presentations":

1) No data pukes.

2) Hard tie to business outcomes. Always.

3) Usage of other tools (True Analytics 2.0).

4) Not boring. Please.

5) Connect insights with actual data.

6) Meet the "expectations of scale".

7) Have something unique. Enough said.

Do you agree with the list? Have something to add? Would you like to "puke" :) on something, or simply disagree? Please share.

From your experience are there techniques in presenting analysis (or conducting them) that have worked particularly well for you? It would be awesome to have your insights and lessons. Thank you.

Happy Birthday to Nelson Mandela! An icon, an amazing human being, a true leader. Here is a great four minute audio and photo tribute, please check it out: Nelson Mandela at 90.

PPS (Bonus!):
Due to popular demand, and thanks to the WAA's permission, here are the top four winning entries from the WAA championships. This is a great way for you to learn more about how to present great analysis (and they each took a different tact).

[It is quite gratifying to me at some level that three of the top four are international entries. Validates for me the superior analytical sophistication that is outside the US.]

Social Bookmarks:

  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite