September 2008

30 Sep 2008 12:13 pm

petalsI get a lot of emails with questions, atleast 10 to 15 each day. Some are easy, others hard, and some mind boggling (due to their length, complexity or audacity!).

"Dear Avinash" is an occasional series where I share some of my answers that might benefit the greater ecosystem. I'll only share the questions that might be universal, and ones where the source would be impossible to identify (to preserve confidentiality).

The first one covered career advice for Stressed "Agency Analysts" and Search Robots messing up your life.

In this post we'll cover a topic near and dear to all our hearts, comparing trends of our Key Performance Indicators and two specific strategies you can use to drive action. One that focuses on presenting data, the second an approaching to analyzing trends.

Let's go. . . the question first. . . . .

We have an ongoing debate and I was wondering if you can shed some light on the issue. We make some reports that track month over month change and others with year over year change.

One argument is that if there is a holiday in a given month or new products come out in that month then month over month will give a distorted picture. (For many companies the same product types usually come out in the same time period each year.) So year over year is what we should look at

On the other hand there should (and usually is) always an increase in year over year. And year over year is less actionable. If I want to know what product/page types need more attention, last year’s types are long gone.

I have tried to find some metric that mixes Y/Y and M/M, but to no avail.

Do you have any thoughts on this matter?

I have often advised that the cheapest way to give context to your performance is to use comparisons to other time periods.

Here's me comparing performance of this blog over two years for the Visitors metric:

visitors trend yoy comparison 1

Overall happiness reigns, I think. [The blue line is '07 and red '08.]

And that in some sense is the catch.

The best that a comparison of historical trends can do is give you some initial context (yea or nay) and the next best thing is that it is a good way to raise initial set of questions. "Hmmm what happened over there?" "Why don't the peaks line up?" [Digg effect]

when all you have is a hammer But, as is pointed out in the email question, it is the context around changes that makes things more valuable. The graph itself won't answer the questions "why is that up or down" or, perhaps more importantly, "is our current performance better than last year's".

Product sales for you might peak each Thanksgiving (in the US). But if Thanksgiving this year was $15 mil revenue and last year was $10 mil then is that good? More importantly, is that good enough?

This is where tribal knowledge comes into play. What is different about this year and last (or this month or last)? Have you doubled the team? You have free shipping this year? This year you spend a lot on AdWords? Or you just hired me to do consulting for you this year at $10 per hour? Etc etc.

Because of that it would be nearly impossible to come up with a perfect historical trend that will be "clean". So the first thing is to realize that and then not expect too much :), except that comparing trends is a good thing and that its purpose is to just raise questions.

Next I have two suggestions:

# 1: Collect the tribal knowledge and annotate the graph.

Rather than the graph above, I'll present this one (in this case to myself! :) . . . .

visitors trend yoy comparison annotated

When I send out the above graph or present it in the meeting everyone will say "ah ha, we can discount that peak, oh we did not do as well at that point, and we need to do more of this thing over here". I.E. important actionable conversation.

So talk to your Marketers, Boss, Cleaning Lady, the dudette you replaced at half her salary! Get the tribal knowledge, paste it in.

One of these days my hope is that Web Analytics vendors will A] Make it easier for us to add the annotations and/or B] Mine other sources and automatically add context / tribal knowledge as Google Trends does today.

google trends omniture webtrends

[And one of these days the term "avinash kaushik" will have enough search volume to show up on the top graph! Miraculously it does show up in the 2nd graph above - though it is quite likely that my friends at google are just drawing a "pity line" for me! :)]

But if you are presenting in excel (or powerpoint) then consider annotating your data as you go along. That will be fantastic at providing some immediate answers.

# 2: Segmentation to the rescue!

In aggregate trends can hide insights and hence "dirty" the data. If you want to compare "clean" trends then your best option is to compare different segments within your data. [In all scenarios segmentation rules!]

For example you could just look at Organic traffic trends. Or performance of email campaigns. Or everyone who comes to your site from Florida. Or number of people who see more than five pages. Or % of Direct (free!) traffic. Or…. You catch my drift.

The benefit of comparing segmented trends is that you are able to go from trying to figure out which of the 1,800 variables is causing a impact to having to investigate just a couple of variables. This means you'll understand cause and effect (what you did and what was the outcome) much faster.

Here is an example. This graph shows, for the same time period as above, the Visitors from Organic Search for the keyword "avinash kaushik". . . .

avinash kaushik search trend

Now the cause and effect can be understood much faster and actionable insights delivered intelligently.

Let's say I did lots of things to drive SEO from Nov 2007 to May 2008. Well clearly it worked, I wrote a lot less content yet my traffic increased very nicely (even better YOY).

So I could summarize that paying Matt Cutts $1.6 million to help me with search engine optimization of my blog's URL stems was a genius investment! [QQ for Matt: Why is my Page Rank still 4? Tears. :]

Do the same for your business. No no no, not hire Matt, segment your trends when you compare them.

One of my other favorites was segmenting out Direct Traffic. That is so very cool because Direct traffic (non campaign, non search, etc) was usually less influenced by other things (acquisition related thing you do – places you spend money) and hence served as a great barometer for over health of the site.

occams razor direct traffic trends

Are you getting better at getting free traffic? Do people remember your site and just show up? Do you have enough engaging content (in case of pure content non-ecommerce sites) that people return again and again each month? Etc etc.

To me that is a "clean" segment / trend to look at. For your websites there will be others. Dig and poke.

Ok its your turn now.

What are your tips when it comes to comparing trends? What things fail spectacularly? What works really well? Do you dread these things or love 'em? Care to share your war stories? What is right about the approach above? What's wrong? We would love to have your thoughts.


Couple other related posts you might find interesting:

23 Sep 2008 12:42 am

focus"Most Analysts / Marketers / HiPPO's don't realize the amount of pain and effort that is involved in getting the web analytics data they are seeking. We need to teach them what it takes to get the 'ultimate golden everything' so that they'll make smarter decisions about what to ask for and what to skip."

That's my good friend John Marshall starting yet another 'lets go out and change the world' conversation with me. : ) He is also convincing and I quickly agreed. The result is that you all get these delightfully informative videos!

Two things connect John and I (we are the co-founders of Market Motive):

1] A passion for Analytics, web or otherwise.

2] A love of teaching, with a particular stress on cutting through the FUD and stressing actionability.

We hope to bring those two things together in this blog post through three videos. They share our perspective on how you should go into a "I want everything from my web analytics tool" session.

The videos recommend an evolutionary process for your "meet all my desires now Omniture (!)" sessions. [Of course replace Omniture with your favorite vendor.]

They also share what specific things you need to be aware of in order to ensure maximum success. I am sure your boss (or is it you?) wants it all, x, y and zebra metrics or q, r and parrot reports. Here you'll understand what you can get out of the box, what might need a bit more work and what might need selling your soul!

And I will fail my fiduciary duty if I did not let you know that in some cases we are going to recommend you walk away. The cost of what you want is simply not worth the data you are asking for.

Let's go. . . .

My answer to John's challenge was to sketch this out on the white board (in all of five minutes – comes from being close to the pain for so many years!):

web analytics the journey to nirvana

Clearly I am not winning any awards for my calligraphy. At the end of this blog post is a PowerPoint version of this sketch (thanks to Tyler Link!).

I called my picture (/piece of art) "The painful path to web analytics nirvana", to which John remarked "let's call it stairway to heaven". Much better I think.

If you want join the contest and come up with a better title then please add it in the comments section.

Act One: Basics, Self Service, The Start of Pain. (17 Mins.)

In this video we cover three steps of the ladder.

Stairway to Nirvana Step 1: "Tag baby tag!"

What you'll learn:

    If you implement the javascript tag what do you get?
    What don't you get?
    What's www . hotbrits . com? : )

Stairway to Nirvana Step 2: "Update tool side settings"

What you'll learn:

    Which simple things you can update without needing your IT resources?
    Ideas for goals you could set.

Stairway to Nirvana Step 3: "Campaign tracking / ROI of acquisition efforts"

What you'll learn:

    What does it mean to do online campaign tracking?
    Why will you cry for your mommy?
    What can you do to be successful?

Got your learning hat on? Hit play. . . .

Was that not fun?

Did you realize you could do so much of Web Analytics without really aiming for a 18 month installation of Omniture and WebTrends and any other tool you choose?

That is the amazing part, and the reason for these videos. If you add a javascript tag to your site, update a few settings associated with Internal Site Search and Goals etc then you are off to the races.

Yes the tool won't help you into your underwear quite yet.

But in the name of all that's holy you can start taking action and improve big things (even as world domination and crushing of all competitors by your company will have to wait a little while!!).

web analytics nirvana whats the cost of the benefit

In step three hopefully you learned why the Web Analytics Trifecta (People + Systems + Process) is so important – and oh so painful to pull off.

Now you can make plans accordingly.

[PS: Here are 30 points of progress you can make with just three hours of your time:
A Primer on Web Analytics for Everyone - you'll be surprised how much you can do.]

Act Two: Money, Money, Money! (11 Mins.)

In this video we try and kick things up a notch, focusing on things that often many of want to track (and should track deeply like Revenue) and others that we feel like tracking (but we might not realize the associated cost).

Stairway to Nirvana Step 4: "Revenue & Über Intelligence"

What you'll learn:

    Where does the standard javascript tag come up short?
    URL, Title, Cookies, what if the data you need does not exist there?
    Who do you have to beg for help?

As Chef Emeril would say: Bam! Bam! Bam!

See there is still hope in the world. If Step 4 has less pain than Step 3 then there has to be something terribly right with this universe. Of course while we talk about pain in Act One, we actually do that scoring in this video. Sorry.

Again here the stress is on the importance of planning and accounting for the Systems and the Process parts of the game. And did you know about the legal stuff?

Also did you notice that we are really having fun? Perhaps as only two friends on a common mission can. :)

Yes, we do need to buy some lights to ensure you can see the sparkle in my eyes!

Act Three: The Price (& Pain) of Greatness! (11 Mins.)

I, humbly, believe that the web became "fluid" a while back, we have just not realized it yet. Well most of us who have grown up in the log file parsing, page viewing, "my site clickstream data is all I need to care about" crowd.

Life is now about RSS and Widgets and Flex and Flash and AJAX and Video and Web Applications (and all of them mashed together) all of which make the web experience a lot more fun, a lot more fluid. And a lot more nightmarish to track.

Stairway to Nirvana Step 5: "Rich Media"

What you'll learn:

    Why page views are truly "to die for"?
    Did you realize "geeks" are your BFF's here? [John has a definition for you!]
    What's the most fundamental difference between the "old world" (javascript based data collection) and the "new world" (fluid web)?
    How to make good choices on your path to Nirvana?

Intrigued? Sure you are. . . .

Get a feel for your new life, you are going to have to get out of your cubicle and be something of a social butterfly and get to know your Marketers and Developers and Business Folks and others.

It is your relationships and your pre-planning and huge investment in time and love that is going to ensure that you succeed in this step.

In the video we refer to Event Logging as a scalable non-fake-page views data capture methodology, learn more here: Google Analytics Event Tracking Guide. [A feature that is in wide invitation release, with access available to anyone who asks. :)]

I request other vendors (Unica, Omniture, IndexTools, WebTrends etc) to please add links to their guide in comments.

Nirvana Summary:

As promised here is a pretty version of the complete picture of what was on the white board. . . . .

web analytics the journey to nirvana picture

One step at a time.

Don't overestimate your ability to get things done.

No need for a 18 month web analytics installation, before the tool bears fruit.

Make wise choices, evolve intelligently.

Good Luck!!

Want more such educational videos on Analytics, PR, SEO, PPC, Social Media etc? Check out Market Motive:

Ok now it's your turn.

What has your experience being in extracting value from Web Analytics? Is implement, pause, extract value, evolve, pause, extract value . . . . and strategy? Are there things we completely missed? What has worked for you? What has not?

Please share your feedback, ideas and critique.


Couple other related posts you might find interesting:

Next Page »