April 2010


19 Apr 2010 01:36 am

clusterIt is surprising how often these "simple" things come up.

"What is the difference between a metric and a key performance indicator (KPI)?"

"What is a dimension in analytics?"

"What is segmentation?"

"Are goals metrics?"

And many more.

There seems to be genuine confusion about the simplest, most foundational, parts of web metrics / analytics. So in this short post let's try and see if we can fix this really basic problem.

Definitions and standard perspectives on these terms will be covered in this post:

  1. Business Objectives
  2. Goals
  3. Metrics
  4. Key Performance Indicators
  5. Targets
  6. Dimensions
  7. Segments

A standard definition will be provided, but more than that my hope is to solidify your understanding with concrete examples and pictures.

The post will end with a "Web Analytics Measurement Framework" – a very lofty name for something that will help you put your understanding of this post into practice.

Business Objectives:

This is the answer to the question: "Why does your website exist?"

Or: "What are you hoping to accomplish for your business by being on the web?"

Or: "What are the three most important priorities for your site?"

Or other questions like that.

Without a clearly defined list of business objectives you are doomed, because if you don't know where you are going then any road will take you there.

The objectives must be DUMB: Doable. Understandable. Manageable. Beneficial.

progress objectives directions90% of the failures in web analytics, the reasons companies are data rich and information poor, is because they don't have DUMB objectives.

Or they have just one (DUMB) Macro Conversion defined and completely ignore the Micro Conversions and Economic Value.

Your company leadership (small business or fortune 100) will help you identify business objectives for your online existence. Beg, threaten, embarrass, sleep with someone, do what you have to get them defined.

Point of confusion: People, like me, often also use the term Desirable Outcomes to refer to business objectives. They are one and the same thing.

[Full disclosure: Depending on the specificity of your business objectives my asking you for your "desirable outcomes" could refer to "what are your goals". See below...]

Goals:

Goals are specific strategies you'll leverage to accomplish your business objectives.

Business objectives can be quite strategic and high level. Sell more stuff. Create happy customers. Improve marketing effectiveness.

Goals are the next level drill down.

It goes something like this. . .

Sell more stuff really means we have to:

    1. do x

    2. improve y

    3. reduce z

Improve marketing effectiveness might translate into these goals because currently they are our priorities:

    1. identify broken things in m

    2. figure out how to do n

    3. experiment with p type of campaigns

Get it?

The beauty of goals is that they reflect specific strategies. They are really DUMB. They are priorities. They are actually things almost everyone in the company will understand as soon as you say them.

I would not have included the step of identifying Goals were it not for the fact that almost every C level executive, every VP and SVP, give very high level nearly impossible to pin down business objectives.

Point of confusion: Many web analytics tools, like Google Analytics, have a feature that encourages you to measure Goals. Like so. . .

goal conversions in google analytics

It is possible that some Analytics Tool Goals directly measure your business objectives or goals. Usually though Analytics Tool Goals do not rise to the strategic importance so as to measure either your business objectives or your goals.

For example only one of the above, Subscribers, is an actual goal ("increase persistent reach")for me that lines up directly with a business objective ("effective permission marketing"). Others are nice to know.

So to be clear: Just because you have Goals in your analytics tool defined is not a sure sign that you know what your business objectives or goals are.

Before you touch the data make sure your business objectives (usually 3, or 5 max) are clearly identified and you have drilled down to really DUMB goals!

Metric:

A metric is a number.

That is the simplest way to think about it.

Technically a metric can be a Count (a total) or a Ratio (a division of one number by another).

Examples of metrics that are a Count is Visits or Pageviews.

Examples of a Ratio is Conversion Rate (a quantitative metric) or Task Completion Rate (a qualitative metric).

This is a crude way to think about it but. . . Metrics almost always appear in columns in a report / excel spreadsheet.

This is what metrics look like in your web analytics tool:

web analytics metrics

Metrics form the life blood of all the measurement we do. They are the reason we call the web the most accountable channel on the planet.

Key Performance Indicator:

Key performance indicators (KPI's) are metrics. But not normal metrics. They are our BFF's.

Here is the definition of a KPI that is on Page 37 of Web Analytics 2.0:

A key performance indicator (KPI) is a metric that helps you understand how you are doing against your objectives.

That last word – objectives – is critical to something being called a KPI, which is also why KPI's tend to be unique to each company.

I run www.bestbuy.com. My business objective is to sell lots of stuff. My web analytics KPI is: Average Order Size.

Business objective: Sell Stuff. KPI: Average Order Size.

I might use other metrics in my reports, say Visits or # of Videos Watched or whatever. But they won't be my KPI's.

Makes sense? No? Ok one more. . .

I run www.nytimes.com. My business objective is to make money. One of my KPI's is: Visitor Loyalty (number of visits to the site by the same person in a month) and another one is # of clicks on banner ads.

So one thing should be pretty clear to you by now. . . if you don't have business objectives (from your HiPPO's) clearly defined, you can't identify what your KPI's are.

No matter how metrics rich you are. You'll be information poor. Forever. So. Don't be.

Business Objectives -> Goals -> KPI's -> Metrics -> Magic.

Targets:

Targets are numerical values you have pre-determined as indicators success or failure.

It is rare, even with the best intentions, that you'll create targets for all the metrics you'll report on.

Yet it is critical that you create targets for each web analytics key performance indicator.

missed target

I am still at Best Buy. My KPI is still Average Order Value. But how do I know what's good or bad?

I'll consult with my finance team. I'll confab with my Assistant Senior Vice President for American Online Sales. I'll look over my historical performance.

Through this consultative process we'll create a 2010 AOV target of $95.

Now when I do analysis of my performance (not just in aggregated but segmented by geo and campaign and source and…) I'll know if our results are good or bad or ugly.

I will do this for every single KPI whose responsibility is thrust on em.

You can create targets for the quarter (Christmas!) or for the year or to any drill down level of specificity. But at least have one overall target for each KPI.

Business Objectives -> Goals -> KPIs -> Metrics -> Targets -> Minor Orgasms.

Dimension:

A dimension is, typically, an attribute of the Visitor to your website.

Here's a simplistic pictorial representation. . .

web analytics dimensions

The source that someone came from (referring urls, campaigns, countries etc) is a dimension in your web analytics data.

So is technical information like browsers or mobile phones or (god save you if you are still doing daily reports on) screen resolution or ISP used.

The activity a person performed such as the landing page name, the subsequent pages they saw, videos they played, searches they did on your website and the products they purchased are all dimensions.

Finally the day they visited, the days since their last visit (if returning visitor) the number of visits they made, the number of pages they saw are all dimensions as well. I know, I know, they sound like metrics. But they are, as the definition says up top, attributes of the visitor and their activity on your website.

This is a crude way to think about it but… Dimensions almost always appear in rows in a report / excel spreadsheet.

Here are the metrics and dimensions in one of my favorite Yahoo! Web Analytics reports, it shows me how many clicks it takes for visitors to get to content I consider valuable. . .

yahoo web analytics visits average clicks to a page

Columns and rows. Get it?

Let's solidify this with another example of a report that shows metrics and dimensions. This report might not come to your mind most easily. I am looking at the internal site searches (on this blog) and the continent from where the search is done and a set of metrics to judge performance. . .

google analytics multiple dimensions and metrics

Dimensions allow you to group your data into different buckets and they are most frequently used to slice and dice the web analytics data.

In your web analytics tools you'll bump into dimensions when you are either creating custom reports (love this!) or doing advanced segmentation (worship this!). The chooser thingys look like this. . .

web analytics tools dimension chooser

In Yahoo! Web Analytics they are called "Groups" or "Group Selection" but they are the same thing: Dimensions.

There are many long and complicated definitions of dimensions. There are some nuances that I have simplified. But I hope that this definition and explanation helps you internalize this key concept in web analytics.

Segments:

A segment contains a group of rows from one or more dimensions.

In aggregate almost all data is useless (like # of Visits). The best way to find insights is to segment the data using one or more dimensions (like # of Visits from: USA, UK, India as a % of All Visits).

You segment by dimensions and report by metrics.

Here are some examples of segments I use in my Google Analytics account:

analytics segments kaushik.net

Checkout the dimensions I am using to segment my website traffic to understand performance better.

  • Analyzing people just from North Carolina (because there was an ad campaign targeted just to NC)

  • People who spend more than one minute on the site

  • People who click on the link to go to Feedburner to sign up for my RSS feed

  • People who come from images.google.com and smart mobile phones

  • People who visit from one source, Wikipedia, AND only one page on Wikipedia (the bounce rate article)

These are just a few of the 28 advanced segments I have created in my analytics profile.

And I am not even a real business.

Think of how many segments I would analyze to truly analyze my Key Performance Indicators to understand causes of success or failure of my Business Objectives!

The Analysis Ninja rallying cry: Segment or Die!

: )

So now you know the seven most fundamental, yet critical, things you need to know about online analytics.

If you fee that you did not understand it all, please go back and re-read it. You are very welcome to ask questions or for clarification via comments. Whatever it takes, make sure you are able to internalize this.

Let's move to the last step. . .

Web Analytics Measurement Framework

As promised I want to wrap up this post with a couple of examples that pull this whole thing together.

Let's say I am responsible for the National Council of La Raza (a wonderful organization I support). Here is how the measurement framework could possibly look for me. . .

Business Objective:

Attendance at immigration rallies.

Goals:

Increase web sign ups.

Key Performance Indicators:

# of NCLR Sign-ups for NCLR Action Alerts

# of Individual Memberships

Target:

Action Alert: 14,000 per month

Memberships: 4,800 per month

Segments:

Acquisition: Organic search, Email campaigns, Mid-western US states

Behavior: Visits to conversions (Action Alerts, Memberships)

All this before I cracked open any web analytics tool.

I have a framework I can use to ensure that the work I do is focused on what's important to the organization, what good or bad looks like in terms of performance and finally I have a segmentation plan ready to do the preliminary analysis of the data.

No fishing expeditions. No data puking. No begging people to pay attention to data!

One more example.

If you are a student in the MarketMotive Master Certification course as a part of your final dissertation you have to submit complete analysis of two websites. One eCommerce and one non-eCommerce. You are supposed to start from scratch, do all of the above and present actionable recommendations. The path you follow, the quality of your analysis and your insights determine if you are awarded the certification, or not.

One of the web analytics students in the just concluded course was Matt Smedley.

In his dissertation Matt used the above framework very effectively to focus and structure his analysis.

Here is Matt's actual picture from his dissertation that tells the whole story:

matt smedly web analytics measuremet framework sm

[Click on the image for a higher resolution version.]

I really liked Matt's presentation for his motor bike company analysis. In less than half a page one could see the complete picture of what the business was solving for and what the expectations were.

Particularly clever I thought was his inclusion of the segmentation in his framework presentation. At a glance for the most important goal for the quarter (build a robust customer database for future marketing) you can see how their campaign strategy worked.

Don't even get me started on how awesome it was for him to including Profit as a KPI. Truly heart warming.

I hope you will find inspiration from Matt's presentation to go create a version of this framework for your company.

We worry so much about tags and data collection and Omniture vs. WebTrends. What we should actually worry about is above. It is not easy to arrive at, but without it all you have is unlimited potential for failure.

And I know that is not going to happen to you.

I wish you all the very best.

Ok now your turn.

What do you think of the seven fundamental terms and their definitions? Agree? Disagree? Which one surprised you the most? Was there a point you think I missed in explaining these complex concepts? Do you have a measurement framework you use in your company you want to share with us?

Please share your feedback via comments.

Thanks.

PS:
Couple other related posts you might find interesting:

05 Apr 2010 01:58 am

a promising startSome Marketers / Analysts use Click-thru Rate (CTR) to measure success of their acquisition campaigns. Nothing much to write home about, but certainly better than executing faith based initiatives.

A smaller percent of those Marketers / Web Analysts will move beyond clicks and measure Visits / Visitors and Bounce Rates to measure success. Lovely, warm hugs and smiles for them.

A fraction of those Marketers / Directors will calculate Conversion Rates for those marketing campaigns. They deserve our love. [And if they measure Micro Conversions they deserve our love AND respect for exhibiting savviness by using economic value.]

But all of the above is still focusing on short term success. Even measuring Visitor conversion rates (Visit based conversion rates promote bad marketing behavior) is akin to declaring success after a one night stand.

I reserve the best hugs, kisses, smiles, love, respect and my deepest admiration for Marketers and Analysts who use Lifetime Value computations!

That is focusing on real success, not simply the first conversion (the one night stand!).

That is focusing finding the customers that create value for the company, long term.

That is truly doing the kind of Analysis Ninja work that solves tomorrow's problems today!

For the above reasons I have been meaning to write a post on computing Lifetime Value for a very long time. But perhaps a better idea is to get an expert to do it, the result will clearly be far better than anything I would write. So I emailed my friend David. : )

David Hughes is the Co-Founder of the email marketing consultancy called The Email Academy and the author of one of my most beloved phrases: Non-line Marketing! His blog, Non-line Blogging, is a favourite of mine.

There are a handful of people in the world I could spend the whole day talking work and still have things left over to discuss, to learn. David is one of those super-smart, funny, and nice people. I have consistently found his ideas to be practical, grounded in common sense and instantly useful.

I could not be more thrilled that he agreed to cover this tough, yet rewarding, topic.

In this post David covers:

  • Why Life Time Value is important (especially in context of Acquisition)

  • How to optimally leverage value based segmentation & Lifetime Value

  • Share a sample analysis and, this is so sweeeet, a spreadsheet with a sample model that you can use to jump start your own LTV journey!

Buckle up, this is going to be fun and it just might change your life! :)

Here's David. . .

__________________________________________________

Solve tomorrow's problems today – introducing Life Time Value.

Acquiring new customers isn't getting any easier: We've picked off the low-hanging SEO fruit, we're paying more for quality clicks in AdWords and the going rate for affiliate deals just keeps getting higher.

We are also haunted by the specter of "marginal cost": The more customers you need, the more impressions and clicks you need. But as we drill deeper into worse performing media, or pay out for lower-volume-lower-relevance search terms, our cost per sale gradually rises.

There is a better way to analyze your acquisition strategy than simply using Conversion Rates or Cost Per Acquisition (CPA). Using Life Time Value might be a much better idea.

Life Time Value (LTV) will help us answer 3 fundamental questions:

1. Did you pay enough to acquire customers from each marketing channel?

2. Did you acquire the best kind of customers?

3. How much could you spend on keeping them sweet with email and social media?

I'm going to suggest that maybe you should be paying significantly more money for the right customers.

Let's start at the very beginning…

…that's a very good place to start. Take a snapshot of your customer database for the past 2 years and it may look like this:

average customer profile in numbers

That is an average.

The trouble with averages is they conceal all the really interesting stuff that's going on beneath the surface.

If you look beyond the averages you'll find that some of your clients are "better than average" and some are "worse than average".

Try and segment the customer base by total purchases over a longer time period, say a year, or total spend and you may come to a conclusion that says something like:

My most valuable customers last year bought 4 times compared to an average of 2. They tended to spend 40% more than average per order. However, they might cost significantly more to acquire.

Much better than the average right?

So, where did you get the valuable customers from?

Simply knowing that you are getting lots of conversions is not enough, you might just be getting new low value customers.

This is where Lifetime Value becomes interesting: Some companies are getting really worried about the lasting impact of "buying cheap customers".

For example, in many markets the price comparison intermediary (/engines) is an easy option – you pay your money (affiliate fees) and you take your customer.

But how likely are these customers to buy another product? Or hang around for a few years? With no brand affinity there's no desire to cross-buy and maybe we're filling up our databases with low value, promiscuous customers.

A simple segmentation by channel can easily help us answer these key questions. The output may tell the following story:

gross profit segmentation

But, I hear you cry, Search Marketing is labour-intensive, risky and costly compared to buying customers at a fixed price from an intermediary.

OK, so let's look at how much MORE we should be paying for Mr Right, rather than Mr Average.

Let's change the headings of the table above to be clear what we're talking about…

Best and Average customers will have different Year 1 buying patterns:

segmentation best and average customers

Once we have done this basic segmentation we can then factor in the cost of acquisition per segment to determine the Net Profit per customer per segment.

You'll work with your acquisition team or your finance team to get the cost data. For some of your campaigns this data might not be easily available in your web analytics tool (it is also quite likely you are doing all of this analysis in Excel).

The table you'll end up with might look like this:

acquisition cost net profit customer segments

It should be pretty obvious at this point that simply taking the short-term view with metrics like Cost Per Acquisition (CPA) might not be prudent since you are rewarding the source sending you Mr. Right and Mr. Average just the same. Yet they are not of the same value to your business (Net Profit!).

It is important to move away from a cost-based acquisition model to one that recognises the cross and up-sell rewards of acquiring the right customers over the duration they'll be our customers.

Spend an extra $8.00 per customer, if you have to, and you're still twice as well off than buying rubbish ones!

But we can do so much more… let's take a longer term view.

Value-based Segmentation & Life Time Value.

By now we have established this: Some of your customers are going to be spending more with you, for longer.

Let's say I am a car insurance company, or a subscription publisher, with a desire to sort out some of tomorrow's problems today.

I know that the initial cost of acquiring customers (or policies/subscriptions) will only go up as more of my competitors sail for the calm waters of "cost per acquisition" pricing.

So, if I need to sell 10,000 policies every year I have 2 options.

  • Buy cheap customers and hope that a few may buy again
  • Buy the right customers that stay with me for 2 or even 3 years

Without doing the value-based segmentation we'll never understand which channels bring in the best customers and that would be a terrible shame.

The ground truth is that I can re-new a policy or subscription for considerably less than buying a new one. How?

One strategy might be to spend an extravagant $1.00 of marketing costs to show my love an appreciation to our customers throughout the year via email or social media, increasing the chances they'll buy again.

That means I won't have to spend $20.00 buying a new one… a saving of $19.00 per renewal.

So if I can grow my repeat purchase rate from 20% to 40% that means I will generate 2,000 policies at $1.00, not $20.00.

That's a $19.00 savings on each of the 2,000 policies. BAM!!

Moving to a Life Time Value acquisition strategy will save my company $38,000. Not bad for a couple of days work.

Let's finish off the concepts of value based segmentation and lifetime value by going back to the original example we were working through.

If we can identify channels, campaigns, media or propositions that deliver "better than average" customers we can begin to see how much more profitable they are and decide how much more we should be spending on them.

Here's the (sample) analysis I (or you!) would do:

life time value lifetime net profit 1

Ladies and Gentlemen – select your lifetime!

In the above example I've modelled a 3 year lifetime – that would be sensible for a typical consumer e-commerce player.

Publishers and financial services companies may take a longer term view… certainly off-line we have been building 5 year plans in publishing for decades.

If you're more comfortable with 6 months or 18 months, then go for it!

If you do you'll need to be looking backwards and forwards at the same time.

You may only have 6 months of on-going data for some segments, but use that as a starting point and build some simple scenarios from there:

  • What if 50% of them spent 10% more in the next 12 months?
  • What if 30% of them spent 40% more in the next 6 months?

Over time as you replace modelled data with real data you should be able to re-weight your acquisition spend, replacing one affiliate with another as the cross and up-sell orders begin to roll in (i.e. the customers you acquired begin to make additional purchases from you).

By rewarding the better partners / media / acquisition channels with higher CPA's you'll be building a defensive position that prevents competitors buying their way into the good sources ("How can they afford to pay THAT MUCH?!" they'll all be wondering). It will be our little secret.

Life Time Value is for Life, not just for Christmas.

We've really only just scratched the surface of LTV in this blog post.

Many people have devoted their whole careers to unlocking its mysteries so apologies to all of them for the "top level" content here.

However, it is a concept that deserves the attention of a new generation of digital marketer and it will alter the way many companies approach acquiring and retaining customers.

__________________________________________________

Amazed?

Don't cha feel a little bad that you were making decisions about where to invest your precious marketing dollars based on either Conversion Rate or based on Cost Per Acquisition?

What's scary is that you could currently be using Conversion / Average Order Size / Cost Per Acquisition to invest more in one particular channel, all the while, unbeknownst to you, shoveling "poor quality" customers. Or "high CPA's" might have caused you to not spend enough on a channel where you can get lots and lots of high value customers.

Scary! Yet exciting that finally you can be so much smarter!!

Bonus: As a very special treat David's created an Excel spreadsheet to help jumpstart your Lifetime Value journey.

The spreadsheet has two tabs.

Comparison LTV lets you model two segments of customers by helping you walk through clearly articulated questions.

Detailed LTV kicks things up a few notches by allowing you to make better decisions by modeling out the long term performance for a given customer segment. [Create more copies of this tab to model out multiple customer segments and then compare / contrast to make wiser decisions.]

Download: Comparison + Detailed Lifetime Value Model.

[Please do not click on the link above, rather right mouse click and choose Save Link As or equivalent in your browser. Thanks.]


aim focus shoot win

 

Closing Operational Thoughts:

I wanted to add a few thoughts about the operational things you need to worry about / keep in mind, as you revolutionize your company by using LTV:

1. You'll notice instantly that almost none of the data above is available in your web analytics tool. Not Omniture's Site Catalyst, WebTrends Analytics, Coremetrics, Google Analytics or Unica or whatever. This type of PII and financial data does not exit in these tools (often for a very very good reason).

Even the web analytics tools that say they create Lifetime Individual Visitor Experience (LIVE) profiles to compute Customer Lifetime Value (CLV) won't have the key Margin or multi-channel data, and hence not truly allow you to do the above, contrary to what might have been implied.

Web Analytics tools, even ones with lifetime visitor profiles, usually can't even stich together one person's clickstream behavior over the long term because of cookies and other data erosion issues. So plan on looking outside.

2. [Because of reasons immediately above and more...] Remember to focus not on the "Individual Customer", rather focus on the acquisition channel by analyzing segments of customers.

Individual anything ("you can track every single customer and understand every single customer and react to them in real time!!!") is over-rated.

Optimizing acquisition channels with LTV. Yea! Optimizing for Jim Sterne with LTV. Nah!

3. You'll do most of this type of analysis via your ERP / customer data storage system / financial data warehouse.

Your BFF will be the Finance team, both to initially teach you some of the financial intricacies and give you access to data you need. Look 'em up. Take 'em out for dinner. Trust me when I say that the LTV work will be a tremendous asset to your career and expose you to the highest levels of your organization. A really really good thing.

4. You are going to have to darn near sleep with your IT team/person to ensure the key meta-data required to do this analysis passes from your website to the sources mentioned in #3 above.

For example in my first job I had to request (ok beg) the corporate IT team (ok one person) to enhance the corporate system with two columns so each web order order could be distinctly identified and contain "campaign id" and "acquisition cost".

The lack of this meta-data is where most LTV efforts fail.

Even if you are a 100% web business you'll have to ensure the "backend system" that contains this key web analytics meta-data else you are doomed. Sorry.

5. If you are multi-channel company (web, call center, stores, catalogs) you'll want to ensure an equivalency exists in your backend system to [A] track the same customer's multi-channel orders correctly [B] contain cost data from all multi-channel campaigns.

This is really really hard to do. Don't try to climb mount Everest on day one. Start small and build over time. Remember David's tips on making do with just what you have.

I want you to be aware of these few valuable lessons I have learned in my own journey. I had to learn them the hard way. : )

If LTV sounds like it needs effort and love then you have understood it correctly. Everything worth it is hard in life, but if you put in the effort you'll create an enviable advantage for yourself and your company.

In closing:

1. Focus on long term success, acquiring truly valuable customers…

2. by embracing Net Profit and Lifetime Value…

3. and becoming BFF's with the Finance Team, good things will come of it!

Good luck!

Ok now your turn.

Have you used lifetime value or other such metrics to enhance your acquisition strategies? What was your experience like? If you have not used LTV, do you plan to use it now? What did you find to be of most value in this post? What would you disagree with? Did you want to run to England and give David a hug? :)

Please share your feedback via comments.