Excellent Analytics Tip #12: Unsuspected Correlations Are Sweet!
This post could also have been titled "Tracking Radio Ads" or "Measuring Online Impact of Offline Marketing" or "Success in a Nonline World". It will touch on all of those.
But the title is what it is because the most lovely part of this story was how tracking with a one dimensional mindset (or in a silo) means that you will end up missing so much of the picture. And it is a story about what it means to be a Analyst in this day and age.
[This post is dedicated to my friend Nick: You are a sweetie! Thanks for everything.]
A delightful company, let's call them Market Fire Extinguishers, located at www . marketfireextinguishers . com (there is no such site as of today), would like to grow their business. They have tried lots of nice things online (primarly affiliate marketing). It worked ok.
Then they came upon a radical, for the web, idea: Run radio ads around the country!
This is getting easier to do than ever with many leading companies getting into the space and lowering the barriers to entry. Even I can run a 50 city radio ad for Web Analytics: An Hour A Day.
They ran campaigns across the US, in 50+ DMAs each that had their own set of cities.
The call to action was primarily driving people online, via a easy to remember vanity url (redirect) www . eztz . com (I am making that up here to protect the client). The audio ad also mentioned a toll free 800 number that listeners could call and purchase the product.
Using the vanity was smart, for tracking purposes (hurray!), keeping it easy to remember was even smarter. Driving people to the site was a business requirement because picking up the phone is expensive.
The first "management" level reporting was extremely simple, and visually appealing (after all pretty sells!). It was an attempt at answering the question: "we have all these ads running in 500 plus cities, which of those are effective at driving traffic to our website?".
Here is the baseline picture, before the campaign. . . .
And here is the picture that shows the impact of the radio ads. . . .
Sweet lord that is impact! Get me the champagne!
The nice "chicken pox" geo report is a good visualization in this case because it quickly shows the top 100 cities that sent in the traffic. The before and after makes a nice story in of itself in terms of showing impact of the campaign.
It is also easy, as you can imagine, to dig deeper into the data and analyze all 500 plus cities that sent traffic. You will surely go in and look at the audio ad costs in each city, the number of listeners in each (from a source like Arbitron), the number of resulting Visits and Absolute Unique Visitors etc.
Being a fan of the Trinity Strategy you will surely dive into understanding outcomes (lead forms submitted, orders placed, samples requested and what not). Then you can pick which cities were ROI positive, which were not etc etc.
One of these days it will be easy for you to do much of the above analysis in your web analytics tool.
One graph you will surely create will look like this one. . . .
A correlation of two trends, the brown is the radio ad impressions that you achieved during the course of this campaign. In blue is the resulting traffic to the website.
Nice. Clear correlation between the ad showing up and the traffic spiking on the site, the ebb and flow also match nicely though after every drop in impressions you see residual impact on the site traffic (the blue line drops less dramatically). All very wonderful.
Most people will leave it at this when it comes to measuring success for their campaigns (be they email or audio or tv or SEM/PPC or direct marketing etc). On the web, delightfully I might add, it is rare that a butterfly flutters its wing here and a tree falls in the amazon. (Ok I have been unwell for a few days now and am out of my metaphors juice!)
You take action in one channel / medium and it will surely have a impact in other channels / mediums.
"Unsuspected correlations."
So when you analyze your own campaigns / valiant traffic driving efforts, correlate other sets of data you have to see if there are hidden correlations that could help you understand the story better.
In our story the Analyst Ninja did exactly that and added a line to the graph showing the Traffic from Branded Search for the same time duration. . . .
Lovely!
Not only did the radio campaigns drive people to the site using the easy to remember vanity url, eztz . com, but our lovely radio audience, brainwashed as they are like the rest of us, also ended up typing in queries into the search engine and arriving at our site.
Not a flash in the pan, but a consistent trends, mirroring the ebbs and flows of the radio ads.
This was a surprise because that vanity is not that hard to remember yet people used the terms mentioned in the ad (company name, product name, other brand or category terms) and used a search engine to get to the site.
Lesson: People behave in ways that they are used to and many of 'em won't do what you want them to do!
Unsuspected correlation in this case raised the amount of ROI the audio campaign could claim.
But our brave Analyst was not one yet. She tried one more thing. . . .
Direct traffic in light blue.
Again a clear trend in the impact of the radio ads on the Direct Traffic (your web analytics tool could be calling this "bookmarks" or "type in" or "none" etc). Technically it is people who have "no referrers" in their session.
In this case it was people who, again this is normal on the web, heard Market Fire Extinguishers and typed in www . marketfireextinguishers .com and got to the site. And that is a hard url to spell!
Another unsuspecting correlation.
In the end the impact of the radio campaigns on this particular website was significantly more than original imagined.
There are interesting implications of the above when it comes to your next media buy and the kind of customer behaviour that will impact it.
Net Net: Next time you are asked to produce a ROI analysis perhaps you'll think of this example and ask yourself if you have correlated enough data streams and looked hard enough to paint the complete picture.
Important Web Analyst Skill Observation:
There is one other facet of this lesson that was important to me. Lots of us get so entrenched in numbers and analysis and Omniture / WebTrends / Google Analytics / IndexTools etc that we often lose touch with the outside world.
The worlds of 1) online marketing (latest trends, happenings, changes, techniques, what not) and 2) online customer behavior (again latest behavioral trends, how people experience the web, likes, dislikes etc).
If you want to be a Magnificent Analyst spend 50% of your time with the above two. Anyone can learn to press buttons or extract data into excel. And thousands are learning that every day. What will set you apart will be your superior knowledge of the marketing and customer experience ecosystem in which we all exist.
Less than 10% of the Analysts I meet are proficient in those two things, almost everyone is proficient in the numerous web analytics tools.
It might seem obvious in hindsight to do the above analysis, I assure you that it was not. The person who did this was less a "web analyst" and more a "online marketer", the 10% I mention above.
Be that.
Important Observation #2: Correlation does not imply causation.
Correlations are a very advanced statistical technique that I am using in perhaps its most humble and lame manner above. (That had to be said!)
More importantly it is important to realize that Correlation does not imply causality.
In our case above we controlled for other externals factors (no other campaigns running, no weird seasonality carp – notice the campaigns ran after Christmas etc). This was to ensure that we were not seeing patterns where none should exist.
It is important to internalize this.
Another example. Here are number of posts I have written in the last few months and the number of RSS feed subscribers in each.
Oct: 6 posts, 5,542 subscribers.
Nov: 5 posts, 5,829 subscribers.
Dec: 4 posts, 6,338 subscribers.
Jan: 4 posts, 6,898 subscribers.
Feb: 2 posts, 7,666 subscribers.
The less I write the more subscribers I get (!!).
Those two variables seem to be correlated, but they are (hopefully!!) not causal.
So be careful with causation.
That's it. Lesson over. Hopefully you found it to be of value.
Now its your turn.
Please share your own lessons, perspectives, critique, bouquets and brickbats via comments. What works for you? What does not? Add your voice. Thank you.
[Like this post? For more posts like this please click here, if it might be of interest please check out my book: Web Analytics: An Hour A Day.]
PS:
Couple other related posts:
- Excellent Analytics Tip#1: Statistical Significance
- Personas, Customer Value, Customer Retention and Non-line Marketing
- Data Mining And Predictive Analytics On Web Data Works? Nyet!
- Top Ten: Signs You Are A Great Analyst
Subscribe 
February 19th, 2008 at 04:58
A good reminder on the perils of correlation. If my analyst were handing me the graph and telling me "radio works", I immediately notice two things:
1. The items line up too well. Red flag: the world is never that neat, much as we'd like it to be.
2. There's no lag on the radio start. Red flag: there's normally some lag between starting radio and seeing the traffic begin to spike, as the message takes some time to "set in".
Of course, Avinash, I assume the graphs created are illustrative only. Anything I got that was THAT neat and tidy looking sends off all kinds of alarm bells.
February 19th, 2008 at 08:23
The causality is such a good point, Avinash. I have learned to be very careful in the ways that I publish data as too many times my correlations have nearly been turned into doctrine as people see what they want to see. Mark makes a good point about the tidy report, as well. I have seen that when things line up very well people are ready to close the case and take action. I think it happens because all too many times we load our analysis consumers down with caveats and such to explain fuzziness in the data, and when things do line up in a picture-perfect sense they recognize it as perfect clarity.
Thanks for the great subject.
February 19th, 2008 at 10:30
Mark : Absolutely excellent points. Couple of contextual thoughts. . . .
The data is all real data, untouched in any way, which is what made this whole thing so fascinating to me.
The items seem way too lined up than they should be partly because the graph was "squished" to fit into 480 pixels, the width of my blog. It makes things appear more correlated than they are. That was one of the reason I pointed out the lags in the blue lines, to illustrated that the drop off in traffic was not as steep as the drop off in the radio ads! You'll notice that is the case on almost every data point.
The second point is very true and something that threw me off as well. My personal analysis of the situation was: The product is rather unique, it is something we would die to have ("you have been waiting all your life to get rid of that pain, call now for the cure that is God sent", yes slight exaggeration :). The promos were targeted to be "drive time", that worked well to get traffic when people got to work.
For this business Radio turned out to be a great answer to improve brand awareness and drive people online. It is a function of many things including the youth of the company.
-Avinash.
PS: Hopefully your Analyst won't come running to you just with this graph but rather the one that plots Outcomes (because this just shows that people showed up!). Then She/He could exclaim: "Radio works! Look we made $15 million, net!!" Then bonuses all around. :)
February 19th, 2008 at 11:24
Great post Avinash. Audio has been a key part of our strategy for some time now but tracking the online impact has been quite tough. We were doing some of what you suggest here like track traffic by city. But I did not realize all the other impact that our radio campaigns might be having.
As always you have opened my eyes to something new. I am off to run my correlations now.
Thanks.
February 19th, 2008 at 21:58
Hi Avinash,
I don't have a link, right now, but I remember that Anil Batra once had a post about a negative correlation between the number of posts he wrote and the number of subscribers he got.
He concluded that if there's a bigger pause between two of his posts, people might feel more tempted to subscribe, so they don't have to come to the site every (other) day if there'll only be a post about every two weeks.
I hope I'm not misquoting Anil and I know it's more of a theory than hard evidence, but it sounds like he might be right about it.
Maybe some other people on here have blogs with a high enough traffic number and have witnessed the same thing?
oh and nice post!:-)
February 20th, 2008 at 04:34
It's good to finally see a correlation between radio's ability to drive traffic to a web site and the returns. The radio industry is still far behind in understanding this approach and its potential.
I did a test in May 2007 that was close to what you described above, only it was done where 70% of radio's revenue is generated – on the local level.
http://www.audiographics.com/agd/071007-2.htm
The one point needing to be emphasized is that, in doing a comparison between radio and online advertising, it is almost certain that radio's ROI will be lower.
The results from my May campaign showed radio cost the client $143 for each response (a registration), while the accompanying kiosk data response was just $8 each.
February 20th, 2008 at 07:04
[...]
Excellent Analytics Tip #12: Unsuspected Correlations Are Sweet! –
Very interesting correlations between offline advertising and online activity. Great post!
[...]
February 20th, 2008 at 08:28
Great post Avinash, Thanks a lot.
I am working on the project of finding the increase in the sales of Apparels through Web Advertising, and I found this post. It's interesting.
It would also be nice if you attack my problem sometime. Will be looking for it.
Cheers,
Bhupendra
February 20th, 2008 at 11:15
Ken : I suspect that mileage that each business will get might differ when it comes to ROI from their radio ads.
In places where I have privilege to do the analysis the numbers have been all over the place, depending on the outcome and type of business. But it is not unusual to see conversion from radio to be between a dollar to five dollars.
Though for some businesses it could very well be the $143 you mention.
As always YMMV and experimentation is the way to go, try it, measure it, rinse and repeat if it makes sense. :)
Thanks for sharing the link, it was a enjoyable read.
-Avinash.
February 20th, 2008 at 18:03
Crazy coincidence but, here in Perth (Australia), they just ran an ad for, funnily enough, advertising on the radio.
The point was to show how much a website can get from a radio ad, so the "website guy" takes a picture with his camera of the "radio guy" interviewing him, then says, "I'm posting this on a new site… called… http://www.thatradiobloke.com"
Sure enough, that URL takes you to radioadvertising.com.au or sommat, but then it pops up with a huge closeup of some crusty dude in a radio booth.
It was classic, and got a giggle out of me. If I could afford it, I'd totally advertise on the radio. And I'd make it FUNNY.
February 20th, 2008 at 20:54
Avinash, I saw this on sphinn and sphunn it immediately (the submitter was a trusted source and the destination url was even more trusted ;) ). And boy am I glad I did!
First, this is interesting because I spoke with another excellent online marketer, Mikkel deMib Svendsen. His take was that offline does NOT drive traffic offline, let alone convert. And for support he pointed at the dotcom bubble, which is a pretty good piece of evidence imho. That said, he may have been thinking more about billboard ads. Any experience with those? Also, what's your experience been like with offline promotion generally, as far as getting visits online?
Second, I'm enjoying your book and the relation to your site; the former allows me to better understand your blog :).
Besides that, I hope you get better and suggest you don't blog for another month or two – you'll quadruple subscribers ;D!
February 22nd, 2008 at 12:23
Avinash, this is one post really worth reading. There are numerous articles and commentary about analytics but there are so few who really get deep enough and present it in such a practical way. You Rock!
One think I'd expect to see in the charts, is that the radio ads should have a longer effect i.e rather than just getting visits during the campaign, I'd expect to see a general rise even after the campaign ended (hence the quality of the campaign for a longer term can also be measured).
Secondly, a question: Is there a chance you can share some numbers such as conversion rate or cost per visit from the radio ads converted as visitors? i.e. if you invested $100K in radio, and the difference in traffic (organic, branded, direct) was 100K visitors then radio ad visits would cost 1$ each… I think this maybe be a number a lot of us could appreciate as a benchmark (even without the exact theme/vertical).
Thanks Again!
Ophir
February 22nd, 2008 at 22:18
Ophir : There was slight uplift after the campaign ended but not that much. It was a function of the client and the product being advertised.
The lessons from the Audio Campaign were invaluable in understanding the dynamics of driving traffic to the site and now the client it going with a robust portfolio of campaigns (audio and more).
I am positive as the product gains wider recognition and the company matures we will end up seeing trends that we typically expect. But at this stage in the life cycle it was not surprising.
I am afraid that in this case I can't share any specific success metrics, the client is not comfortable with that. You understand.
But I can say that the campaign was very positive in terms of the ROI (actually traffic far exceeded expectations).
The cost per acquisition was not quite what I have seen with paid search for other such companies but it was close, for this client.
The more delightful thing (as I point out in the post above) was the precious learning about how people behave and get to the site.
Thanks for the kind words, I am thrilled you found the post to be of value.
-Avinash.
February 23rd, 2008 at 16:16
Great post Avinash. As Patrick M, mentioned above, I saw a negative correlation between blog post frequency and number of subscriber, here is the link to my post
http://webanalysis.blogspot.com/2007/07/increase-in-subscribers-with-longer-gap.html
February 25th, 2008 at 18:39
Avinash,
Another great post. What so amazing to me is that media today is bought and measured in silos while from the consumer's point of view there is no difference. And in this case, offline drove traffic online.
I can envision some web team giving each other high-5's for a successful online campaign oblivious to the fact that their traditional media counterparts just spent 10x budget which drove more people to the site.
I also would think TV, PR and Event Marketing would have the same effect to a website.
Anyways great post. You sure are lucky to work with smart guys like Nick
-SS69
February 27th, 2008 at 07:35
A client read this blog and now wants a report similar to the graph above, but instead have "leads, campaigns, and tactics" instead of "direct visits, visits to extz.com, brand search".
How do I track what my client wants and provide a report on that. I'm scratching my head on this issue.
thanks for your blogs.
February 27th, 2008 at 09:22
Winn : First off tell you client that I love her/him for reading the blog, and more importantly for wanting to take action!! :)
The graph is in good old excel so really not a big deal.
Perhaps you are struggling with how to get the data for "leads, campaigns, and tactics".
Three things can be awesome:
1) It is just a matter of ensuring that you are tagging all the sources that you want to track. So make sure that your affiliate or banner or paid search or email or other campaigns have url's pointing to your site that are encoded with the right tracking.
This post on the Google Analytics Blog shows one example of how to semi automate campaign tagging with the URL Builder:
* Where Did My Paid Search Traffic Go?
2) Make sure your pages are tagged properly with the web analytics code. You will need this, for example, to get a accurate count of leads / goals / conversions.
3) Good old segmentation! What your client is asking you to do is essentially segment the traffic and outcomes and layer that on the graph (which is exactly what I have done in the graphs in the post). Use the segmentation capabilities of your tool to pull out the data for segments you are interested in.
Hope this helps a bit, good luck!
-Avinash.
February 27th, 2008 at 09:49
Avinash,
Great post. I have a question. How/where did you get the baseline numbers to make the 1st Geomap chart from? Was there an existing campaign going on that they captured these from or did the numbers come from elsewhere like a visitor geography report showing existing traffic levels from these locations?
How much faith can you put into visitor geography reports? It seems like some of the data shows the location of internet service providers used by users instead of users themselves. This can be misleading.
Do you have any suggestions for creating Geocharts in terms of software?
February 27th, 2008 at 09:56
Alice : Second question first – the geo graphs are from good old Google Analytics (the "chicken pox" reports). Most web analytics have the ability to make nice Geo reports. I am unaware of a dedicated software (but that is just a gap in my knowledge).
For the first question I am going to cheat. Rick just asked a question via email about Geo data and its accuracy (especially that Virgina often shows up high).
Here is my reply and I think it answers your question:
Hope this helps.
-Avinash.
February 27th, 2008 at 11:16
Hi, Great information. From this site i got the information of on-line marketing and this is really helpful for us to increase our website traffic.
March 5th, 2008 at 09:45
[...] Avinash Kaushik, one of my favorite bloggers and Web Analytic Guru, who also happens to consult Google with regards to Web Analytics, wrote a post a few weeks ago analyzing the correlations between radio ads and web site traffic, with deep insights into web site traffic, branded vs. general searches and direct visits. It now makes even much more sense. Google will now enable advertisers who run audio ads, to analyze their effect on their web site, with integrated data from the Google Audio Ads panel and get insights from the cross-channel advertising not only on the online but also offline. [...]
April 26th, 2008 at 13:39
[...] Excellent Analytics Tip #12: Unsuspected Correlations Are Sweet! [...]
May 30th, 2008 at 04:43
I'm interested in learning more about your book- I noticed your blogs started in 2006- is the book up to date with all of the new Google stuff brought out this year(2008)? If so- I'm ready to order to help my online bath body company soar! you seem to really understand this stuff incredibly clear:-)
June 1st, 2008 at 21:55
Castle: My book is not about Google Analytics.
Web Analytics: An Hour A Day was released 10 months ago and is a book that is about the business and technology and strategy of web analytics. It takes a fresh and deep look at what it means to be successful at web analytics, independent of what tool you might be using and what company you might be working at. Please check out the description at amazon for more: http://www.snipurl.com/wahour
If you want a Google Analytics then might I recommend these two excellent books:
Google Analytics Shortcuts by Justin Cutroni
http://www.oreilly.com/catalog/9780596514969/
Advanced Web Metrics with Google Analytics by Brian Clifton
http://tinyurl.com/4cd6p8
I know both of these guys and they are very very good at what they do (so buy their books right away and you won't regret it!).
Hope this helps.
-Avinash.
June 7th, 2008 at 13:43
[...]
Excellent Analytics Tip #12: Unsuspected Correlations Are Sweet!
February 19th 2008 8:31am
[ok] From: http://www.kaushik.net
[...]
June 26th, 2008 at 05:18
[...] I discovered what web analytics represent to a website just 4 months ago. I saw the book “web analytics on hour a day” from Avisnash Kaushik in a library and the “About the Author” page mentioned his very popular blog on web analytics. I had no idea who was Avisnash and I was also really unaware of Web Analytics. I went to the blog and the article Excellent Analytics Tip #12: Unsuspected Correlations Are Sweet! was just written. I do not know if you read that post already, but all I know is the impact it actually had on me (nothing emotional – don’t worry :-) ). I was so blown away by all the post itself and the correlation you have between metrics. It was just… huge! [...]