A Primer On Web Analytics Visitor Tracking Cookies
If you are using a modern web analytics tool (tag based or log based) it is quite likely that it is using cookies for tracking purposes.
In my conversations it is embarrassingly common to find a lot of FUD and confusion and lack of understanding (or appreciation of!) cookies and the role that they play in any analytics done on the web.
Hence my attempt at this simple easy to understand primer. If you are an Analyst or a Marketer or a Website Owner or a Website User it is critical that you read this short blog post – your data will make so much more sense after are done.
Why are cookies important?
Cookies, usually anonymously, allow the website owner to measure the number of Visits and the Unique Visitors to the website and hence understand the Customer's website experience and segment visitors that are New to the site from those that are Returning.
That's it.
No more and no less.
Lots of other tracking is possible without the use of cookies, they are not the be all and end all of visitor behavior tracking. Wipe that sweat off your forehead. Go get a cold glass of water to drink.
Let's attack the rest of this complex issue in a few bite sized understandable chunks.
Transient vs. Persistent.
There are two types of cookies that the web analytics software will set when you visit a website. They are commonly called "Transient" and "Persistent" cookies. Some folks refer to them as "session" and "user" cookies respectively.
The job of the transient cookies is to help "sessionize" your experience on a website. Put simply, you are going to make a series of clicks and leave. That's a session. The transient cookie helps group those clicks efficiently.
The transient cookie is "set" when you visit the site, it disappears when you leave.
The persistent cookie is set the first time when you visit the website, and it will remain there for the duration that the website determines. For example, Analytics cookies are typically 18 months but many other tools will use anything from 18 months to 18 years. Persistent cookies are there to help identify a unique browser to your website, in as much they are the closest thing to tracking a "person" / "unique visitor".
The persistent cookie is on your browser until you either delete it, reinstall your browser or do other such things.
[
It is important to note that almost always persistent cookies don't contain any PII - Personally Identifiable Information - data. They just have a random string of numbers or alphabets that only the company who set the cookie can read. For example here is a cookie that Webtrends.com just set on my browser as I visited www.webtrends.com: C8ctADY1LjU3LjI0NS4xMS00MTU3MTQwMTc2LjI5OTQ0NzE5AAAAAAAAAAACAAAAo
M0AAINghUgWYIVI.
I should see if I can mess with them by changing that cookie to COREMETRICSWILLALWAYSBEWATCHINGYOURSITE4LOVE+OMNITUREISGREATAND
INDEXTOOLSWINS!! :)
]
First Party vs. Third Party.
A "third party" cookie is set by, well, a third party when someone visits your site. So if www.omniture.com is using WebTrends as the web analytics tool of choice then when I visit omniture.com a cookie will be set on my machine under the www.Webtrends.com domain. On omniture.com a Webtrends.com cookie is considered a third party cookie.
[Omniture.com is actually setting cookies using .2o7.net which would make them third party cookies on that domain.]
In the good old days it was easier for the web analytics vendors to use third party cookies and they were rampant. But it was discovered that there were other players using these cookies in sub optimal ways. This lead to default internet browser settings that would reject third party cookies, and many other anti spyware and malware programs auto deleting them etc etc. Suffice it to say they have fallen out of favor, and are considered quite sub optimal for tracking "unique visitors".
A "first party" cookie, hence, is set by the web analytics tool using the domain of the website itself. As an example when you visit www.coremetrics.com you'll notice (if you have WASP!) that they are setting cookies using the domain data.coremetrics.com – which makes those cookies first party.
First party cookies are the preferred tool of choice for tracking "unique visitors" because they are deleted / rejected a lot less by any objective measure. This means, for example, they are a far superior at tracking repeat visits or new and returning visitor segments etc.
Another reason first party cookies are rejected a lot less is that much of the internet does not work if you don't accept first party cookies. Email providers like hotmail (! :) or gmail.com, ecommerce websites like amazon.com or crutchfield.com, banks, even blogging platforms! They all require you to accept first party cookies.
Almost every single decent web analytics vendor now provides an easy ability for you to use first party cookies. Some like Google Analytics only offer the option of having first party cookies.
If you notice some initial push back from your vendor to use the easier-for-them third party option, do a little push back of your own. Insist on first party. Its good for your health.
Exception for Third Party Cookies.
There are some relevant uses of third party cookies. One of the most common is by ad serving platforms because that is the only way they can track a "unique visitor" across multiple websites. So even if that third party cookie gets blown away and rejected a lot more, they (you) really don't have much of a choice. That's just how the internet protocols work.
Here's a example of how that works.
We saw that omniture.com is using .2o7.net third party cookies. After going to omniture.com I could go to ebay.com and then to nytimes.com. .2o7.net knows that I was at the Omniture site a little while back and then I went to eBay and then NYTimes.
Now as I am reading the latest Maureen Dowd column .2o7.net (if it was a ad serving platform) could serve me a ad for Omniture next to the Maureen Dowd column. Knowing I also went to eBay they could even give me a deal on Omniture in that ad! : )
This is of course just one example to illustrate the use of a third party cookie and why Atlas and DoubleClick and Yahoo and all the others use them (and provide value to their customers).
First party cookies can't be "read" and "carried over" like the above scenario.
Does my choice (1st or 3rd) influence where my data is stored?
No.
The type of web analytics software you use determines that.
If you are using a ASP based solution (say NetInsight or Microsoft AdCenter Analytics or VisiStat) then both your first party or third party cookie data is stored in the data center of your application service provider (vendor).
If you are using a in-house solution (like ClickTracks or Urchin) then your data is stored in your own data center (regardless of what kind of cookie you use).
Cookie Deletion Rates.
It is important to remember cookie rejection is not the same as deletion. With rejection you don't even accept (worsens tracking). With deletion you collect data for the session (visit) but tracking after that visit worsens.
Everyone wants to know cookie deletion rates ("help my web analytics data is crap!"). There is no "global standard". Sadly I have never seen a study that was objective and not pushing the vested interests of the publisher (be it a company or a "analyst").
It is also extremely extremely difficult for a "third party" to have the kind of access required to actual data that would help them develop anything close to a objective "standard".
The biggest determining factors are your customers and their browser settings and software on their computer. And that can vary greatly from site to site.
My own personal experience across a number of ecommerce, support, and other corporate sites (excepting extremely "tech heavy audience" sites) has helped me come up with a "benchmark" of cookie deletion rates of 3% to 5% for first party cookies and 20% to 25% for third party cookies. They all tend to fall in that range.
FWIW.
If you want to know what the number is for you, I recommend putting in the sweat, blood and tears to measure it on your actual site. If it is important to you, it is important that you don't just take someone's word for it and proceed to evaluate your own web analytics data and get your own benchmark. I assure you that you are unique.
Do I have to use cookies?
The current generation of web analytics tools all use cookies to perform the core function of "accurately" compute Visits and Unique Visitors.
If you use cookies those numbers will be better (not perfect, see this post: Data Quality Sucks, Let’s Just Get Over It).
You will get a better understanding of metrics like Visits to Purchase or New and Returning Visitors or even Conversion Rates.
But if your company executives or, more likely, website customers have a preference for you not to use cookies then you don't have to.
You won't be able to measure some of the above Key Performance Indicators, but you can still get good value from the cookie-less data that you do collect. Top Visited Pages, Revenue, Referring Websites (URL's), Search Engine Keywords and on and on and on.
Don't let the fact that you don't use cookies get in the way of being able to use the web analytics data in meaningful ways.
The data won't be perfect but then again perfection is greatly overrated! (Chapter 13, Page 341 of my book.)
[
There are analytics tools that allow you to use alternatives to cookies to compute Visits and Visitors. You can use user_agent_id's, combination of browser_id and operating system etc. See if your Management or Customers are ok with that. If yes, use those. If not, to stress again, the data you collect, anonymously, can still reveal insights of value.
]
Is privacy important?
I know that sounds like the most obvious question in the world, with the most obvious answer in the world.
Yes. It is.
The primary function of your website is to be responsive to your customers. It is important to have a clear privacy policy, it is important to be transparent about what you are collecting (especially if you are collecting PII – personally identifiable information), and to educate your users.
Here's my humble privacy policy (you'll always find it in the footer).
Be transparent, there are few things more important than the trust of your customers.
Besides as I have stressed several times, even with what data you can collect (say you just have your raw server web logs and nothing else) it is possible to find insights. Nothing's impossible for a Analysis Ninja!
That's it.
You are now a graduate of Cookies 301. May the force be with you!
I would love to hear your feedback on this delightful and often beguiling topic. What do you think of cookies? What has worked for you? What did not? How have you overcome obstacles? Any tips for the rest of us?
I am sure you have stories you can't wait to share. Please do.
Thanks.
PS:
Couple other related posts you might find interesting:
July 24th, 2008 at 01:15
Well that depends on your definition of "does not work" :-)
Excluding work's websites, I have only about 20 or so sites whose cookies are accepted within my browser (be honoured Avinash, you're one of 'em. :-P ).
Yet I spend all day browsing around the 'Net reading and interacting with websites. Working.
Internet works fine for me! Come on in!
Sure there are sites that *need* cookies. Logins, Shopping carts etc. No argument. But you might be amazed at just how much of the 'Net as a whole works very fine without them. Or not. :-)
One extra I'd add for your readers (as I know you know this!). You can configure web servers to log cookies into your web server logs. Is *highly* recommended to do so if you use logs.
Cheers!
- Steve
July 24th, 2008 at 03:49
Avinash,
It's a very interesting post this one, especially because i find that many topics that you talk about are the same one that i've found myself covering in many meetings in the past 8 years working in the web analytics space.
A couple of comments that i think are mandatory after reading this.
The first one is about the 1st Vs. 3rd party cookie debate. One very important thing that everybody should always consider is P3P compliance of the cookie. P3P (Platform for Privacy Preferences) was introduced years ago (if i'm not wrong IE 5.0 was just released). It basically defines standard information that should be included in the cookie itself in order to "explain" to the internet users:
- Who is setting the cookie
- the reason why the cookie has been set
- the type of information that the cookie contains
- a link to the privacy page of the vendor setting the cookie
Any cookie can become P3P compliant. If a cookie is P3P compliant the information listed above will appear in the privacy report of the user browser each time the cookie is recieved / read by the website.
This is very important because by default since IE 5.0 (and other browsers later) by default a 3rd party cookie won't be accepted UNLESS it is a P3P compliant cookie. If it's P3P compliant it will be accepted unless the user specifies not to accept ANY cookie (maximum protection level).
How does this matter to me?
This means that if the Web Analytic Vendor uses 3rd party cookie, it should also be P3P compliant. The acceptance of that cookie will be much higher and very very close to the 1st party cookie acceptance (please also note that in some few cases the 1st party cookie non-P3P compliant could be blocked anyway!).
Regarding cookie deletion rates:
it's not all about cookie acceptance/rejection or cookie deletion. We should also consider what i usually define as "cookie generation". To understand what i mean here just think about your monthly surfing habits.
Let's first pretend you do not delete the cookies :-)
How many diferent browsers/devices do you use on a monthly basis? in my own case i usually use 2 PCs (sometimes my girlfriend's Mac too) both with IE and Firefox. Then i use the ipod touch (Safari). That's it. So over a monthly period i'm generating (using, surfing with) at least 7 different cookies. And this if i never delete them. How about you?
A unique visitor should be a real person, but even if we are 100% sure that our users NEVER delete cookies, they will for sure generate more cookies than 1 each.
Is this a problem?
I don't think this IS a problem. This is a scenario that needs to be taken in account, of course, but i don't consider it a problem for the simplest reason: don't Ad-Servers use the same technology (cookies)? Don't we plan an advertising campaign on a site based on Page Impressions and / or unique visitors? This last metric should, in fact, be called Unique Browsers, because that's what cookie measure. Browsers. If I use 3 browsers i'll generate 3 cookies. As simple as that.
I hope this explanation contribute to complete the scenario.
Ciao,
Manuel
July 24th, 2008 at 05:04
Hi Avinash,
I am an ardent fan of your blog.I am getting unusual statistics from google analytics. Surprisingly "Referring sites" section showing that my website is getting visitors through google.com and even from our website.
How can google.com and my own website is considered as reffering site?
Plz crack this million dollar question!
July 24th, 2008 at 06:34
Very relevant!!
Rather than dwell on cookie deletion as something that messes my numbers I use trending when talking about UVs. There is nothing to say cookie deletion will be greater one month over the other. So if the overall traffic is trending the way I hope it is (up or down depending on the purpose of the page) then whatever is happening is OK. For authentication sites (sites requiring a sign-in) you have an additional check – not fool -poof.
July 24th, 2008 at 08:20
For me, "setting expectations" (with clients / co-workers / bosses) has worked the best. It's important to understand that Analytics packages aren't replacement for Quick Books or some other accounting software. They use first or third party cookies to collect visitor data, so your data isn't going to be 100% accurate, not even close (well close, but not too close). So as long as everyone understands this going in, things usually go a lot smoother than if people find out later that their web analytics programs use cookies and that people can block cookies :).
July 24th, 2008 at 09:09
In addition some web analytics providers (such as coremetrics) use the user/persistent cookie to link multiple sessions together. In other words it is used to see that a visitor came in from a banner ad on Monday and then returned via natural search on Tuesday to actually purchase.
July 24th, 2008 at 12:14
Sujith,
The reason you are picking up your own site as a referrer is likely because you have links on your site that are absolute instead of being relevant.
If the URL in your hyperlinks read http://www.yoursite.com/movies/movies.aspx …they should instead read /movies/movies.aspx.
Use relative…not absolute links. Google is a referrer because of search traffic being sent your way or maybe links in google email from the mail servers leading traffic to your site.
July 24th, 2008 at 14:38
Joe,
One important thing would be to measure what is the standard error percentage.
Each site audience has his own cookie deletion ratio and cookie generation ratio. This won't change much over time. So you'll know how "wrong" those cookie based figures are.
But what's wrong for someone might be right for someone else. There is no absolute unique right way to measure everything. It's all about using the right measurement / metric for the right purpose.
:-)
Manuel
July 24th, 2008 at 22:04
This is a amazing post Avinash. I have learned a lot from your blog.
I am a follower of your blog in Korea. I am eagerly waiting the publishing of your book in Korean this weekend. Here is one of our Korean Blogger who write about it:
http://www.youzin.com/blog/?p=686
Good luck to you.
July 24th, 2008 at 22:11
Hi Avinash,
You mention that, in general, first-party cookies have a deletion rate of 3% to 5%. For what time frame is your estimate?
Cole
July 24th, 2008 at 22:52
Steve: I admit that line was "comment bait" and specifically directed at you. I had missed you not commenting on my recent posts! It is so much fun to argue with you. :)
You are absolutely right on logging the cookies into server logs, that is the reason I opened the post by saying you can use cookies with either tag based or log based solutions.
Manuel: Your comment is brilliant! Thank you, thank you.
You are absolutely right that even if no one deleted their cookies there are still issues with getting "accurate" counts of "persons" on the website. But the data is still very good (not perfect) and quite actionable.
And you are right about "unique browsers", I am sure you notice my cute use of that exact term, browsers, in my post several times. :)
I am deeply appreciative of your comment, thanks for adding to the conversation.
Joe: Here is one of my opening lines in these types of conversations:
"Consider what you can measure if you take out an ad in a magazine. Subscribers to the magazine at best. Maybe newsstand sales. How does that measure effectiveness of the ad? Its a total faith based initiative! Now consider that same ad but on the website of that magazine. Imagine what you can measure: Impressions, clicks, visits, revenue, leads….. Now it's not 100% accurate, but its not a faith based initiative!"
:)
Cole: In most cases it was going back three months (usually that yields a number closer to the first one). In a bunch of cases we went back six months, and that usually yielded a number closer to the higher in the range.
Hope this helps.
-Avinash.
July 24th, 2008 at 23:21
Dear Avinash,
This post has deleted most of *FUD* about cookie issues since Manuel also added his additional points too.
I have question in my mind. Can we capture the data of visits such as First Referrer & Last Referrer?
I would like to know that if new visitors come by organic source first time then next time is he visited the same site through PPC source.
You might be think what is this funny! :)
But I got one GA code from research which will provide you first referrer & last referrer too.Is this possible in GA?
I have not used that code till.
But if you want to look at this I will post it to you.
Thanks,
Bhagawat Jadhav.
July 25th, 2008 at 00:51
Avinash,
Thank you for the post. One question that I was hoping you could answer is the approximate % of users who block 1st party cookies. I have not found any reliable statistics covering this topic. The closest thing I found was from a post on Diablo Media but who knows from where they got there data.
What are your thoughts? I suppose this could vary greatly across segments and countries.
July 25th, 2008 at 08:10
Michael,
Global average of cookie acceptance rate is higer than 96% (not users, but page mpressions).
Source: Nielsen Online.
This means basically that given 100% of page impressions more than 96% will be generated from a user browser accepting cookie (being either 3rd or 1st party cookie). Please also note that this is the global average, therefore some sites might have an higher rate (e.g. technical users sites), but from my own experience i've never seen a site where the number of non-cookied page impressions was higher than 9-10%.
I hope this helps.
Ciao,
Manuel
July 25th, 2008 at 08:18
This post is very helpful, but what about if your company has several websites, all on different domains, and you want to understand whether visitors are going to just one, or several of your sites? Are there impacts to using 1st vs 3rd party cookies here? I assume if domainA.com has its own 1st party cookie, we need to set up domainB.com with its own 1st party cookie- but am not sure how to track a user who travels across those domains. Or, should I use some sort of custom variable/metric in my web analytics tool to track this separate from the cookies?
Thoughts?
July 25th, 2008 at 09:01
Great Post, Avinash!
And Alice Cooper, thanks for that. Its amazing what you can learn in the span of a cup of coffee!
July 25th, 2008 at 19:29
[...] As collecting, aggregating, synthesizing, and analyzing data is becoming a very important part of advertising (today on the web, tomorrow across all channels), understanding cookies is a must. Avinash Kaushik, the Analytics Evangelist at Google has written a fairly comprehensive review of cookies. You should probably read [...]
July 26th, 2008 at 18:33
Just wanted to correct something in the comment thread:
1) The #1 reason that a site in GA is its own referrer is that the configuration is wrong. Either one has subdomains or cross domains and the webmaster has (talk about topical!) not set up the configuration to manage the cookies correctly. The #2 reason is that it used to be set up wrong, it is set up correctly now, and you still have people with the wrong (yup) cookies. There is #3 reason, and it is escaping me….
2) When Google shows up as a referrer, it is not because of search. For search, it will show up under search engines. As a referrer, I see it the most often in customer accounts because visitors are using Google Reader or Google Home or Google Something Else where they have a link to you (usually a feed). It is also possible that you, the website owner, have answered a question on a Google Group and left a link there, and someone clicked it. I believe that gmail always shows up as gmail and never as Google.
July 26th, 2008 at 18:49
… continuation of comment above
OK, I remembered. The #3 reason that your site is your own referrer is untagged pages.
July 26th, 2008 at 21:56
Nancy: Ahh…. a tough question. :)
Let's take it in parts.
Each domain for optimal collection of data should set a first party cookie. That is the cleanest way to get numbers for each domain.
It also means that you won't be able to truly calculate the total de-duped number of Unique Visitors across all the domains.
If you want to track multiple domains as one then you can do something like what this article advices:
How do I install the tracking code if my site spans multiple domains?
You can also also try a couple of other techniques that are listed in this post (look for the question and answer right under the image of the earthen pots!):
Google Analytics Help: Questions, Answers, Tips, Ideas, Suggestions
Both these apply to GA but of course your vendor might have a solution as well, please check with them.
Net net this is a bit of work, but if it is important enough for you then you should put in the effort.
Robbin: You are a star! Thanks for adding the excellent answers for self referrers and for "the" google showing up as a referrer.
-Avinash.
July 29th, 2008 at 12:56
Robin – On your point # 2, you are right. Gmail, ymail would show as mail.google.com or mail.yahoo.com. The other referral for Google can also be the blending of top links that Google displays. A search for the word "Dictionary" on Google would provide you the links to various properties on dictionary.com 's website
(http://www.google.com/search?source=ig&hl=en&rlz=&=&q=dictionary&btnG=Google+Search)
Avinash – You might have also heard vendors talking about dedicated third party cookies. It is nothing different than your example above for Coremetrics.
As always, another great post.
Thanks,
Rahul
July 29th, 2008 at 13:38
Avinash this is a excellent post. Having been in the industry for ten plus years I can honestly say that I have yet to read such a cogent explanation of cookies.
Most discussion around cookies is either too harsh or too complementary. Your post is balanced and pragmatic and provides clear guidance.
Simon.
July 31st, 2008 at 10:31
Avinash,
It is interesting the topics of Cookies and FUD (Fear, Uncertainty, and Doubt) have come up on your blog.
I am a privacy advocate.
I realize my traffic data (usage, statistics, analytics, etc.) is out there and able to be collected.
I consider my traffic data to be a private exchange between me and and my intended destination (in this case, you).
First-Party and Third-Party both have legitimate uses for web optimization.
They help us build better website and make the internet stronger.
However, cookies do not exist in a vacuum.
Since the internet was not build with security in mind, there is room for trusted parties (especially Internet Service Providers) to manipulate and analyze traffic that is not intended for their use.
This topic gets sticky when you begin to consider the potential security implications of Third-Party and First-Party cookies.
Some recent news, you and your readers may not have heard of, has emerged in the security community that may be contributing to the "FUD" you references.
Fear of Third-Party cookies is not entirely unfounded because real-world examples of the use of First and Third-Party cookies being used for secret data gathering (and money making) have been found in the wild.
If you or your readers are interested in learning about potential origins of the FUD, or why some of it may not be FUD, I recommend you all review some the June and July 2008 episodes of Steve Gibson's Security Now podcast.
He reviews the "Phorm" technology and how certain ISPs have allowed a third party vendor to implement it, secretly, to gather data on the ISPs users in exchange for money.
One particular episode #153 covers this but it has been discussed in several: podcasts regarding Phorm
http://www.grc.com/sn/sn-153.txt
So yes, third party cookies themselves are fine, but there are real privacy and security issues that exploit vulnerabilities in the web. Unfortunately they will contribute to a certain level of "cookie rejections".
In my case, I browse the web with FireFox 3 which features 3rd Party cookies off by default. I run No-Script so I may control the execution of Java Script. I run flash block too.
I would recommend, as a best practice or at least a consideration, to avoid the use of 3rd party cookies whenever possible.
July 31st, 2008 at 22:47
James: Let me unequivocally say that the "FUD" that I was, coyly, referencing was not from the sources that you have very kindly identified. While I am quite aware of the sad sad story of "Phorm", I was unaware of Mr. Gibson's podcasts. They certainly make an interesting listen.
I am a Analytics Evangelist and while data is important to me, I am a big believer in both privacy and transparency. I am also a staunch advocate for collecting non-PII data becuase online you don't need PII to understand and optimize the customer experience.
Every single company that sets cookies or collects any kind of data online owes it to its customers to be 100% transparent (in easy to understand language).
Every individual then should have a option to set their own levels of privacy.
Knowledge + Choice = Trust.
Thanks so much for the thoughtful comment and the links.
-Avinash.
August 1st, 2008 at 10:11
[...] This was actually from last week, but it's still fresh information: Avinash Kaushik's for-dummies guide to tracking cookies. [...]
August 1st, 2008 at 14:47
I have been asked the question about how your own site can appear as the referrer a few times before and while I have made guesses (untagged pages generally), I have never felt that I had the real reason. But I was asked the question again today and had an idea which appears more logical and which the data subsequently proved to be accurate.
If a visitor accesses your site but clicks on a navigational link before the first page has fully loaded, the first measurement that is received will be for the second page, with the referrer being the URL for the first page. This is assuming that the page measurement code for your web analytics package is at the bottom of the page as it is for Nedstat.
I can't prove that this is the primary reason for having your own site as a referrer but it makes more sense to me than anything else. It is more likely to be an issue with content heavy pages or for visitors with lower download speeds. Can anyone let me know if they can prove or disprove this theory?
August 4th, 2008 at 06:11
This is a really interesting post, it explains more about cookies than I knew before.
But I think it underestimates the number of people who delete cookies. Except for cookies from sites I need to log on to (bank, discussion forums, Amazon, etc.), all of my cookies are deleted every time I close the browsers I use. So, if I understand you rightly, I'm a unique visitor every time I visit a site even if I visit it every day or several times a day (if I close my browser between visits). This puts a new spin on the stats websites brag about.
August 4th, 2008 at 13:33
[...]
Other Changes in Leadership: Having seen many cycles…both good and bad….it seems only too predictable that when things get challenging, changes are made in leadership that shake up existing plans. If organizations have their sales and marketing centralized then they decentralize. If they are decentralized, they centralize. The new CEO is a sales guy. You name it…major leadership changes will have an effect on marketing and sales…both in planning and in budget allocations.
Successful marketers embrace change and thrive on new challenges and ideas. These pros are close to the market, continually analyzing trends, and have plans in place that are both nimble and measureable. That way when the going gets tough….the tough get going….and they do it with confidence!!
[...]
August 7th, 2008 at 21:19
[...] Unique Visitors. A unique visitor is always counted according to a time frame. There are daily unique visitors, weekly unique visitors, monthly and so on. A unique visitor is a user that visits your website one or more times within a specified period. Unique visitors can be tracked using authentication or a persistent cookie. A persistent cookie, as opposed to a session cookie (which expires when you close the session or visit), is a cookie that is put onto your computer and expires only after a specified period of time (usually 365 days). Unique visitors can easily be skewed by users deleting their cookies or denying third party cookies (cookies deserve an entire post!). [...]
August 9th, 2008 at 13:19
Good reference post – always looking for well-written, objective discussions of this subject for people I'm working with.
Thank you for the NetInsight mention. One tiny amendment – NetInsight is available as ASP or in-house using 1st, 3rd or no cookies at all (and there are still plenty of folk that want/have to avoid them)
-bob
August 11th, 2008 at 06:52
Another quality post Avinash.
I am a bit puzzled about the cookie deletion rate. If I recall correctly ComScore did a reaserch about cookie deletion a year ago and they found a deletion rate of about 30% on first party and third party cookies. Correct me if the numbers are wrong but according to the study a few users inflated the unique cookie by 2,5 times by deleting very often.
So how do you calculate your 3-5%?
Looking at our site we have looked at daily versus weekly versus monthly cookies to try and see the deletion rate and the 30% seems to be supported.
Also do you have any comment on ComScore study?
Thanks in advance.
August 11th, 2008 at 15:11
Theodor: You might have noticed my own point of view about comScore in my blog post where I mentioned the dearth of objective and sound points of view on cookies. There are a whole bunch of problems with that data, I'll simply refer you to a pretty nice write up from my good friend Ian Thomas from Microsoft:
Cookies are evil! Burn them!
http://www.liesdamnedlies.com/2007/04/cookies_are_evi.html
My own experience is based on analysis on multiple websites over the course of the last few years of two primary sources of site data: javascript tag based and the raw server log data. In each case data gets collected differently, there are common keys (if nothing else then IP and User Agent strings) and both use cookies (common one set by the website platform). This allows detection of uniqueness using couple different algorithms, and not just cookies. At the end, after a bunch of sweat, you have insights into cookie deletion rates.
The data is not based on observing using "monitoring software" what a small panel of users do (even though I realize that the data has good intentions, is adjusted multiple multiple ways to adjust for sample bias and user type bias). It is based on actual data from the website, and all the data.
Let me rush to add that you should not take anyone's number at face value.
I stress repeatedly in the post that if this is important to you then put in the work required to measure it rather than jumping off based on someone else's work (and that includes me! :). Each website and web business is unique, it is possible you are 30% and it is likely that I am 85% and just as likely that pricegrabber is 1%.
-Avinash.
August 13th, 2008 at 09:34
[...] Cookies scare the heck out of some people and the Internet wouldn’t function very well without them. What’s a site owner to do? Here’s a short course in Cookies 301 in Avinash Kaushik’s usual friendly, easy style. [...]
August 20th, 2008 at 04:02
Few comments to add to this debate …
1 – Cookie deletion / blocking
Until last Nov I worked for a UK analytics vendor called RedEye for many years. Back in 2003 RedEye produced a comprehensive report on cookie deletion; go here for more info http://www.redeye.com/bestpractice/white_papers.php. This paper is still highly relevant today and goes in to more detail than pretty much any report written since and is worth a read.
The key to RedEye’s report was being able to look at the behaviour of a set of logged-in users (of transactional sites) and to analyse how many different cookies and IP / user agent strings each visitor had used over varying periods of time.
In summary, cookie deletion and blocking can be rife or negligible, but to what extent largely depends on the nature of the web site and the make-up of its visitors.
For example, consider a site whose best customers access it many times per day, such as a gambling site, verses a site whose best customers only visit once or twice a month, such as a supermarket. The number of cookies per visitor for the gambling site is likely to be much higher for reasons already discussed; such as multiple PCs (I can place a bet from any PC in approx 1 minute), multiple browsers (phone and PC), deletion (do I want people to know that I visited a gambling site?) and so on.
And yes, it’s also possible for each logged in visitor to have multiple logins which really does make ‘100% accurate’ measurement a pipe-dream, but as Avanash says the data is still fantastic for planning and analysis. Tip – steer clear of any vendor who says that their data is 100% accurate – you’d be surprised at how many claim they are.
2 – 3rd party verses 1st party
It’s not just a question of which type of cookie you set as many companies will use both. For example, a brand can set a 1st party cookie on the web pages which it manages and use the same tracking domain to set a cookie on any white labelled content which is hosted by a third party. In this case, cookies set from the white labelled pages will be 3rd party cookies during the same session.
3 – Cookies are no longer just browser cookies
You can also set cookies through Flash. In principle a Flash Cookie has the same basic objective as a browser cookie but as it is set via an embedded object in Flash it’s browser independent.
They’re also tricky to block and delete.
This is good news because Flash Cookies can very easily and significantly improve the ‘accuracy’ of reported visitor figures due to very low deletion and blocking rates. However, it can also be bad news because their use is not flagged up in IE’s Web Page Privacy Policy or through the usual Firefox plug-ins (WASP etc), so any brand using them needs to be even more conscious of best practice data protection and privacy wording. Go to http://www.e-consultancy.com/news-blog/author_25880/paul-cook.html for a view on this and also http://www.adobe.com/products/flashplayer/security/privacy_policy/faq.html
September 9th, 2008 at 09:10
[...]
There are a number of reasons why your tracking will never be a 100% accurate account of what happens to your website in real life. For example, some tracking is dependant on Cookies, and/or JavaScript which are often blocked or deleted (Avinash has a great article on types of Cookies involved in tracking). And tracking using server logs can vastly overestimate your visitors if each image and page element is tracked as a unique hit.
However, most of the large errors and inaccuracies with Web Analytics will be down to the installation rather than the tool itself. So take the time to check your work, using tools mentioned above, and cross referencing with the number of enquiries/purchase actually received.
[...]
December 11th, 2008 at 21:20
Avinash –
Thank you for this article. It is amazingly helpful indeed.
We are currently faced with a similar situation as Nancy described above (comment # 20) We have a collection of branded websites where a visitor can easily move from one to the other. Until recently all websites were using the same domain. We used 1st party delegated solution offered by Coremetrics to track.
Currently, we are redesigning our sites and moving them to different domains starting with the 2 of them. In principle, i would like to use the approach you are suggesting and use 1st party cookies for each domain. The internal concern however is losing paid search mmc parameters information when a user jumps across domains. For example if a visitor comes via Google paid search to domain1 and then to domain2 and purchases, the purchase will be attributed to a domain2 and categorized as a refered visit from domain1. However, internal marketing would want to get full credit for the purchase based on mmc.
Can you please offer an advise on what approach we might take to not lose paid search relevant information and to track our metrics accurately.
Thank you,
Olga
December 12th, 2008 at 02:32
@Olga – I see two options, but it's early and I may miss a trick:
1. Use 3rd-party cookies for a single, other, domain. (urgh!)
2. If you can manage the links that a user may use to move from one site/domain to another, then you could pass the cookie value over in the URL – it's not bulletproof, but can work well.
(there was a third thing as I started writing this response, but it's escaped me for now)
December 29th, 2008 at 16:26
[...]
Jezeli jednak używasz Google Analytics, czyli ciasteczka 1st party, to tutaj znajdziesz bieżącą wartość tego wskaźnika (w chwili pisania tej notki wynosił on ok 1,5%). Ktoś powie, że to bardzo mało — z pewnością, ale nawet ta wielkość utwierdza mnie w tezie, że analityka stron webowych sprowadza się tylko do wyznaczania trendów. Obliczanie konwersji do 2 miejsc po przecinku jest nie na miejscu.
PS. Jeżeli nie wiesz czym różni się 1st part cookie od 3rd part to wytłumaczenie znajdziesz tutaj
[...]
January 19th, 2009 at 21:11
[...] These self-referrals can be caused by a couple of things, but I recently came across a post on Avinash Kaushik’s legendary Occam’s Razor blog where, in the comments below the article, Robbin Steif laid out the most common reasons for self-referrals: [...]
June 21st, 2009 at 20:07
Avinash, I have a very naive question to ask. You say that cookie deletion rates lie in the range of 3% to 5% for first party cookies and 20% to 25% for third party cookies. My understanding is that cookie deletions are in most cases manually performed by the user. The user just deletes all cookies stored on the local drive. This act doesn't distinguish between 1st and 3rd party cookies. This being the case, how could their deletion percentages be so vastly different?
July 3rd, 2009 at 14:38
I am researching Google Analytics for possible use on web site for survivors of domestic-abuse. The site administrator is understandably concerned that cookies set by Analytics in visitors' caches could become potential safety risks for those visitors, whose abusers might detect and be provoked by evidence of a visit to the site. From what I understand in the discussion above, persistent tracking cookies usually show up as a string of numbers, but do they necessarily do so? How can this masking be ensured? Can the Analytics user *choose* what appears in the tracking code? How does one do that?
July 3rd, 2009 at 15:51
Ame: This applies regardless of which web analytics tool you use (Google Analytics or Yahoo! Analytics or Omniture or….)….
Cookies are strings of characters and numbers and for the web analytics tools they are anonymous. For example here is the cookie set by Omniture on my computer to track me:
Key: s_vi_qpasgvx7Bwx7Cqwx7Cwfex7Dx60y
Value: [CS]v4|4A301DEC00004139-A0208310000005B|4A301DEC[CE]
Created: 6/10/2009 9:05:15 PM
Expires: 6/9/2014 9:05:15 PM
You can use this free program to check all the cookies set on your computer:
http://www.karenware.com/powertools/ptcookie.asp
Just clear your cookies, then go to any site, open the Karenware viewer, see what cookies that site is setting.
Some websites do collect PII (personally identifiable data). That is normally not stored in cookies, unless they really don't know what they are doing.
Tools like Google Analytics explicitly prohibit collection of any PII data by its users using Google Analytics.
To answer your other questions:
You cannot choose what string shows up in the cookie set by the Analytics tool (you'll quickly realize allowing you to set your own strings defeats the purpose of creating and setting random string cookies).
You don't have to use a hosted solution like Google Analytics or Omniture or WebTrends. You can use a log based solution. You implement it in-house (at your company / non profit). You own the data. It never goes any where outside your company.
You can find logs based solutions from WebTrends (http://www.webtrends.com), Urchin (http://www.google.com/urchin/index.html) or ClickTracks (http://www.clicktracks.com). They all cost some money, both Urchin and ClickTracks are pretty cheap.
You can find free logs based solutions here:
http://www.analog.cx/
http://awstats.sourceforge.net/
http://www.mrunix.net/webalizer/
All logs based solutions, free or paid, require you or someone at your company to be slightly technical to implement them.
Hope this helps.
Avinash.
July 3rd, 2009 at 16:31
I am not entirely sure if you answered my question–I am less concerned (here) about whether PII is available to outsiders (including but not limited to the website doing the tracking). I am really asking about whether others in the same household as a user would be able to see where those with whom they share a computer have been going online via cookies. For example, in clearing out my TEMP files, I once found cookies for pornography sites on my computer, which tipped me off about what my ex-husband was doing in his free time. Most people do this cache clearing automatically, but a suspicious (me) or controlling (an abuser) person might not. Our question: is there any chance that an abusive spouse would run across a tracking cookie that includes all or part of the abuse shelter's name? We want the info provided by Analytics; we don't want to endanger the people we're trying to help. Your Omniture-cookie example is relevant and reassuring; however, I cannot tell from what you are saying whether it is *necessarily* the case that the code would be anonymous. Are these cookies *always* randomly generated? Sorry for being so thick-headed and long-winded.
July 3rd, 2009 at 20:58
Ame: Let's try this again, this time with additional context from you.
When you open your browser (Firefox, Internet Explorer, Chrome) and visit a website two types of 'tracking" happen.
1. Tracking by the Server (website you went to).
2. Tracking by the Client (the browser on your machine).
In scenario one the website collects typically anonymous data about your visit (pages seen etc) and also sets a anonymous cookie on your computer. This anonymous cookie can't identify you as a person, it can identify the site which set the cookie. When I visit the NY Times a cookie gets set and on my computer it will say nytimes.com (or whatever domain used to set the cookie).
In scenario two (regardless of anything the website is tracking or not) the websites you visit, the pages you see, the url's you type, the forms you filled etc are all logged by your browser. This information is only stored on your computer.
So:
If you are a website owner and want to keep the data tracked by your website private, see my comment above.
If you are a person using a browser to surf the web and wants to keep your behavior private then your best option is to use a new feature in browsers called Private Browsing.
The way it works is you open, say, FireFox (version 3.5 or later), in the top menu you'll see Tools, click on that, then choose Start Private Browsing. This will start a complete private browsing session on your computer, when you close the browser all the history, cookies, cache, your download history (not the download itself) all will be cleared. There will be no trace left on your computer.
I use Google's Chrome browser and in it I click the Wrench icon (top left) and choose New Incognito Window. Same thing as above happens, no trace of my browsing is left on my computer after I close the browser.
I hope this helps.
-Avinash.
July 4th, 2009 at 07:16
Thanks, Avinash!
January 11th, 2010 at 17:31
Hi Avanish,
I've been thoroughly enjoying both your blog and web analytics 2.0, thank you for taking the time to write so well on the subject.
There is one subject around cookies that seems to be rarely touched on but which becomes very important when you have a long user-cycle as we do for our online website creator.
Our users are always logged in but have life cycles of many months. With customers using different machines and taking long periods to build their site our data becomes more and more dirty as their cookies are duplicated or deleted.
We really want the cookie to be set based on our user_id rather than simply the machine.
Looking at the GA cookie it seems possible to store the user's original GA cookie then overwrite the userID portion of it with the original should they log in from a different machine. That way we can be sure that GA is correlating events with visitors consistently rather than creating two different visitors for the user who logs on both at home and at work.
Is this a reasonable approach? Can you recommend any articles that talk about how to fix this problem in either GA or Omniture (which we're also using).
Thank you in advance for your help
- Peter
January 11th, 2010 at 22:41
Peter: The optimal solution for you might be to use the new Custom Variables functionality in Google Analytics:
http://code.google.com/apis/analytics/docs/tracking/gaTrackingCustomVariables.html
I think setting the "scope" at a Visitor level might give you want to you (assuming your users will log in or have some other way of telling you they are the same person). Then you can use the standard reports or, like I do, cheat and just get what you want from the GA API and create your own custom reports.
If you need help then I am sure a GAAC could be happy to do so, a list is at http://bit.ly/gaac
Good luck!
Avinash.
January 12th, 2010 at 13:30
Avanish, thank you for the prompt response. The new custom variables are great and get us 90% of the way there. We're able to do most of what we want to with them but unless we also store the data on our side we don't get the cross-machine tracking.
Let's say a visitor comes to us in month one and signs up through a landing page heavily promoting a particular feature. We store that they were shown the promotion in a visitor-scoped custom variable
In month three they then purchase the feature but do so from their work machine rather than the one they signed up on.
Because of the machine switch we wouldn't be able to correlate the two events and the efficacy of the early promotion.
That said, we haven't tested how much of an edge case this is. I think we need to do that first and If it does prove to be we will look up a GAAC.
Thank you again for the input.
- Peter
January 12th, 2010 at 15:33
Peter: Ahh…. slight misunderstanding on my part. My apologies.
You can't do that. Not because it is not doable via the solution I had shared but because it would be a violation of Google's terms and conditions.
But if you really want to do this then now you know how you could do this with a different web analytics tool.
-Avinash.
January 20th, 2010 at 05:33
[...]
It’s unclear how the two metrics are affected if users block the setting of cookies. I tried looking online but couldn’t find a definite answer but intuition would lead me to believe that a user who blocks cookies would always be counted as a unique visitor (and thus a visitor). You would lose tracking of their behavior, since there’s no session cookie to tell Analytics where they’ve been, but you’d have correct-ish visit counts. Fortunately the number of people rejecting cookies is fairly small.
Here is a detailed discussion of Cookies and Google Analytics, which should answer pretty much any question you could possibly have, and a fantastic primer on cookies and tracking overall.
[...]
January 24th, 2010 at 22:19
[...]
Comme plusieurs d’entre vous, je n’ai aucun problème à ce qu’une organisation ramasse mes informations avec des cookies. Il est vrai que ça améliore ma relation avec les entreprises électroniques sur le Web.
Du point de vue de l’utilisation, j’aime bien pouvoir conserver les articles de mon panier d’une session à l’autre, quand je magasine sur Amazon, Futureshop ou bien Bestbuy. De la même façon, j’utilise beaucoup de services Web en ligne comme Gmail, google calendar, google analytics, basecamp ou twitter et j’apprécie ne pas être obligé de m’identifier à chaque fois pour accéder à mes données.
[...]
May 11th, 2010 at 06:35
[...]
Podéis leer más información sobre las cookies en la página de Google Analytics o en Occam’s Razor.
En los siguientes posts seguiré hablando de las bases de Google Analytics y entraremos más en detalle sobre cada cookie.
[...]
February 2nd, 2011 at 14:16
[...]
Por último, no quisiera acabar sin recomendar un post de Avinash Kaushik sobre las cookies:
http://www.kaushik.net/avinash/2008/07/web-analytics-visitor-tracking-cookies.html
[...]