Overview & Importance of Qualitative Metrics
Imagine walking into and out of a supermarket. If you did not purchase anything then the supermarket managers probably don't even know you were there. If you purchased something, the supermarket knows something was sold (they know a bit more if you use a membership card).
Visiting a website, you leave behind a significant amount of data, whether you buy something or not. The website knows every "aisle" you walked down, everything you touched and everything you put in your cart and then discarded. If you buy, the site manager knows where you live, where you came to the website from, which promotion you are responding to, how many times you have bought before and so on. If you simply visited and left the website, it still knows everything you did and in the exact order you did it in.
Add to this the fact that now there is a massive proliferation of tools that will instantly create reports presenting data in every conceivable slice, graph, table, pivot or dump that you can imagine the challenge.
But, no matter what tool you use, the best that all this data will help you understand is What happened. It cannot, no matter how much you torture the data, tell you Why something happened. This is the reason qualitative data is so hyper important. It is the difference between 99% of the website analysis that happens that yields very little insights and the 1% that provides a window into the mind of a customer.
Combining the What (quantitative) with the Why (qualitative) will provide a company with a long term strategic competitive advantage.
There are many types of qualitative data at your disposal including brand buzz, customer satisfaction, net promoter indices, visitor engagement, stickiness, blog-pulse, etc.
While there are many options for qualitative analysis, perhaps the most important qualitative data point is how Customers/Visitors interact with your “web presence.� Visitor interaction can lead to actionable insights faster while having a richer impact on your decision making. There is a lot of "buzz" around "buzzy" metrics such as brand value / brand impact, blog-pulse, to name a couple. IMHO these "buzzy" metrics might be a sub optimal use of time/resources if we don't first have a hard core understanding of customer satisfaction and task completion on our websites.
There are many different methodologies to collect Customer qualitative data, including:
- Lab Usability Testing (inviting participants to complete tasks, guided or unguided)
- Follow Me Homes (observing in a customer's "native" environment)
- Experimentation/Testing (the latest new and cool thing to do, a/b or multivariate)
- Surveying (the grand daddy of them all)
If you are new to this world the last one is a great way to step into this new world and unlike what you might have heard it is both easy to implement, can be a continuous methodology, highly quantitative and is most often chock full of insights that will lend themselves to be very action oriented.
In future posts I hope to dive deeper into options, best practices and examples.
Agree? Disagree? Please share your feedback via comments.
Subscribe 
May 15th, 2006 at 12:00
Welcome to the blogosphere Avinash, I love reading your thoughts.
You're correct, web analytics is only "one" slice of how to measure the full customer or user experience. Motivation, needs, usablity, analytics, and other measurables must be taken into account.
Aside from the excellent other methods you listed, you could add 'opinion tracking' on a website.
In my experience in conducting or managing UX research The granddaddy in my opinion is not surveys –its the simple question "what do you think?"
The casual in person interview can glean the most amount of information that other methods can't track or measure –this method is too often overlooked.
Other methods:
Eye tracking
http://jeremiahthewebprophet.blogspot.com/2006/01/eyetracking-collective-intelligence-20.html
Opinion Tracking (during site session)
http://jeremiahthewebprophet.blogspot.com/2005/09/measuring-online-user-opinion-during.html
May 15th, 2006 at 14:06
I think you've nailed the problem – it's not the "What" it's the "Why".
What I find fascinating about reading your post is the following, I can enter the (web) store and they can track everything I do, they can even ask me to fill in forms and leave some cookies on my machine… and yet, if I clear my cache and delete my cookies when I go back to the "store" they have no idea it's ME again.
This is even worse on a mobile device where I have to re-enter all my data again. Why can't "it" know it's Me and what I like. Where is the ability for the web server to "talk" to me (my device) and know more about the things (preferences) that I like?
If permission based marketing is the future then it's clear that we must understand the "why" not the what. To do that the web and it's stores have to learn that it's Me calling and what I like.
All the best,
Peter
May 15th, 2006 at 17:34
Jeremiah: Thanks for the feedback…
Perhaps I should have been more explicit, there are many other options to get the VOC (voice of customer) to the table including the two you mention and certainly there is a hierarchy from asking my mom what she thinks of the site to Intel chips inside your brains transmitting via rfid’s exactly what our customers think of our websites (or products etc). Most of us are drowning in data and unsure of how to hear the screaming customer. Starting with simple surveys is a step towards evolving towards some of the more advanced techniques.
More importantly all the advanced techniques in the world will not add any value if you don’t understand what your core customer segments are and what the needs and wants are of each of those segments. The web is such a complex environment that anything done on a sample of six or eight or ten or twenty is rarely representative of the “real world” interacting with our websites.
Challenges galore!
May 15th, 2006 at 17:45
Peter: Great points, couple thoughts….
As a guy whose job it is to track people I have to admit this is really a tough one. For websites to know it is You in today’s environment one of two things need to happen, #1 either the website puts something on your system that identifies you (cookie) or #2 it “reads” something that can help it figure out it is You. With both Privacy is the challenge.
#1 There is enormous hype the big brother is watching and so anonymous cookies (or cookies with your login id for example if you use My Yahoo or My Google etc) is the only way to go. If you blow cookies away the site does not know who you are, simply because it is trying to respect your privacy.
#2 This is clearly spyware. The website can install something, or read your outlook email address or your Windows login and know every time it is you. But I don’t think that is preferred by many people.
I was talking to a big seller of ringtones in the US and one problem with Mobile tracking (not just that most mobile browsers don’t accept javascript or cookies) is that the Telco is not willing to share information. For example if only they could pass the first six digits of your phone number back these guys could know better it is you. In Japan I just read of a mobile game seller sharing a cut of profits with the Telco in exchange for tracking. Maybe that is a model that will work.
If that happens in the US every time you use your mobile browser it will know it is you and it will save all your settings (hopefully with some kind of opt in method). Hooray!
May 16th, 2006 at 09:54
Avinash,
Great to see you blogging.
One point I'd love to see you elaborate on: A/B and multivariate testing are very hot, but they all too commonly get to the what and not the why. I'd actually wager few marketers take the time to draw deeper conclusions from this, even when making a significant investment of time (if not money) in the testing.
Can't wait to read more. I'll feature your blog on mine shortly.
May 17th, 2006 at 08:44
Hi. My name is Joel Slavis.
I represnt a small lighting company
We will need some tool to grow our business we represent many manufacturers from China and need to move products to keep them and us in momentum.
My IT people use google and the other stuff but it is not seeming to work…
Any info or help on the three
* Webside Story
* CoreMetrics
* OmniTure
or any other recommendations might be helpful
May 17th, 2006 at 09:46
Joel: Thanks for your post….
I recommend you first read this page on my blog: Disclaimers – Disclosures to ensure you are aware of any bias that might be inserted into what I say below.
It is very difficult to make a specific recommendation without understanding your core needs and the kinds of decisions you want to make. But let me make this generic recommendation:
These are some generic thoughts, please email me if you want to know more and we can see what options are there to share knowledge.Welcome to the world of Web Insights. Good Luck!
May 29th, 2006 at 19:55
[...] http://www.kaushik.net/avinash/2006/05/overview-importance-of-qualitative-metrics.html [...]
September 17th, 2008 at 11:15
It becomes even more challenging to consider qualitative metrics when much of our thoughts are in the subconscious (the figure of 95% is offered by Gerald Zaltman in his book How Customers Think).
As this blog demonstrates, effective web analytics (leading to insights that help reach/better objectives) is no easy matter, especially when we are dealing with the human mind…
September 17th, 2008 at 11:28
To follow up an give a real world example on my post, consider online banking (the area I work in). A major barrier to the uptake to online services (logging in, checking your balances, transferring money, making payments, buying products) is security fears.
It's possible to build the world's most secure website, however, this will be no good if it's not usable.
You can achieve a trade-off between actual security and usability, however again, it's no good if the user is still scared.
Web analytics on their own make it hard to track the emotions involved, as we are usually talking months to years until somone first starts to use the internet, until they are comfortable performing financial transactions online. There are of course tactics to speed this along, although without offline research it's hard to gain this insight to make your web experience better…
January 6th, 2009 at 03:05
[...] Answering the question is critical, since analyzing what your visitors did online (Clickstream analysis – clicking on the banner, clicking in links on your site), does not directly relate to the results brand advertisers are targeting (often brand awareness and brand preference). Many of us have tried more complex models like engagement modeling or combining quantitative analytics data with qualitative metrics to come closer to understanding the branding impact of a campaign. The challenge remains unanswered neither approach is able to tell you what effect the a branding campaign perceptions and attitudes, since both lack effective control groups. [...]
July 29th, 2009 at 10:14
[...]
Optimize your web presence. It goes without saying that your website should be quantitatively optimized to increase traffic and drive conversions. Less obvious, but equally important, is the need to understand what your traffic really wants, and deliver it to them. For example, you may find yourself on a plateau of a 3 percent conversion rate, and extrapolate how much revenue you can drive by increasing that to 4 percent. But this implicitly assumes all traffic wants to convert. What if a survey of your traffic found 20% was on the verge of buying, and looking for information to help with negotiations? You could create pages touting the high price of ownership of your competition, thus driving down their profits. What if a survey found 10% were looking for information to give to their bosses? You could create collateral specifically focused around this segment. The point is, you probably don’t know this today, because you don’t implement surveys. Fortunately, like they are fast, friendly, and free.
[...]
April 21st, 2010 at 07:33
[...]
Let’s say your homepage was totally redesigned and the conversion rate for its primary conversion goal dropped by 2%. Each person that has any say in the company probably has a different opinion about what the problem may be. “The picture should be different” or “The headline should be different” or “It should be my picture there instead of the CEO’s” or whatever. Well, who’s right? Who knows.
The good news is, there are two ways to gain truly accurate information about why something is happening on a website that most companies simply are not taking advantage of … surveys and testing.
[...]
April 26th, 2011 at 05:11
[...]
From a personal point of view, I am constantly amazed at how in general customer service and call centres seem to be way down the list of important touch points with a consumer. Brands are constantly trying to understand the ‘Why‘ when it comes to customer interaction, and no doubt many spend countless hours and resources trying to do it, when there is a great opportunity and wealth of data available to them if only they choose to tap into it. As the great man, Avinash Kaushik explains, ‘the greatest nugget of insight is the Voice of the Customer’.
[...]