August 2010

23 Aug 2010 02:35 am

Fresh Every believer in Web Analytics 2.0 knows that awesomeness comes not from answering just the "What" question but from also answering the "Why" question.

What comes from Google Analytics, Adobe Site Catalyst, WebTrends, CoreInsight / NetMetrics and more.

Why comes from lab usability studies, website surveys, "follow me home" exercises, experimentation & testing, and other such delightful endeavors.

Why gives context to the What, and delightfully helps you not have to overlay your biases when you try to infer visitor intent form all the What (clickstream) data.

I know that you agree Why is important.

I know that you even realize Why is ever easier to accomplish (usability studies are economical, surveys and testing platforms start at the sweet price of free!).

Yet your site stinks like a skunk.

The reasons are complicated.

You are smart, so that is not it. Maybe it is internal politics. Maybe it is the agency you have outsourced the site to, the agency whose only competence seems to be gratuitous use of flash. Maybe it is that it is not your job, you are the "quant" guy or "GA girl". Maybe even after taking one of the team and going out on three dates the IT Dude still refuses to put Website Optimizer tags on the site. Maybe the well meaning but "never met our real customers" HiPPO dictates site design.

Bottom-line: Your site stinks and you need to fix it.

Allow me to introduce you a User Centric Design that is, I think, the solution you have been waiting for: Heuristic Evaluations

I love heuristic evaluations because they are cheap, fast and you probably already have resources you need in your company. A large part of my adoration also comes from the fact that heuristic evaluations are us going back to the basics in an attempt to create un-stinky websites.

doctor stethoscope

What Are Heuristic Evaluations?

A heuristic is a rule of thumb. In as much, heuristic evaluations follow a set of well established rules (best practices) in web design and how website visitors experience websites and interact with them.

When conducting a heuristic evaluation a user researcher (or an HCI expert) acts as a website customer and attempts to complete a set of predetermined tasks related to a website's existence. For example: Trying to place and order, or looking to find out the status of an order, or the solution to an error code, or decide which of many products on a site are optimal for a specific customer persona.

But here is the lovely part, and why almost anyone can perform heuristic evaluations. in addition to best practices the researcher (or you!) will raw from their own experience, knowledge and common sense.

Heuristic evaluations are best when they are used to identify what parts of the customer experience are most broken on your website. They can be very beneficial if you have not yet conducted any usability tests or when you would like to have a quick review of prototypes that the designers might be considering.

In either case, you can quickly determine the lowest hanging fruits in terms of "broken" parts of the customer experience. With this feedback there can be iterative improvements to the customer experience. You'll probably already have connected the dots and realized that this is a fantastic way to identify ideas for A/B or Multivariate experiments on the live website.

group of people solution

[I Heart] Group Heuristic Evaluations

There is one more thing, a way to amplify the impact and get even better results.

Get everyone involved!

Heuristic evaluations can also be done in groups!!

Invite people around you with key skills, such a designers, information architects, web analytics professionals, that girl in accounting you really like, other analysts in the company (and their quantitative understanding of site data), search experts, the intern who only communicates via Posterous, and so on and so forth.

The goal is simple: Identify flaws by attempting to mimic the customer experience (if possible under the stewardship of a User Researcher, if not then under the gaze of your haunting brown eyes) by completing the tasks on the website as a custom.

The great benefit of the group heuristic evaluation method is that you can tap into the "wisdom of crowds". This is especially powerful because the web is such an intensely personal medium, and the group members can offer different points of view that highlight issues quickly.

The process that worked optimally for me was to send an email to 10 or so folks (a diverse set!). Invite them to a noon meeting in a largish conference room and order lunch for them (best $50 I ever spent). Once everyone was settled (by 1205!) project the website on the screen and try to complete the most common customer tasks.

I (or you) have to do a good job of moderating the discussion and ensure everyone participates. There is no such thing as a bad opinion, diversity is good. Collect all pertinent feedback.

Heuristic evaluations can provide valuable feedback at a low cost ($50 in my case) in a very short amount of time (an hour in my case) and identify obvious usability problems. Hence they are best for optimizing work flows, improving user interface design and for understanding the overall stinkiness (or lack thereof) of the website.

Here is another subtle benefit of the group evaluations: improved communication and, dare I say, camaraderie between different groups in your company (big or small).

There is nothing that quite brings people together like a,  pardon the expression, bitchfest. Everyone contributes, everyone commiserates, everyone loves it. Next time they are doing something they'll know what you do. Next time you need help, you'll know who to call (and they'll pick up the phone!). It is a great way to bring a sense of common purpose and a sense of ownership to improving the website experience.

business plan steps success

Conducting A Heuristic Evaluation (The Glorious Process!).

You're excited right?

Here are six steps to conducting a successful evaluation process, either when you do it or you are doing it as a group:

1. Write down the tasks that customers are expected to complete on the website. If you are using surveys on your site (even a simple site level survey like 4Q from iPerceptions or page level survey like Kissinsights) then that is a fantastic source of this information. You should also, if possible, talk to the site owner. Here is what you might end up with on your list:

  • Find information about the top-selling product on the website.

  • Locate a store closest to where the customer lives.

  • Place an order on the website by using PayPal. (If the website doesn't accept PayPal, how easily and quickly can a customer find that out?)

  • Sign up to show up at a protest march against taxes on the richest Americans.

  • Successfully contact tech support via email.

  • Pick the right product for customer profile x (where x can be a small business owner or a family of four or someone who is allergic to peanuts).

2. This is sometimes hard but try to establish success benchmarks for each task. For example success rate for placing an order for the best selling product = 80%, signing up for the protest march = 99%, contact tech support = 90%, etc etc.

3. The fun part. Walk through each task as a customer would and make note of the key findings from the experience – everything from how long it takes to complete the task to how many steps it takes to the hurdles in accomplishing the tasks to how profound your embarrassment was that this was your own company's website.

4. If you were using a best practices checklist (more on this below) then make a note of the specific rule violations.

5. The hard part. Create a report of the findings. You can use PowerPoint with a screenshot of the webpage and clear call-outs for issues found. Or you can use Camtesia / a screen recording software to capture the session (and the group discussion). This can be distilled to a "the best of the bitchfest" collection for your superiors.

6. The hardest part. Categorize the recommendations into Urgent, Important and Nice to Have. We all get swept into emotions fervor. It is also possible that when you present your findings to your Sr. Management they might be a bit HiPPOish (not that there's anything wrong with that).

You want to go in with the Urgent, Important and Nice to Have based on impact on the customer experience and the company bottom-line. This helps drive a "what we should do" discussion rather than "I think we should do that" discussion.

That's it.

The process, as always in web analytics, is important and I hope the above six steps help you create a process in your company that is repeatable and yields impactful results.

I want to stress again that this is a great way for you to get into the customer's shoes, for your to build camaraderie, involve the cross-functional team of people, and finally find the lowest hanging fruit for sure and perhaps even some big ones.

25 point usability checklist

The Cheapest Heuristic Evaluation Exercise.

I have hinted about the cheapest possible heuristic evaluation exercise a couple of times in this post.

It is: You sitting down with your common sense and a list of "best practices" and checking how well, or badly, your website does against that list. Then do steps 5 & 6 above.

This is fast and impactful. Even in the worst case you identify the most broken things / annoyances.

I have used lots of website best practices usability checklists over time and have settled on using Dr. Peter Meyers's 25-point checklist. It is simple, effective and quite expansive. You can download it here. [If you' have Web Analytics 2.0 then you'll also find an "extended edition" on the CD that is attached to the back cover of the book.]

The usability checklist has four sections. Here's a brief summary:


    1. Site Load-time Is Reasonable
    2. Adequate Text-to-Background Contrast
    3. Font Size/Spacing Is Easy to Read
    4. Flash & Add-ons Are Used Sparingly
    5. Images Have Appropriate ALT Tags
    6. Site Has Custom Not-found/404 Page


    7. Company Logo Is Prominently Placed
    8. Tagline Makes Company's Purpose Clear
    9. Home-page Is Digestible In 5 Seconds
    10. Clear Path to Company Information
    11. Clear Path to Contact Information


    12. Main Navigation Is Easily Identifiable
    13. Navigation Labels Are Clear & Concise
    14. Number of Buttons/Links Is Reasonable
    15. Company Logo Is Linked to Home-page
    16. Links Are Consistent & Easy to Identify
    17. Site Search Is Easy to Access


    18. Major Headings Are Clear & Descriptive
    19. Critical Content Is Above The Fold
    20. Styles & Colors Are Consistent
    21. Emphasis (bold, etc.) Is Used Sparingly
    22. Ads & Pop-ups Are Unobtrusive
    23. Main Copy Is Concise & Explanatory
    24. URLs Are Meaningful & User-friendly
    25. HTML Page Titles Are Explanatory

Seems simple right?

I bet your website currently breaks 10 of the rules above. It is hard to believe. Set up a quite hour aside. Go through the checklist. But first go download the detailed checklist at Dr. Pete's website.

When you are done remember to do steps 5 and 6 of the recommended heuristic evaluation process outlined above.

 erase errors

Benefits of Heuristic Evaluations.

In case you somehow made it here and were not convinced of the value doing heuristic evaluations here is a quick summary of the benefits:

  • Heuristic evaluations are extremely fast to perform, with a very short time to insights.

  • They leverage existing resources in your company (what could be awesomer?).

  • You'll identify the most egregious customer issues on your website (often all the low and medium hanging fruit, quickly).

  • They can be used very effectively early in the website development process to find potential customer hurdles / deal breakers.

  • If you have an existing UCD program or hire an external company/agency. heuristic evaluations can reduce the cost of full usability tests by helping fix the obvious problems. The $900 an hour charged by the Agency can then be focused on hidden / really tough challenges.

Thing to Watch For.

It should be clear that I am a fan of the heuristic evaluation process. And it is every fan's duty to also highlight things to watch out for. Here they are:

  • Both single person led or group evaluations contain company employees, and sometimes usability experts, but none of them are the actual customers. Despite using best practices and our wisdom we might miss some subtle problems, even as we identify the obvious ones. Be very aware of this.

  • [It follows from the above point that.]  The better you are at step #1 outlined in the heuristic evaluation process, the better your outcomes will be.

  • When there is disagreement in recommendations from the heuristic evaluations there can be great value by doing live website tests or usability studies (whatever is faster, usually experiments).

  • Heuristic evaluations are best for optimizing work flows, larger more obvious parts of website design and the overall usability of the website.

In summary: Not quite God's gift to humanity, but rather the best thing you could do to identify the low and medium fixes to your site that will significantly improve the experience of your customers.

Don't spend your day immersed in Google Analytics and just the "What" analysis. Understanding "Why" is the key, use it to unlock actionable insights.

Ok your turn now.

Have you done heuristic evaluations for your website? Who leads them in your company? Do you have a list of usability best practices that you use on your website? What other methods of of listening / collecting voice of customer / answering "Why" do you use in your company?

Please share your feedback / critique / questions / answers via comments.


Couple other related posts you might find interesting:

09 Aug 2010 01:23 am

One Focus The hardest nut to crack in any type of analytics is getting our decision makers (bosses, leaders, marketers) to take action based on data.

The hard nut is not that we all are doing basic reporting about Visits and Bounces. Ok doing just that is lame. But still that's not all of it.

It might shock you that the hard nut is not even that we, the people who "play" with data, the Analysis Ninjas if you will, are not leveraging custom reports and advanced segmentation and activity alerts and all that cool stuff.

It is that no one wants to take action on our data. Ok not no one. But most of the time no one takes action based on our hard work.


To illustrate perhaps the most obvious, and yet hidden, problem I wanted to share with you a recent experience I had with data. It was not in the context of web analytics and yet I think it presents a solution to our problem.

On my (beloved) Nexus One android phone one of my absolute favorite applications is Cardio Trainer. I just love it. The UI and the UX and how it is so darn easy to use (and much more well thought than competitors like My Tracks). You must get it.

I use it when I do any kind of exercise (or hike with the kids). You just open the app, say Start Workout, choose the type of work out (running, biking, elliptical, horseback riding, and 15 other choices) and you are up and running.

It starts tracking you (GPS and what not), pauses when you pause, and if you choose even talks to you (motivation!).

When the workout is done it gives you. . . . data!

cardiotrainer workout data

Awesome right? What analyst / data person would not love this.

There is a sweet map (I can switch to the satellite view and see the buildings I went around!) and there is lots of data.

Time: 0:22:07  Distance:  6.74 kilometers Speed: 18.3 km/hr Calories: 207

It is a great dashboard.

For the first few times I used CardioTrainer I loved seeing the data (almost more than the fast bike ride). I would switch the map to the satellite view and drill down into the woods and the lake and the trees.

Of course I would look ay the data and maybe remember fragments.

Remind you of something?

Perhaps your web analytics dashboard. Or perhaps your "data summary report". Or, hopefully not, the data puke that goes to the entire management team.

Initially everyone's happy, nothing much happens, yet it feels good.

In my case pretty soon I switched to another view of the data, this one. . . .

workout data summary cardio trainer

Shorter summary. No gimmicks like the map. : ) A bit more data.

I can see Total Climb (elevation): 16 m. I can also see the min and max for Speed (max 25.1, min 7.5). Pretty good right?

This view felt better. It seemed like less data.  Bit easier to remember the Calories.

I am embarrassed to admit that having two more speed numbers (min and max) made me forget the speed even faster, and that's a number I love to remember because I want to constantly beat it.

More data did not make life any more data driven.

Then the other day I noticed that CardioTrainer was auto-updated over the air (love this Android feature!).

When I was done with the workout I was astounded to see this when I hit the End Workout button. . . .

cardio trainer new dashboard

All my beloved data was gone and it was replaced by this simple message:

2 Pears

OMG! Awesome!

Here is what was so cool (and not it was not just the Great Job! line):

    1. All that data I had before had meant nothing. I remembered little. There was not much I changed or actioned based on it. Having data felt good. Did nothing. [Remind you of your Visits, Visitors, Page Views and Bounce Rate reports?]

    With the new "dashboard" I realized I needed to ride more / harder instantly. Two pears was simply unacceptable!

    [I am up to four pears, in just one week! ]

    2. This one is key.

    The CardioTrainer team took time to truly understand why I, and many others, workout and translated all that data into a clearly relatable and understandable "bottom-line outcome".

    I exercise to stay fit (and not become overweight!). Calories and elevation and speed and distance are all great. But they are two steps removed from the outcome.

    All those numbers don't matter. It is hard for the "HiPPO" (me!) to translate that data to understand what was the impact / outcome. Burning more calories seems obvious, but how much? And what did 206 calories mean?

    Not any more. Now I can take those pears and tie them to how much I had consumed all day (vegan healthy food!), make a decision and take action.

I had all that data before, detailed data, and yet I hardly new what it meant. Now it was crystal clear: I had burned off an equivalent of 2 pears. I needed to change.

When you think of your day job as an Analyst. . . are you simply communicating data or do your reports have the equivalent of two pears?

For most of us the answer is no. No pears.

We love and provide data.

Not surprising that getting leaders to make decision is such a hard nut to crack right?

peeled bulbpear unwrapped

Here are two lessons I took away from this experience that are relevant to our efforts to create a data driven experience.

The "2 Pears" Web Analytics Action Driving Philosophy:

1. Simplify & focus.

    Become a ruthless killing machine when it comes to metrics/data. Especially if the audience is our beloved HiPPO's or Sr. Management.

    Our job is not to impress them with data or show them how clever we are with our segmentation strategy. Our job is to communicate, convince, drive action. CCDR. If you need a new acronym.

    Ask yourself why every piece of data is on the report / dashboard / presentation. If it is not "instantly useful" kill it. And enjoy the kill.

    Here's how: Kill Useless Web Metrics: Apply The "Three Layers Of So What" Test

2. Truly madly deeply understand the desired outcome.

    The glorious thing was not just removing the data, it was the profound tie between the activity (exercise) and the outcome (more pears!).

    In our case activity = website and outcomes = what?

    That is the secret magical awesome question that needs answering.

    It is not page views and bounce rates and time on site and browser versions. Surprisingly it is not even conversion rates or goals met. Very often all these metrics fall in activity.

    What is the point of Videos? Downloads? NO!

    What is the point of Ecommerce? Surprisingly.. NO!

    What is the point of Page views? Not really.

    What is the point of Current disasters listing? NO!

    As an analyst for each site my job is to figure out what the "2 pears" are. I mean really figure it out. Not look for proxies. Pears! Talk to people, beg your management, be willing to go out on dates and take one for the team.

    What is the one thing that matters the most, the reason why everything else happens, the singular reason all that data exists?

    Then focus just on the pears on my Sr. Management reporting / presentations.

    What was the change in "pears"? What we should do more? What should we do less?

    End of the story.

    Here's how: Win With Web Metrics: Ensure A Clear Line Of Sight To Net Income!

That's it.

Two simple lessons that if applied well will begin to move you towards communicating your brilliance the way it was meant to be communicated, and shift your organization to being more data driven.

Now every single time you look at your dashboard / performance summary / reports ask yourself this simple question:

Got Pears?

Ok it's your turn now.

Is the reporting you do crystal clear on the "pears"? If not why not? Have you seen other examples in the wild of insights presentation that have made you go "ah ha!!"? Care to share tips you have used to communicate effectively with your data audience? What is your company's secret to taking action based on data?

Please share your thoughts / critique / tips / screenshots / experience via comments.

cardio trainer qr codeOh and before I forget. . .

If you are on the Android platform go get CardioTrainer now!

Simply scan the QR code on the right with the Barcode Scanner app and before you say abracadabra you'll be on your way to happier data driven workouts!

Couple other related posts you might find interesting: