We got feedback from usability experts and they have additional information they would like to share. Take a look at their new post.
A common practice in usability testing is to ask people to describe what they are doing as they complete a task. The “Think-Aloud” protocol, or method, asks test participants to verbally express their thoughts, feelings, and reactions to a website. By asking participants, “Was that task easy to perform?” or “Tell me what you’re thinking,” usability professionals gain valuable information about how users experience websites.
In a recent DigitalGov University webinar, Usability experts Betty Murphy (Human Solutions, Inc.) and Erica Olmsted-Hawala (U.S. Census Bureau) explained the theory behind the Think-Aloud method and the different ways it may be applied in usability testing. The options include:
- Traditional and Speech-communication: These two methods require that the test administrator say as little as possible. Verbal cues are limited to “Keep talking.” “OK,” “Mm-hmm?”, or “Uh-huh?”
- Coaching: The test administrator can ask the participant for feedback and actively intervene with probes, such as “Did you notice that link up here?”, “You’re doing great,” or “Can you explain why you clicked on that link?”
Slide showing the results of the 4 different conditions plotted against accuracy.
Far and away the most common practice among usability practitioners is to alternate between methods without distinguishing between them. But this may not be the best practice.Murphy and Olmsted-Hawala’s study reveals that using more than one Think-Aloud method during a test can cause problems – not all Think-Aloud methods are created equal, and some researchers value certain kinds more highly than others. But when you mix up the results, you can’t differentiate between which types were used.
In addition, people who received the “coaching” Think-Aloud protocol seemed to do exceptionally well on their tasks – maybe too well. The study revealed that testers did better when they were being coached by a test administrator then they would actually do in real life. In other words, their accuracy rates and satisfaction with the website were slightly exaggerated and therefore somewhat misleading. The coaching method did not, however, affect the amount of time it took participants to complete their tasks.
That doesn’t mean that the coaching method should be tossed out. Each method can provide good results in different ways:
Pros of the coaching method: By allowing the test administrator to ask participants detailed questions about what they are thinking, the coaching method extracts useful opinions, rich anecdotes, and fosters an active dialog between the administrator and the participant. This creates a lively viewing atmosphere and good qualitative data.
Pros of the traditional methods: As the study shows, more traditional Think-Aloud methods will result in more accurate data for how users will complete tasks on their own, or “in the field,” because the test administrator is very quiet and plays a passive role in the test.
While the “coaching” Think-Aloud method may still be worth using because of the reasons listed above, Murphy and Olmsted-Hawala recommend that usability practitioners make the following changes based on the findings of their study:
- Only use traditional or speech-communication methods for evidence of how users will complete tasks “in the field”
- Accurately report and document the type of Think-Aloud used for a usability test
- Develop and follow Think-Aloud standards across the usability profession
For more information, watch the webinar on-demand at howto.gov. Be sure to check back in for cutting-edge usability and user-experience research!
Loading tweets...
3 comments
1 ping
Wm Oms says:
February 11, 2013 at 8:57 am (UTC -5 )
Thank you for sharing excellent informations. Your website is very cool. I am impressed by the details that you have on this website. It reveals how nicely you understand this subject. Bookmarked this web page, will come back for more articles. You, my friend, ROCK! I found just the info I already searched all over the place and simply could not come across. What a perfect site.
Betty Murphy says:
February 5, 2013 at 3:19 pm (UTC -5 )
Molly, as one of the presenters, I need to point out that your summary misrepresents a good many of the points that Erica and I were making. We are developing a new summary, and I hope you will take down the current posting before too many people get the wrong impression about our webinar.
Molly says:
February 13, 2013 at 11:40 am (UTC -5 )
Betty, thank you so much for pointing out ways in which we can make this blog post better! We are fully dedicated to disseminating accurate information, and we would like to present your research in a format that you are comfortable with. We are anxious to receive your input so that we can edit the post accordingly.
What we’re reading | Webtools says:
February 4, 2013 at 2:26 pm (UTC -5 )
[...] “Think-Aloud” Protocol Can Influence Usability Testing [...]