Website Usability Testing
These are the basics on how to collect customer feedback via usability testing, which is a technique for evaluating a website, application, system, form, or product by observing representative user interaction and ease of task completion. Feedback from these tests can improve design and performance. While the FAQs below specifically focus on website usability testing, the principles apply to many other types of usability testing as well.
Basics
- What can you learn from usability testing?
- What kind of feedback can you expect, and how can you use it?
- What are the limitations of usability testing?
- When should you run a usability test?
Participants
- How do you identify potential usability test participants?
- How many people should participate in the test?
- From whom should I get buy-in prior to conducting the test?
Test design
- What is the format of a usability test?
- How do you choose which tasks to include in the test?
- How frequently should you conduct usability testing?
- Who at your agency should observe a usability test?
- Should you test your usability test questions/tasks?
- Should you collect personally identifiable information?
- What kind of follow-up work is required after a usability test?
Operational details
- Are there technologies to help with usability testing?
- Will you need to get approval under the Paperwork Reduction Act (PRA)?
- What resources do you need to run a usability test?
- What are the major cost considerations?
Basics
1. What can you learn from usability testing?
Usability tests can reveal problems with website design and functionality. Specifically, usability tests can uncover navigation issues, determine content relevance, highlight overall design effectiveness, and assess customers’ ability to complete critical tasks. In addition, usability testing can provide feedback about the overall customer experience with the website. Examples of some questions that could be answered through usability testing include:
a. Are sections of the website difficult or confusing to navigate?
b. What steps do customers take when trying to complete a transaction?
c. What are the major stumbling blocks when looking for a specific piece of information?
d. How easy is it for your customers to use your website?
2. What kind of feedback can you expect, and how can you use it?
In general, feedback falls into two major categories. Certain tasks can reveal quantitative usability metrics, such as the amount of time or number of clicks it takes to complete a given task. Other tasks can reveal more qualitative data, such as a participant’s perceived ease-of-use for a given web page.
The feedback from web usability tests should be incorporated into improvements in your website design and functionality, and can provide insight into how customers seek and access information. Because ease of access to information is a critical component of customer service, improvements in this area can help improve the overall customer experience.
3. What are the limitations of usability testing?
Due to the very small sample size and because participation is not random, feedback will not be representative of your entire customer base, nor will any data gleaned be statistically significant. However, important insights can be gotten by leveraging usability tests.
4. When should you run a usability test?
Because these tests identify issues with web site design and performance, these tests can be run anytime in the website lifecycle. They are particularly useful early on in the design process for catching major flaws, or whenever changes are being made.
Participants
1. How do you identify potential usability test participants?
Selection criteria should be informed by a) the purpose of the website, b) the overall goals of your test, and c) by the characteristics that make different types of users behave differently. For example, if you’re trying to assess how hard it is for any user to find key information on your website, you may want to recruit participants with both basic and advanced web knowledge. Unless you are specifically testing returning customers, it is helpful to recruit individuals who are unfamiliar with your website in order to avoid biases.
A common way to identify potential participants is to look internally to your organization, or to use an internal customer list (as allowed by privacy laws). You can also purchase a list of potential participants from a third party, or use a professional recruiter. When looking for potential participants, keep in mind your selection criteria, since users might need to have a specific skill set (e.g., first-time users), not just a specific demographic background.
Also, the General Services Administration runs “First Fridays” usability testing that you may be able to participate in. In a 4-hour period on the first Friday of every month, GSA web teams test and evaluate the top tasks of a specific website or application. The site might be a GSA website or one from another agency.
2. How many participants should participate in a usability test?
- Usability tests usually involve between 5-10 participants per test.
- Early on in the design process, usability tests with an even smaller number of participants can be sufficient for catching big design flaws.
- For usability tests further along in the web design process, the number of participants in each test will depend on the percentage of issues you want to catch—the less frequently an issue occurs, the more participants you will need to catch that issue.
- If your website serves a wide variety of customer segments, you may need to run separate usability tests for each group (for example, you may want to run one test with 6 average users, and one with 6 sophisticated users).
3. From whom should you get buy-in prior to conducting the test?
In order to act on the feedback from a usability test, you will need buy-in from key stakeholders. This can mean including content, technical, and web managers, as well as managers of related business or program areas that might need to make changes based on the results of this usability test. In some instances, there may be others who should be informed, such as the public relations and communication departments.
Test design
1. What is the format of a usability test?
While usability tests can have many different formats, they generally include a set of predetermined tasks that participants complete after prompting by a facilitator. In addition, there is often a free-form observation period, where participants are simply observed while interacting with the website. At the end of the session, there is sometimes more discussion between the facilitator and the participants, occasionally followed up with a written or oral survey. The length of each usability test will depend on the goals of the usability test, the number of tasks, the duration of each task, and the amount of time the participant spends with the facilitator. Average usability tests last for somewhere between 20 minutes to one hour per test, although some are longer.
2. How do you choose which tasks to include in the test?
Some key criteria when selecting tasks are:
- Frequency (i.e., how often will a customer be completing this task on your website)
- Importance (i.e., the most critical tasks)
- Seriousness of errors (i.e., tasks where if they are done incorrectly it is a serious problem)
- Uncertainty (i.e., tasks you have questions about)
- Actionable (i.e., if users have difficulty with a task, will you be able to make changes)
3. How frequently should you conduct usability testing?
This can be done as often as needed, and will depend on how often you plan to make changes or updates to your website, as well as available resources.
4. Who at your agency should observe a usability test?
While a basic usability test requires only a facilitator and an observer/note-taker, it can be helpful to also have key stakeholders who will be involved in implementing changes to observe the tests as well. (Observers should not actively participate in the usability test itself in order not to disrupt it.) For example, if you are testing a section of your website that is tied to a specific operational or business process, it can be helpful to have stakeholders in those areas observe the usability test.
5. Should you test your usability test questions/tasks?
Yes. It is useful to test the tasks to ensure that the feedback is actionable, and this can be done through a small pilot.
6. Should you collect personally identifiable information?
You generally do not need to collect personally identifiable information. However, collecting some demographic information can be useful for segmenting your customer base.
7. What kind of follow-up work is required after a usability test?
The most important part of a usability test is turning the feedback into action. Typically, this means analyzing the usability test results and putting findings into a report or other format that is easy for the key stakeholders to understand and act upon. In addition, you will want to work with key stakeholders to ensure that the results are clear and that follow-up plans are made.
Operational details
1. Are there technologies to help with usability testing?
There are numerous companies that enable you to run, track, and capture usability test feedback, including videos of participants, recording web navigation, recording eye tracking and other movements, analyzing behavior, and providing reports. Visit usability.gov for more information.
2. Will you need to get approval under the Paperwork Reduction Act (PRA)?
Usability tests require approval if you are including 10 or more individuals. If your test does need PRA approval, it will likely be eligible for the five-day service delivery fast-track process. Check with your PRA officer to learn more.
3. What resources do you need to run a usability test?
At minimum, a web usability test requires a computer connected to the website you are testing, a facilitator to ask questions and request tasks, and an observer to take notes. Some organizations make an audio and video recording of the tests, or use software to record and analyze website navigation (there is also software to track and record eye movements). While a usability test can be run anywhere, some organizations use a separate room or a special usability lab.
4. What are the major cost considerations?
Costs vary greatly depending on whether you outsource or conduct the usability test in-house, the different technologies used, and the internal resources needed to be devoted to the usability test process. Other cost factors include how many tests you run, the number of participants and their incentives, and how you plan to analyze, share, and follow up on the usability test results. Other costs to consider include recruiting and facilitator costs. See Cost and Return on Investment at usability.gov.
If your budget is limited, you should consider whether you already have resources in-house to run a usability test. For example, your agency may already have employees in your IT department who are familiar with running usability tests, or individuals who fit the profile of potential participants. In addition, the General Services Administration runs “First Fridays” usability testing that you may be able to participate in. In a 4-hour period on the first Friday of every month, GSA web teams test and evaluate the top tasks of a specific website or application. The site might be a GSA website or one from another agency.
Resources
- Usability.gov
- Review the Usability & Design guidance on HowTo.gov
»Back to How To Collect Customer Feedback
Content Lead:
Alycia Piazza
Page Reviewed/Updated: September 28, 2011
