Explanation of Ratings

The 'Relative Importance' of Each Guideline

To determine the 'Relative importance' of each guideline, 16 external reviewers were recruited. Half of these reviewers were Web site designers and half were usability specialists. Each reviewer evaluated each guideline and assigned a rating based on the question, 'How important is this guideline to the success of a Web site?' Those guidelines that were rated as having little importance to the success of a Web site were eliminated.

The 'Strength of Evidence' for Each Guideline

A group of eight usability researchers, practitioners and authors were recruited to generate a 'Strength of Evidence' rating for each guideline. These reviewers were all published researchers with doctoral degrees, experienced peer reviewers, and knowledgeable of experimental design. These reviewers constructed a set of criteria for judging the strength of the evidence for each guideline, which was used as the 'Strength of Evidence' scale.