Controlled Comparative Studies

Collect quantitative data on how well the general public uses your final document.

Controlled comparative studies can be done in several different ways, but they all have similar characteristics. Before you do a controlled study, you should know what results you will consider a success. For instance:

• Do you want more calls regarding a certain program?
• Do you want fewer calls asking for clarification?
• Do you want more people to return an application or a payment?
• Do you want fewer errors on forms people fill out?

Having answers to these questions will help you determine whether your document is successful. Controlled comparative studies are often called A/B testing. You have two (or more versions) of your page — A, B, etc.

Websites and web pages.

Set your web server to send each on a specific schedule (every other call to the page, or one today and the other tomorrow, or one for a certain longer period and the other for the next equal period). Just be sure you can track whatever measure you want by which version the web site visitor saw.

Paper documents.

Send a small test group of people the new version of your document. Let's say you're sending the new version to 700 people. You should also send 700 people, your control group, the old document. Track the responses to all 1400 documents and compare the results. Note that it is much easier to test results when people return a written response than when you try to track the number of phone calls you receive. (If you have a statistician or actuarial staff, they can tell you how many people you should use to make your study scientifically valid. If your agency doesn't have an expert on staff to help you, statistics books will give you a formula to determine a good sample size for your study.)

There are numerous other ways of collecting quantitative data. For instance, you can record what percentage of your "before" letters generates correct responses compared to your "after letters," or what percentage of each letter results in your customer calling you asking for an explanation.

Before you do a controlled comparative study, you should do paraphrase testing or usability testing and change you document based on what you learn in these smaller scale studies. Controlled comparative studies (especially for paper documents) are best near the end of the process. This is because controlled testing will tell you if the new document is a success, but it won't tell you why it is or isn't a success.