Site Map     Search the Catalog     Find a Library     FAQ     Sign Up     Contact Us

Home > Patron Survey (2003) Index Page > Section 3. Methodolofy

NLS: That All May Read

2. Study Background     4. Data Analysis

3.0 Methodology

The methodology employed to conduct the NLS Patron Survey is presented in the following sections: 

• Sampling Plan and Procedures
• Questionnaire Development
• Data Collection
• Data Analysis Issues

Sampling Plan and Procedures

The population of interest consisted of active NLS subscribers who were adults (ages 18 and over, born in 1985 or before) living in one of the 50 United States. The sampling frame was constructed from the Comprehensive Mailing List System (CMLS).  Since there were likely to be important age or generational differences in subscribers’ needs with respect to the new Talking Book technology, the design involved stratified sampling based on age.  Study participants who represented a given age stratum, based on CMLS data, were systematically selected from the total population of NLS patrons in that age group. The strata were defined by the following age groups:

• Group 1.  Ages 18 – 39 (born between 1964 and 1985)
• Group 2.  Ages 40 – 64 (born between 1939 and 1963)
• Group 3.  Ages 65 – 84 (born between 1919 and 1938)
• Group 4.  Ages 85 and over (born in 1918 or before)
• Group 5.  People whose year of birth is missing in the CMLS

The first four strata corresponded to young adulthood, middle-age, early-to-mid old age, and late old age. The fifth stratum consists of the subscribers for whom CMLS information on year of birth is missing (2% – 3%). These subscribers were used in the sampling frame to avoid bias from any systematic differences that may have existed between patrons for whom birth year is available from the CMLS and patrons for whom it is missing. For final analysis, these "age unknown" cases were reclassified into the appropriate age group based on information that participants supplied during the interviews.

Stratified random sampling, specifically disproportionate stratified random sampling, was used rather than simple random sampling in order to reduce confidence intervals, increase the precision of the findings, and make sure the sample contained enough subscribers in each age group for comparisons.  Instead of sampling each age group at a rate that reflects its prevalence in the total NLS population, each age group was sampled at a rate that produced age subgroups of approximately equal size. Stated differently, the younger age groups (which are less numerous in the NLS population) were over-sampled, and the older age groups (which are more numerous) were under-sampled.

Create Sampling Frame

The sampling frame consisted of all study-eligible NLS patrons as defined in Exhibit 3-1 below:

To create the frame, we removed ineligible records, removed duplicate cases, created age group-specific sampling frames, and scrambled the five frames in preparation for sample selection. 

Sample Selection

We used systematic sampling with a random start (SSRS) to select the sample for each age group. After determining the sampling interval for each age group, we used a table of random numbers to select the first person. After that, participants were selected systematically, according to the appropriate sampling interval. 

The main sample was to include 725 subscribers equally distributed among four age groups (175 or 176 subscribers in age groups 1-4) plus 22 subscribers with birth dates unknown.  The (first) reserve sample was to include 250 subscribers equally distributed among four age groups (60 or 61 subscribers in age groups 1-4) plus 8 subscribers with birth dates unknown. (The actual main and reserve samples received from NLS included 723 and 250 subscribers, respectively).  We anticipated that this sample would generate a total of 500 completed interviews (125 completes in each of the  4 groups).   However, the original main and reserve lists included subscribers with no or incomplete phone information as well as subscribers who were deceased or suspended.  Once the NLS main and reserve sample lists were cleaned, the main and reserve sample lists included 522 and 180 subscribers respectively.   These 702 subscribers did not produce the number of completed interviews needed (because of inability to reach subscribers or finish the interview process) so NLS drew an additional sample of 1,001 subscribers.  From this second reserve sample, 744 subscribers were considered eligible; however, only 549 subscribers were needed as replacements for the main and first reserve sample.  Exhibit 3-2 provides a breakdown of the sample size and number of interviews completed from each sample. 

* A total of 744 (or 74%) of reserve 2 was considered eligible prior to calling; however, only the replacements needed to complete age group interviews (a total of 549 subscribers) were included in the Reserve 2 sample.

Questionnaire Development

At the start of the project, key issues of the survey were identified and served as the foundation for developing the questionnaire.  These key issues were as follows: 

• Patrons’ levels of physical and cognitive functioning, as reflected in their past and recent experiences with their current playback equipment 
• Patrons’ rating of their current equipment and NLS service
• Patrons’ current patterns of technology use, both with respect to their current playback device and other technology
• Benefits of the new technology that are most and least important to patrons
• Patrons’ background characteristics, especially health-related and social characteristics that are likely to affect patrons’ needs and preferences.

The questionnaire consisted of 82 questions and was designed to be completed within 25 minutes.  Appendix A presents the NLS Patron Survey Questionnaire.   In addition to the survey questions, the response codes for the survey were developed and are defined in Appendix B.  

Pre-Test and Interviewer Training

Prior to conducting the survey, interviewers received training which included an overview of NLS, its subscribers, its playback equipment, and the NLS Patron Survey.  Training included the rationale for the information collection, the survey questions, refusal script, and use of the computer-assisted telephone interviewing (CATI) system.  Training activities also included role-playing and mock interviews to prepare the interviewers for potential questions about the survey and refusals to participate.   Following training, the questionnaire was pre-tested on 20 subscribers from the sample.   The results of pre-testing indicated that subscribers understood the questionnaire and interviewers were able to complete the questionnaire within 25 minutes.  Only minor script and questionnaire adjustments were necessary.

Once NLS approved the content of the questionnaire, the survey questions and the interviewers’ script were programmed into a CATI system.  The programmed version was carefully checked for proper execution.  Test data were keyed into the system to ensure proper operation.  This method ensured that once the data collection began, the risk of operational errors or delays was minimized. 

Additionally, a pre-survey letter was sent to all eligible subscribers in the main and first reserve sample to announce the survey and inform subscribers they might be asked to participate.  The pre-survey letter also indicated that participation was voluntary and responses were confidential.  A copy of the pre-survey letter is provided in Appendix C.

Data Collection

Data collection took place between October 10 and November 14, 2003. CR Dynamics conducted telephone interviews Monday-Saturdays from 10am to 9pm Eastern Time.

After the first two weeks of calling, it became clear that the main and first reserve sample would not produce 500 completed interviews.  As described previously, NLS drew a second reserve sample of 1,001 subscribers.  Of those, 744 subscribers were determined eligible; however only 549 were used as replacements in the age groups for which we had not already achieved 125 completed interviews. 

The CATI system managed the sample, automatically produced status reports, dialed the subscribers’ telephone numbers, and scheduled at least 5 callbacks to subscribers who were not reached on the initial call. 

Cooperation Rates

The cooperation rate is all people interviewed as a percentage of all eligible people for whom contact was ever attempted. NLS sought to achieve a cooperation rate of 70 percent or better based on the Cooperation Rate 2 (COOP2) formula for random digit dialing (RDD) surveys as described in the American Association for Public Opinion Research report Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys.

As defined in the American Association ofr Publid Opinion Research. 2000. Standard Definitions: Final Deispositions of Case Codes and OUtcome Rates for Surveys. Ann Arbor, Michigan: AAPOR, the formula for COOP2 is (1+P)(I+P)+R+O.

The COOP2 formula counts as respondents people who provided either a complete or a partial interview. The denominator encompasses all eligible people selected for the sample, including

• Complete and partial interviews
• Refusals and break-offs
• Other eligible non-respondents.

For the NLS Patron Survey, the criteria used for defining complete and partial interviews reflected NLS’ particular interest in subscribers’ demographic characteristics and the physical and cognitive capabilities most likely to affect machine operation. The criteria were as follows: 

• A complete interview had valid (non-missing) data for at least 90% of all applicable capability and demographic items and for at least 60% of all remaining items that applied to the respondent.
• A partial interview had valid (non-missing) data for 50% to 89% of all applicable capability and demographic items.
• Interviews with valid (non-missing) data for less than 50% of all applicable capability and demographic items were counted as break-offs regardless of how complete the data are for other areas of questioning.

Using the COOP2 formula, the NLS Patron Survey achieved a cooperation rate of 51 percent for the entire sample. The rate, however, varied considerably among age groups, as Exhibit 3-3 shows. We met our goal of a 70% cooperation rate for subscribers between the ages of 65 and 84 (72%), and we approached that goal for subscribers in the 40 to 64 age group (65%). The cooperation rates for the youngest and oldest subscribers, however, were much lower (33% and 44% respectively).

*Note for column, Unknown: Eligible non-respondents for whom their age was unknown based on CMLS data are included in the COOP2 rate. Their status as unknown is kept since an interview was unable to establish their birth date. Completed interviews of age unknowns were reclassified to the appropriate age group following completion of the survey.

Data Analysis Issues

The following section presents two factors that affected the analysis of the NLS Patron Survey:  1) assessment of the survey’s eligible non-respondents, and 2) how the completed interview data was weighted.

Assessment of Non-Respondents

By a margin of almost two to one, non-participation of eligible non-respondents resulted from unwillingness to participate rather than subscribers’ apparent busyness or mobility (63% vs. 37% of all eligible non-respondents). The single most common reason for non-participation by far is refusal to be interviewed. Subscribers who refused make up almost half of all eligible non-respondents (207 out of 430, or 48%). Further, data collection took place just after the Federal government established its do-not-call registry. Although the program does not prohibit calls for research purposes or calls to current customers, we lost an additional 34 eligible subscribers who said they had signed up for this registry. When added to the subscribers who declined to participate on other grounds, people who refused to begin the interview make up more than half of all eligible non-respondents (241/430, or 56%).  Exhibit 3-4 presents the interview termination codes for non-respondents by age group.

The reasons for non-response are strongly age-related. Although unwillingness to participate occurs in all age groups, it increases with age. It is least common among the young adults (39%). Unwillingness accounts for much of the non-response among working-age adults (64%) and nearly all of the non-response among retirement-age subscribers (78% to 80%).

The youngest subscribers’ apparent busyness or mobility accounts for much of the non-response in this age group (61%). People between the ages of 18 and 39, moreover, make up more than half of all eligible subscribers whose non-participation is attributable to our inability to establish contact with them.

Ineligible non-respondents were those subscribers that were either found to be ineligible at the time of interviewer contact or were never reached due to inaccurate phone information.  A significant percentage of all of the non-respondents in this group were ineligible due to their contact information (78% of the ineligible non-respondents).  Again, the younger age groups (18 – 64) had the highest rate (81%) of having a disconnected, fax line, or wrong number; however, in all age groups inaccurate phone information proved to be an obstacle in contacting potential respondents.

Weighting

All data were weighted during analysis to adjust for the disproportionate stratified sampling strategy. The weight applied to each member of a particular subgroup was based on a formula that took into account the rate at which members of that group were sampled, the total sample size, and the total population size. In short, weighting restored the ability to generalize from the stratified sample to all NLS subscribers.  Exhibit 3-5 presents the sample weights for the survey and Exhibit 3-6 presents the unweighted and weighted data for each age group.

Exhibit 3-5 note. Sample Weight Formula:  The weight for each stratum = (1 / sampling rate) x (Total sample N/ Total population N)

Figures were multiplied by 1,000 to make them less awkward to work with. Because 1,000 was used as a constant, i.e., it was applied to both elements of the weight formula, this strategy does not distort the resulting weights.

The percentages, although calculated using weighted sample data, describe the age distributions both for the sample and for the portion of the NLS population from which the sample was drawn (i.e., subscribers age 18 or over who were eligible for the study based on CMLS information). Our weighting procedure ensured that the age distributions in the sample and in the population are the same. One would obtain identical results by using the NLS population information shown in Exhibit 3-5. For example, according to CMLS information, there are 92,472 study-eligible subscribers who are at least 85 years old; dividing this figure by 367,729 (the total number of study-eligible subscribers) would show that people in the 85+ age group make up 25% of the total study-eligible population, the same figure presented in Exhibit 3-6.

Once the data were weighted, the data were analyzed using SPSS 12.0.  The next section presents the data analysis.

2. Study Background     4. Data Analysis

Library of Congress Home    NLS Home    Comments about NLS to nls@loc.gov

About this site    Comments about this site to the NLS Reference Section

Posted on 2006-05-30