Appendix B: Statistical Methods and Measurement

B.1 Target Population

An important limitation of estimates of drug use prevalence from the National Survey on Drug Use and Health (NSDUH) is that they are only designed to describe the target population of the survey—the civilian, noninstitutionalized population aged 12 or older. Although this population includes almost 98 percent of the total U.S. population aged 12 or older, it excludes some important and unique subpopulations who may have very different alcohol and drug use patterns. Within the population aged 12 or older, this report focuses on persons between the ages of 12 and 20, that is, those who are below the legal drinking age.

B.2 Sampling Error and Statistical Significance

This report includes tables for national and State estimates, produced using a multiprocedure package called SUDAAN^® Software for Statistical Analysis of Correlated Data. SUDAAN was designed for the statistical analysis of data collected using stratified, multistage cluster sampling designs, as well as other observational and experimental studies involving repeated measures or studies subject to cluster correlation effects (RTI International, 2004). The final, nonresponse-adjusted, and poststratified analysis weights were used in SUDAAN to compute unbiased design-based drug use estimates.

The sampling error (i.e., the standard error or SE) of an estimate is the error caused by the selection of a sample instead of conducting a census of the population. The sampling error may be reduced by selecting a large sample and/or by using efficient sample design and estimation strategies, such as stratification, optimal allocation, and ratio estimation.

With the use of probability sampling methods in NSDUH, it is possible to develop estimates of sampling error from the survey data. These estimates have been calculated using SUDAAN for all estimates presented in this report using a Taylor series linearization approach that takes into account the effects of NSDUH's complex design features. The sampling errors are used to identify unreliable estimates and to test for the statistical significance of differences between estimates.

B.2.1 Variance Estimation for Totals

Estimates of means or proportions, _d, such as drug use prevalence estimates for a domain d, can be expressed as a ratio estimate:

,     D

where _d is a linear statistic estimating number of substance users in the domain, and _d is a linear statistic estimating the total number of persons in domain d (both users and nonusers). The SUDAAN software used to develop estimates and their SEs produces direct estimates _d and _d and their SEs. The SUDAAN application also uses a Taylor series approximation method to estimate the SEs of the ratio estimate _d.

When the domain size, _d, is free of sampling error, an appropriate estimate of the SE for the total number of substance users is

SE (_d = _dSE(_d).

This approach is theoretically correct when the domain size estimates, _d, are among those forced to match their respective U.S. Census Bureau population projections through the weight calibration process (Chen et al., 2005). In these cases, _d is not subject to sampling error. For a more detailed explanation of the weight calibration process, see Section A.3.2 in Appendix A.

For estimated domain totals, _d, where _d is not fixed (i.e., where domain size estimates are not forced to match the U.S. Census Bureau population projections), this formulation may still provide a good approximation if it can be assumed that the sampling variation in _d is negligible relative to the sampling variation in _d. This is a reasonable assumption for most cases in this study.

For a subset of the estimates produced from the 2002 to 2006 data, the above approach yielded an underestimate of the variance of a total because _d was subject to considerable variation. In these cases, the SEs for the total estimates calculated directly within SUDAAN are reported. Using the SEs from the total estimates directly from SUDAAN does not affect the SE estimates for the corresponding proportions presented in the same sets of tables.

B.2.2 Suppression Criteria for Unreliable Estimates

As has been done in other NSDUH reports, direct survey estimates produced for this study that are considered to be unreliable due to unacceptably large sampling errors are not shown in this report and are noted by asterisks (*) in the tables containing such estimates. The criteria used for suppressing all direct survey estimates were based on the relative standard error (RSE) (defined as the ratio of the SE over the estimate), nominal (actual) sample size, and effective sample size for each estimate.

Proportion estimates () within the range [0 < < 1], rates, and the corresponding estimated number of users were suppressed if

RSE[-ln()] > .175 when ≤ .5

or

RSE[-ln(1 - )] > .175 when > .5.

Using a first-order Taylor series approximation to estimate RSE[-ln()] and RSE[-ln(1 - )], the following equation was derived and used for computational purposes:

or

     D

The separate formulas for ≤ .5 and > .5 produce a symmetric suppression rule; that is, if is suppressed, 1 – will be suppressed as well. This ad hoc rule requires an effective sample size in excess of 50. When .05 < < .95, the symmetric property of the rule produces a local maximum effective sample size of 68 at = .5. Thus, estimates with these values of along with effective sample sizes falling below 68 are suppressed. See Figure B.1 for a representation of the required minimum effective sample sizes as a function of the proportion estimated.

Below is a graph. Click here for the text describing this graph.

Figure B.1 Required Effective Sample as a Function of the Proportion Estimated

A minimum nominal sample size suppression criterion (n = 100) that protects against unreliable estimates caused by small design effects and small nominal sample sizes was employed. Prevalence estimates also were suppressed if they were close to 0 or 100 percent (i.e., if < .00005 or if ≥ .99995).

Estimates of other totals along with means and rates that are not bounded between 0 and 1 (e.g., mean age at first use and incidence rates) were suppressed if the RSEs of the estimates were larger than .5. Additionally, estimates of the mean age at first use were suppressed if the sample size was smaller than 10 respondents. Also, the estimated incidence rate and number of initiates were suppressed if they rounded to 0.

The suppression criteria for various NSDUH estimates are summarized in Table B.1 at the end of this appendix.

B.2.3 Statistical Significance of Differences

This section describes the methods used to compare prevalence estimates in this report. Customarily, the observed difference between estimates is evaluated in terms of its statistical significance. Statistical significance is based on the p value of the test statistic and refers to the probability that a difference as large as that observed would occur due to random variability in the estimates if there were no difference in the prevalence estimates for the population groups being compared. The significance of observed differences in this report is reported at the .05 level. When comparing prevalence estimates, the null hypothesis (no difference between prevalence estimates) was tested against the alternative hypothesis (there is a difference in prevalence estimates) using the standard difference in proportions test expressed as

,     D

where ₁ = first prevalence estimate, ₂ = second prevalence estimate, var(₁) = variance of first prevalence estimate, var(₂) = variance of second prevalence estimate, and cov(₁, ₂) = covariance between ₁ and ₂. In cases where significance tests between years were performed, the prevalence estimate from the earlier year (2002, 2003, 2004, or 2005) becomes the first prevalence estimate, and the prevalence estimate from the later year (2006) becomes the second prevalence estimate.

Under the null hypothesis, Z is asymptotically distributed as a normal random variable. Therefore, calculated values of Z can be referred to the unit normal distribution to determine the corresponding probability level (i.e., p value). Because the covariance term between the two estimates is not necessarily zero, SUDAAN was used to compute estimates of Z along with the associated p values using the analysis weights and accounting for the sample design as described in Appendix A. A similar procedure and formula for Z were used for estimated totals; however, it should be noted that because it was necessary to calculate the SE outside of SUDAAN for domains forced by the weighting process to match their respective U.S. Census Bureau population estimates, the corresponding test statistics also were computed outside of SUDAAN.

When comparing population subgroups across three or more levels of a categorical variable, log-linear chi-square tests of independence of the subgroups and the prevalence variables were conducted first to control the error level for multiple comparisons. If the chi-square test indicated overall significant differences, the significance of each particular pairwise comparison of interest was tested using SUDAAN analytic procedures to properly account for the sample design. Using the published estimates and SEs to perform independent t tests for the difference of proportions usually will provide the same results as tests performed in SUDAAN. However, where the significance level is borderline, results may differ for two reasons: (1) the covariance term is included in SUDAAN tests, whereas it is not included in independent t tests; and (2) the reduced number of significant digits shown in the published estimates may cause rounding errors in the independent t tests.

B.3 Other Information on Data Accuracy

The accuracy of survey estimates can be affected by nonresponse, coding errors, computer processing errors, errors in the sampling frame, reporting errors, and other errors not due to sampling. They are sometimes referred to as "nonsampling errors." These types of errors and their impact are reduced through data editing, statistical adjustments for nonresponse, close monitoring and periodic retraining of interviewers, and improvement in various quality control procedures.

Although these types of errors often can be much larger than sampling errors, measurement of most of these errors is difficult. However, some indication of the effects of some types of these errors can be obtained through proxy measures, such as response rates and from other research studies.

B.3.1 Screening and Interview Response Rate Patterns

Beginning in 2002 and continuing through 2006, respondents received a $30 incentive in an effort to maximize response rates. Of the 151,288 eligible households sampled for the 2006 NSDUH, 137,057 were screened successfully, for a weighted screening response rate of 90.6 percent. In these screened households, a total of 85,034 sample persons were selected, and completed interviews were obtained from 67,802 of these sample persons, for a weighted interview response rate of 74.2 percent. The overall weighted response rate, defined as the product of the weighted screening response rate and weighted interview response rate, was 67.2 percent in 2006. The interview response rate for persons aged 12 to 20 was 84.9 percent.

The weighted screening rates for the 2002 to 2005 NSDUHs ranged from 90.7 to 91.3 percent, the interviewer response rates ranged from 76.2 to 78.6 percent, and the overall response rates ranged from 71.3 percent in 2002 to 69.2 percent in 2005. For the sample aged 12 to 20, the interview response rates ranged from 84.9 to 89.2 percent.

Nonresponse bias can be expressed as the product of the nonresponse rate (1 - R) and the difference between the characteristic of interest between respondents and nonrespondents in the population (P_r - P_nr). By maximizing NSDUH response rates, it is hoped that the bias due to the difference between the estimates from respondents and nonrespondents is minimized. Alcohol and drug use surveys are particularly vulnerable to nonresponse due to the difficult nature of accessing heavy alcohol and drug users.

B.3.2 Inconsistent Responses and Item Nonresponse

Among survey participants, item response rates were above 99 percent for most drug use items. However, inconsistent responses for some items were common. Estimates of substance use from NSDUH are based on responses to multiple questions by respondents, so that the maximum amount of information is used in determining whether a respondent is classified as a drug user. Inconsistencies in responses are resolved through a logical editing process that involves some judgment on the part of survey analysts. Additionally, missing or inconsistent responses are imputed using statistical methodology. Editing and imputation of missing responses are potential sources of error.

Respondents were asked the dependence and abuse questions if they reported alcohol use on more than 5 days in the past year, or if they reported any alcohol use in the past year but did not report their frequency of past year use. Therefore, inconsistencies could have occurred where the imputed frequency of use response indicated less frequent use than required for respondents to be asked the dependence and abuse questions originally.

Respondents might have provided ambiguous information about past year use of alcohol, in which case these respondents were not asked the dependence and abuse questions for alcohol. Subsequently, these respondents could have been imputed to be past year users of alcohol. In this situation, the dependence and abuse data were unknown; thus, these respondents were classified as not dependent on or abusing alcohol. However, such a respondent never actually was asked the dependence and abuse questions.

B.3.3 Validity of Self-Reported Substance Use

Most drug use prevalence estimates, including those produced for NSDUH, are based on self-reports of use. Although studies have generally supported the validity of self-report data, it is well documented that these data often are biased (underreported or overreported) by several factors, including the mode of administration, the population under investigation, and the type of drug (Bradburn & Sudman, 1983; Hser & Anglin, 1993). Higher levels of bias also are observed among younger respondents and those with higher levels of drug use (Biglan, Gilpin, Rohrbach, & Pierce, 2004). Methodological procedures, such as biological specimens (e.g., urine, hair, saliva), proxy reports (e.g., family member, peer), and repeated measures (e.g., recanting), have been used to validate self-report data (Fendrich, Johnson, Sudman, Wislar, & Spiehler, 1999). However, these procedures often are impractical or too costly for community-based epidemiological studies (SRNT Subcommittee on Biochemical Verification, 2002). NSDUH utilizes widely accepted methodological practices for ensuring validity, such as encouraging privacy through audio computer-assisted self-interviewing (ACASI). Comparisons using these methods within NSDUH have been shown to reduce reporting bias (Aquilino, 1994; Turner, Lessler, & Gfroerer, 1992).

B.4 Measurement Issues and Additional Findings for Alcohol Items Added in 2006

As noted in Section 1.4 of Chapter 1, NSDUH in 2006 incorporated a new consumption of alcohol module that collected additional information about respondents' last use of alcohol for those who indicated that they had consumed alcohol at least once in the past month. The module included some items that were administered only to persons aged 12 to 20. Among the items in the new module were two related to binge drinking among females based on consumption of four or more drinks on an occasion, rather than the usual NSDUH criterion of five or more drinks. Other items in the consumption of alcohol module included the source of alcohol, location, and social context of the last drinking episode among past month alcohol users aged 12 to 20; the number of drinks consumed on the last drinking occasion; and the use of illicit drugs in combination with alcohol or within 2 hours of consuming alcohol on the last drinking occasion. Findings for many of these items are covered in the tables in Appendix C and discussed in Chapter 4. This section provides further information on some of the items in the new module, including a discussion of data collection issues with the new four-drink binge drinking measure for females and additional findings for this and selected other variables.

B.4.1 Data Issues Involving the Measurement of Binge Alcohol Use for Females

In 2006, new items were added within the consumption of alcohol module to investigate whether the current binge drinking definition based on drinking five or more drinks on the same occasion should be changed to a lower threshold of four or more drinks on the same occasion for females. The four or more drinks definition corresponds to that used by the National Institute on Alcohol Abuse and Alcoholism (NIAAA, 2004). Although all persons aged 12 or older were asked the new items regarding lifetime binge use and age of initiation of binge use based on the five or more drinks definition, the question pertaining to binge use based on four or more drinks and related follow-up questions were asked only of females. The intent was to route all female respondents who were lifetime drinkers into the questions regarding their history of having four or more drinks on the same occasion.

During the editing process, it was discovered that females who had some history of consuming five or more drinks on the same occasion were skipped out of the questions regarding their history of having four or more drinks. The sole exception to this was that females who had a history of having five or more drinks on the same occasion and who also indicated that they had four or more drinks when they last used alcohol were properly routed into the questions about their history of having five or more drinks on the same occasion. As a result of this error, 1,235 females aged 12 to 20 who were lifetime alcohol users (8.0 percent of all females in this group) were not asked the questions regarding their history of having four or more drinks on the same occasion.

In most cases, females who were incorrectly skipped out of the four or more drinks questions could be assigned a four or more drinks status for both past month and lifetime use either from previous responses to questions about the number of drinks on the last occasion or questions about binge use based on five or more drinks earlier in the same module. In cases where there was unknown data, the information from the core computer-assisted interviewing (CAI) module about binge use status, an imputed revised measure, was used to determine whether the female should be recoded to be a binge drinker based on the four or more drinks definition. Data on initiation of binge alcohol use based on the four or more drinks definition were not available for those females incorrectly skipped from these questions. Because of the large number of females who were improperly skipped, estimates of the age of initiation of binge drinking based on four or more drinks definition are not presented in this report.

B.4.2 Results from Selected Alcohol Use Items Added in 2006

The results based on many of the new items in the consumption of alcohol module are presented in Chapter 4. This section presents supplementary tables on the number of drinks consumed when persons aged 12 to 20 last consumed alcohol, as well as comparisons of binge drinking rates using the criterion of five more drinks on the same occasion for males and females and the criterion of four or more drinks on the same occasion for females.

Table B.2AB presents the number of drinks consumed the last time past month drinkers aged 12 to 20 drank alcohol, both for the full sample and by gender and age group. The majority of underage drinkers in each age and gender group reported drinking four or fewer drinks the last time they used alcohol. Current drinkers aged 12 to 14 were more likely to have had only one drink on their last drinking occasion (43.7 percent) compared with those aged 15 to 17 (23.7 percent) or those aged 18 to 20 (17.0 percent). The 18 to 20 year olds had the highest percentage of persons who reported drinking nine or more drinks at the last occasion (14.6 percent) compared with the other age groups (5.9 percent for those aged 12 to 14, 10.8 percent for those aged 15 to 17). Females were more likely than males to have had one drink (22.8 percent for females, 19.2 percent for males), two drinks (20.2 percent for females, 15.3 percent for males), or three or four drinks (27.1 percent for females, 19.8 percent for males) when they last drank alcohol. In contrast, underage males were more likely than underage females to have had five to eight drinks (26.6 percent for males, 24.1 percent for females) or nine or more drinks (19.1 percent for males, 5.8 percent for females) when they last drank alcohol.

Information on the mean number of drinks for these groups is also included in Table B.2AB. Overall, current drinkers aged 12 to 20 averaged 4.5 drinks when they last used alcohol. The mean number of drinks reported increased with age. Persons aged 12 to 14 had the lowest mean number of drinks (2.8 drinks), those aged 15 to 17 reported a higher amount at 4.3 drinks on average, and the 18 to 20 age group reported the highest average number at 4.8 drinks on the last occasion. Among current drinkers, underage males consumed more drinks on their last drinking occasion (mean of 5.3 drinks) than underage females (mean of 3.7 drinks).

Tables B.3A and B.3B present information from 2006 on binge alcohol use as defined by drinking five or more drinks on one occasion for both males and females, and an alternative definition based on five or more drinks for males but four or more drinks for females. Using the definition of five or more drinks on one occasion for both males and females, the rate of lifetime binge drinking was higher for underage males (32.0 percent) than for underage females (28.0 percent). This difference was principally found for those aged 18 to 20, where males had a higher rate of lifetime binge use (59.0 percent) compared with females (49.5 percent). A similar pattern was found for past month binge use using the five or more drinks criterion, with males having a higher rate than females (21.8 to 16.9 percent, respectively). Among both 15 to 17 year olds and 18 to 20 year olds, males had higher rates of past month binge drinking compared with females when binge drinking was defined as five or more drinks for both males and females.

Although a statistical comparison cannot be drawn between the binge drinking rates based on the two different definitions, it can be seen that using the less stringent definition for lifetime binge use increased the rate of lifetime binge alcohol use for females from 28.0 to 31.0 percent, and it increased the rate of past month binge alcohol use for females from 16.9 to 18.5 percent. This higher rate of binge drinking for females using the four or more drinks criterion also can be seen in the overall rate of binge drinking including both males and females; lifetime binge drinking increased from 30.1 to 31.5 percent, and past month binge drinking increased from 19.4 to 20.2 percent. The increase in the prevalence of binge drinking among females using the four or more drinks criterion was primarily found for females aged 18 to 20.

Regarding initiation of binge drinking, 10.2 percent of all those aged 12 to 20 reported that they first had five or more drinks on the same occasion during the past year. Information on the measurement of initiation can be found in Section B.4.1 of the 2006 NSDUH national findings report (Office of Applied Studies [OAS], 2007a). The past year initiation rate of binge drinking was higher for those aged 15 to 17 and those aged 18 to 20 (13.9 and 13.1 percent, respectively) than for those aged 12 to 14 (3.5 percent). In addition, a higher percentage of males (10.9 percent) than females (9.5 percent) reported that they engaged in binge drinking for the first time in the past year. However, this pattern of gender differences varied by age group. Among those aged 12 to 14, the rate for past year initiation of binge drinking was higher among females (3.9 percent) than among males (3.0 percent). Among those aged 15 to 17, there was no statistically significant difference between males and females in past year initiation of binge drinking. Among those aged 18 to 20, however, males had a higher rate for past year initiation (15.2 percent) than females (10.9 percent).