Frequently Asked Questions
- Will model-based estimates be generated for 2004-2006 and 2007-2009?
- Will the model-based estimates be expanded to other screening or cancer related variables?
- What are direct estimates? Why are there differences between the direct estimates and the estimates from the model?
- When should the model-based estimates be used?
- Are there factors other than noncoverage and nonresponse that contribute to the differences in the BRFSS and NHIS direct estimates?
- Why did you use a state level model to produce state estimates rather than aggregating the county estimates from the county level model to state level?
- Did you control the county estimates so that the aggregated county estimates equal to the state estimates?
- Can these model-based estimates be used to rank and compare counties or states?
- Who developed this methodology?
- Whom should I contact if I have additional questions?
1. Will model-based estimates be generated for 2004-2006 and 2007-2009?
Yes, plans are underway to develop estimates for more recent periods. However, because of the increase in cell-phone only households, we felt that the methodology had to be modified to account for the fact that BFRSS does not capture households of this type.
2. Will the model-based estimates be expanded to other screening or cancer related variables?
For each new potential variable to be added, a careful evaluation must be conducted to determine if the questions are asked in an identical (or at least very similar) manner in the two surveys. Performing analyses when the questions are substantially different could lead to inappropriate estimates. We would like to expand to additional screening or cancer related variables in the future. Suggestions for future variables to include are welcome.
3. What are direct estimates? Why are there differences between the direct estimates and the estimates from the model?
The direct estimates are constructed based only on the survey data and the design weights for the sample in the small areas of interest. When the sample sizes are small, the direct estimates are unreliable. The model-based estimates are based on an explicit statistical regression model which combines the direct estimates and information from auxiliary data on small areas obtained from a variety of external sources. The model-based estimates are generally more reliable and stable assuming that the data fit the model reasonably well. However, when there is not much local data for a particular area, the model-based estimates rely on the relationship between the auxiliary data describing the characteristics of the area and the response variable of interest, e.g., the relationship between median family income and smoking prevalence at the county level.
4. When should the model-based estimates be used?
The data user needs to make a decision according to the situation in their area. The model-based estimates are expected to be better than the direct estimates on average, if the models used are appropriate; but that doesn't mean that the model-based estimates are close to the true values for every area. For areas with large NHIS and/or BRFSS sample sizes, the model-based estimates are influenced by the associated direct estimates. As the area level sample sizes get smaller, the model-based estimates increasingly represent estimates for all the areas with similar profiles based on characteristics as reflected in the auxiliary data, rather than an estimate that reflects any unusual anomalies of a specific area. If, in this later situation, there are some special programs implemented in a specific area to promote certain cancer prevention or cancer screening, those will not be reflected in the estimates, and one needs access to the local data to obtain more accurate estimates.
5. Are there factors other than noncoverage and nonresponse that contribute to the differences in the BRFSS and NHIS direct estimates?
Differences in data collection time, proxy responses, question wording, response modes (telephone vs. in-person), sample design and weighting methodology could cause differences between the BRFSS and NHIS prevalence estimates. For small areas there is also larger sampling error and hence, estimates could differ by chance.
Both BRFSS and NHIS are designed to represent the adult population (18 years old or over) living in households. However, the NHIS is a complex, multistage area probability sample that incorporates stratification, clustering, and oversampling of some subpopulations (e.g., Black, Hispanic, and Asian in later years); while the BRFSS is a state-based random-digit-dial (RDD) probability sample that incorporates disproportionate stratified sampling in which listed residential telephone numbers are sampled at a higher rate than unlisted residential telephone numbers. Weighting adjustments are implemented in both surveys to compensate for design-imposed differential selection probabilities, nonrespondents, and under-covered population. Differences in the weighting adjustment process may cause some observed differences in the direct estimates from the two surveys.
6. Why did you use a state level model to produce state estimates rather than aggregating the county estimates from the county level model to state level?
We used a state level model mainly for two reasons. First, the direct estimates at the state level are more reliable than at the county level. Second, the relationship between the direct estimates and the auxiliary data at different geographical levels might be different.
7. Did you control the county estimates so that the aggregated county estimates are equal to the state estimates?
We produced the county and state estimates using separate models—one at the county level and the other at the state level. Health Service Area level estimates were aggregated from the county level estimates. While the county level estimates aggregated to the state level generally were similar to the estimates from the state level model, no attempt was made to force them to agree. A procedure sometimes used to control the county level estimates to the state level estimates is to apply a simple ratio adjustment to the county level estimates so that they sum to the state estimates. However, we felt that such a procedure is based on ad-hoc decision rules. We decided not to do any controlling at this point, but will keep this in mind for future efforts.
8. Can these model-based estimates be used to rank and compare counties or states?
The model-based estimates alone cannot be used to rank and compare counties because these estimates are associated with random errors which were due to sampling error in the direct estimates and the lack-of-fit of the models. Ranking the model-based estimates accounting for the associated standard errors involves complex statistical techniques and is beyond the scope of this project.
9. Who developed this methodology?
This work was undertaken as a collaboration of the Statistical Research & Applications Branch at the National Cancer Institute, the National Center for Health Statistics, and academic researchers from the Department of Biostatistics
and the Institute for Social Research at the University of Michigan, and the Center for Clinical Epidemiology and Biostatistics at the University of Pennsylvania School of Medicine. Both survey groups (NHIS at the National Center for Health Statistics, and BRFSS at the National Center for Chronic Disease Prevention and Health Promotion, both within the Centers for Disease Control and Prevention) have been involved in the development of this Web site, and are supportive of the development of methodologies of this type which takes advantage of the strengths of two complimentary government sponsored national health surveys.
10. Whom should I contact if I have additional questions or comments?
Please email the Small Area Estimates Web Staff with any questions or comments and your message will be forwarded to the appropriate staff member. We hope that researchers and cancer control planners will provide feedback to the NCI on the utility, strengths, and shortcomings of these new estimates. While these estimates may have great utility in local and regional cancer control planning, they should be supplemented with local knowledge and information when available. Feedback is greatly appreciated, both in terms of the global utility of these estimates, as well as local anomalies.
