Limitations & Uses of the Model-Based Small Area Estimates

The data user needs to make a decision on when to use the model-based small area estimates according to the situation in their area. The model-based estimates are expected to be better than the direct estimates on average, if the models used are appropriate; however, that doesn't mean that the model-based estimates are close to the true values for every area. The new estimates may offer the most improvement in geographical areas where BRFSS direct estimates have a low response rate and/or low coverage, and have a medium or large number of NHIS respondents. For the BRFSS average state response rates for the two time periods, and for the county level non-telephone coverage obtained from Census 2000, see Rationale for the Model-Based Small Area Estimates.

Researchers and cancer control planners should decide on the utility of these model-based small area estimates for their particular application based on the strengths and limitations discussed here. We hope that users will provide feedback to the NCI on the uses of these estimates. While these estimates may have great utility in local and regional cancer control planning, they should be supplemented with local knowledge and information when available. Feedback is greatly appreciated, both in terms of the global utility of these estimates, as well as local anomalies, and can be given by emailing the Small Area Estimates Web Staff.

The estimates presented here are based on NHIS and BRFSS data and an underlying model based on the demographic profile of each geographic area. When there is sufficient NHIS and BRFSS data for a specific geographic area, the combined estimates depend mainly on the available data from that geographic area. However, for areas with little or no NHIS and/or BRFSS sample, the estimates increasingly depend on using the demographic model to produce estimates for areas with "similar" profiles from across the country in terms of their covariates. These latter estimates cannot capture unique characteristics of the county not represented by the demographic profile, and also cannot capture specific cancer control programs that may have been implemented to increase screening rates or decrease smoking rates in that area.

Based on model assumptions, the state model-based estimate generally improves with increasing NHIS state sample size. When the NHIS state sample is small, the model-based estimate depends heavily on the covariates (e.g., economic, educational, demographic, etc.). Additionally, the model assumes that the impact of the covariates upon outcomes is similar across all states. For example, since Alaska has the smallest state sample size, and because of how much it differs from states in terms of sample size, population density, remoteness, age distribution, and other factors, the impact of covariates on Alaska's estimates for health related outcomes may differ from those in the lower 48 states.

Model assumptions are necessary to correct for nonresponse and non-coverage biases and to smooth the estimates. The more the method smoothes the estimates (to reduce variance) and corrects for biases, the more model assumptions are necessary. We feel that the assumptions made in our model-based approach are reasonable and are sufficient to address the two potential sources of bias and to reduce the variability in the "BRFSS direct" estimates.