Statistical Policy Working Paper 3 - An Error Profile: Employment As Measured By The Current Population Survey

Federal Committee on Statistical Methodology
Office of Management and Budget

Statistical Policy Working Paper 3 - An Error Profile: Employment As Measured By The Current Population Survey

Statistical Policy Working Papers are a series of technical documents prepared under the auspices of the Office of Federal Statistical Policy and Standards. These documents are the product of working groups or task forces, as noted in the Preface to each report.

These Statistical Policy Working Papers are published for the purpose of encouraging further discussion of the technical issues and to stimulate' policy actions which flow from the technical findings. Readers of Statistical Policy Working Papers are encouraged to communicate directly with the Office of Federal Statistical Policy and Standards with additional views, suggestions, or technical concerns.

Office of Joseph W. Duncan Federal Statistical Director Policy and Standards Statistical, Policy Working Paper 3 An Error Profile: Employment As Measured By The Current Population Survey Prepared by Camilla A. Brooks and Barbara A. Bailar, U.S. Bureau of the Census Subcommittee on Nonsampling Errors Federal Committee on Statistical Methodology U.S. DEPARTMENT OF COMMERCE Juanita M. Kreps, Secretary Courtenay M. Slater, Chief Economist Office of Federal Statistical Policy and Standards Joseph W. Duncan, Director Issued: September 1978 Office of Federal Statistical Policy and Standards Joseph W. Duncan, Director George E. Hall, Deputy Director, Social Statistics Gaylord E. Worden, Deputy Director, Economic Statistics Maria E. Gonzalez, Chairperson, Federal Committee on Statistical Methodology Preface This working paper was prepared by Camilla A. Brooks and Barbara A. Bailar, U.S. Bureau of the Census, members of the Subcommittee on Nonsampling Errors, Federal Committee on Statistical Methodology. The Subcommittee was, chaired by Monroe G. Sirken, National Center for Health Statistics, Department of Health, Education and Welfare. Members of the Subcommittee are listed on the following page. The Subcommittee considered various measures of the errors in survey results. One of the alternatives is the preparation of error profiles, that is, a systematic and comprehensive account of survey - operations that yield survey results. The errors in those results are discussed in the error profile. To illustrate the error profile approach, the Subcommittee decided to prepare such a profile for employment statistics based on the Bureau of the Census' Current Population Survey. We expect that the error profile approach will prove useful to both users and producers of statistics. Thus, it should help to enhance the users' appreciation of the limitations of the statistics. In addition, an error profile may guide the producers in their, efforts to identify those survey Operations which need to be re- designed and/or controlled better in order to improve the quality of the survey results. We hope that in the future error profiles will be prepared for other important Federal statistical series. iii Subcommittee on Nonsampling Error Monroe Sirken (Chair) National Center for Health Statistics Barbara Bailar Bureau of the Census Camilla Brooks Bureau of the Census John Cremeans Bureau of Economic Analysis Tore Dalenius (Consultant) Brown University and Stockholm University Richard Deighton U.S. Postal Service James Duffet U.S. Postal Service Maria Gonzalez (ex officio) Office of Federal Statistical Policy and Standards Tom Herzog Social Security Administration Alexander Korns Bureau of Economic Analysis Lillian Madow Bureau of Labor Statistics D. H. McElhone Civil Service Commission Ron Poland U.S. Postal Service Morton Raff Bureau of Labor Statistics Jack Scharff Health Care Finance Administrator Frederick Scheuren Social Security Administration Otto Schwartz Internal Revenue Service Gary Shapiro Bureau of the Census Joel Stubbs Internal Revenue Service Robert Tortora, U.S. Department of Agriculture Rolf Wulfsberg National Center for Education Statistics Acknowledgements This paper was written as a result of the work of the Subcommittee on Nonsampling Errors of the Federal Committee on Statistical Methodology. The authors wish to thank both the members of the Subcommittee and the full Committee for their suggestions, comments and support in writing this paper. However, it represents the compilation of a great deal of work by Bureau of the Census staff members. Many of these persons are listed in the references and are referenced in the text. The authors acknowledge their vast contributions to this paper. Though many persons who helped make this paper possible are listed in the references, not a persons who work with the Current Population Survey (CPS) author papers or memoranda. We have, listed below persons who are involved in the various survey operations and we express our appreciation to them. Several,persons listed also gave the authors helpful comments and suggestions, made available requested information, and suggested sources of additional information. Thomas Plewes of the Bureau of Labor Statistics and Estela Dagum of Statistics Canada were the major authors of Section V.B. on "Seasonal Adjustment". Specification of Definitions Earle Gerson, Stanley Greene, Gregory Russell, and Marvin Thompson of the Demographic Surveys Division; Arthur Young and Aneda France of Housing Division; Meyer Zitter, Murray Weitzman, Charles Johnson, Roger Herriot, and Arno Winard iv of Population Division; Charles Jones, Gary Shapiro, Maurice Kniceley, and Irene Montie of Statistical Methods Division. Sampling Design Gary Shapiro, Maurice Kniceley, Margaret Schooley, Christina Gibson, Wayne St. John, Charles Jones, David Diskin, Leonard Baer, Christine Jorgenson, Jerome Roth, and Peter Bounpane of Statistical Methods Division. Data Collection Procedure Gregory Russell, Ronald Dopkowski, George Gray, and Marvin Thompson of the Demographic Surveys Division; Marvin Postma, Richard Bitzer, Jon Spendlove, William Clark, and Lincoln Steigerwalt of Field Division. Questionnaire Design Gregory Russell, Ronald Dopkowski, Paul Michalak, and Ronald Taylor of Demographic Surveys Division; Erne Wilkins of Engineering Division; and Irene Montie, John Paletta, and Tacho Aguilar of Statistical Methods Division. Data Collection Staff-Regional Offices Joseph R. Norwood, Charlotte, North Carolina; Robert G. McWilliam, Detroit, Michigan; Rex L. Pullin, Kansas City, Missouri; John E. Tharaldson, Seattle, Washington; Richard C. Burt, Denver, Colorado; C. Michael Long, Los Angeles, California; Porter S. Rickley, Philadelphia, Pennsylvania; Percy R. Millard, Dallas, Texas; Stanley Moore, Chicago, Illinois; John C. Cullinane, New York, New York; Arthur G. Dukakis, Boston, Massachusetts; Thomas W. McWhirter, Atlanta, Georgia; and the 1300 interviewers and 125 Supervisory Field Representatives throughout the country. Interviewer Training Paul Michalak, Ronald Taylor, Kenneth Riccini, Kathryn Creighton, and Gregory Weyland of Demographic Surveys Division; Lincoln Steigerwalt, Lynn Minneman, Sally Spurlock, and John Mahan of Field Division; Irene Montie, John Paletta, and Tacho Aguilar of Statistical Methods Division. Quality Control of Field Work Leo Schilling, Lincoln Steigerwalt, Marvin Postma, William Clark, Richard Bitzer, and Jon Spendlove of Field Division; Irene Montie, John Paletta, Tacho Aguilar, John Linebarger, Irwin Schreiner, Douglas Moye, and Dorcas Graham of Statistical Methods Division. Data Input Operations Eleanor Cooper, Ralph Hughes, Harley Bean, and Lowell Wrucke of the Demographic Surveys Division; Erne Wilkins of Engineering Division; and Kathern Clay, Patricia Clark, and Harold Garwood of the Data Preparation Division in Jeffersonville, Indiana. Cleaning, Editing, Imputation Gregory Russell, Lowell Wrucke, Carl Jablin, Maxine Perry, and Philip Hampton of the Demographic Surveys Division; Gary Shapiro, Maurice Kniceley, Margaret Schooley, and Christina Gibson of Statistical Methods of Division. Weighting Procedures Gregory Russell and Lowell Wrucke of Demographic Surveys Division; Gary Shapiro, Maurice Kniceley, Margaret Schooley, Christina Gibson, Leonard Baer, Carrie Pasewark, and Pearl Deimel of Statistical Methods Division. Estimation Procedure Charles Jones, Gary Shapiro, Maurice Kniceley, Robert Jewett, George Train, and Larry Cahoon of Statistical Methods Division; Thomas Plewes of the Office of Current Employment Analysis of the Bureau of Labor Statistics; and Estela Dagum of Statistics Canada. Analysis and Publication Diana Harley of Statistical Methods Division; John Bregger of the Division of Employment and Unemployment Analysis at the Bureau of Labor Statistics and members of his staff. The authors also thank other persons not directly involved with the CPS, but who contributed to their, understanding of various aspects of the CPS and gave useful comments on the paper. These persons are Nash Monsour of Business Division; Carol v Corby and Leroy Bailey of the Research Center for Measurement Methods; Dennis Schwanz, David Bateman, William MacKenzie, Mason Malmuth, James Dinwiddie, and Charles Edwards of Statistical Methods Division; Donny Rothwell, Anitra Rustemeyer of the Statistical Research Division and Harold Nisselson, Associate Director for Statistical Standards and Methodology. In addition, the authors would like to express their gratitude to Christine Walker, Myrtle Rice, and Karen Scott of the Research Center for Measurement Methods (RCMM) for their help in the preparation of tables for this report; and to Patricia Petrick and Mary Jane St. John, also of RCMM, for the typing, reproduction, and distribution of this paper throughout the many phases of its development. The authors extend special thanks to Robert Hanson, formerly of the Statistical Research Division, who was kind enough to allow them the use of a draft copy of the revision of Technical Paper No. 7 which is now published as Technical Paper No. 40 under the title The Current Population Survey: Design and Methodology. A considerable part of the survey description was taken from this work, and it served as a guide to the CPS survey operations throughout the compilation of this paper. Indeed, without it this paper would have taken a great deal more time and would probably have suffered in both accuracy and comprehension. vi Authors' Preface The Subcommittee on Nonsampling Error of the Federal Committee on Statistical Methodology decided to illustrate the ways in which nonsampling error could affect survey statistics by constructing "error profiles". An error profile has as its objective the listing of the survey operations with the investigation of the potential sources of error for each operation. Ideally, the impact of the errors on the survey statistics would be presented. The ideal is rarely possible because the measurement of nonsampling errors is rarely undertaken. This error profile describes the potential sources of error in the Current Population Survey (CPS) as they affect the national employment statistics. The sample design considered in the profile does not include the supplementation for improved state estimates that will be used in obtaining the national estimates of employment as of January 1978. The purposes of this document are as follows: 1. To illustrate how an error profile is-created in an effort to encourage government statisticians to provide error profiles . for the major recurrent survey statistics; 2. To compile in a single document the sources of error and the information that is available about these sources of error and their impact; 3. To illustrate the need for controlled experiments to measure nonsampling errors because of the lack of knowledge of the impact of these errors; 4. To stimulate development of a mathematical model that will reflect the ways in which the errors from different sources interact. The Current Population Survey was selected to illustrate the creation of an error profile for many reasons. It is a survey with a long history, and subsequently much has been written about it. Additionally, a considerable amount of research has been done on the survey methods used. Finally it is noteworthy as a survey that produces data on important statistics. We have chosen to focus on "employment" to narrow the scope of the profile. Though we have tried to follow through the complete survey process, and find all relevant information either through written memoranda or discussions with persons responsible for certain operations, there are undoubtedly gaps 'in our information. A sincere vote of thanks goes to all those who helped us compile the information for this profile. Any errors in interpretation or any gaps in the reports are our responsibility. vii Members of the Federal Committee on Statistical Methodology, (September 1977) Maria Elena Gonzalez (Chair) Office of Federal Statistical Policy and Standards (Commerce) Barbara A. Bailar Bureau of the Census (Commerce) Norman D. Beller Economics, Statistics, and Cooperatives Service (Agriculture) Barbara A. Boyes Bureau of Labor Statistics (Labor) Edwin J. Coleman Bureau of Economic Analysis (Commerce) John E. Cremeans Bureau of Economic Analysis (Commerce) Marie D. Eldridge National Center for Education Statistics (HEW) Fred Frishman Internal Revenue Service (Treasury) Thomas B. Jabine Social Security Administration (HEW) Charles D. Jones Bureau of the Census (Commerce) Alfred D. McKeon Bureau of Labor Statistics (Labor) Harold Nisselson Bureau of the Census (Commerce) Monroe G. Sirken National Center for Health Statistics (HEW) Wray Smith Office of the Assistant Secretary for Planning and Evaluation (HEW) viii Table of Contents Page Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . .iv Authors' Preface . . . . . . . . . . . . . . . . . . . . . . . . . vii List of Figures. . . . . . . . . . . . . . . . . . . . . . . . . . xii List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . xiv Key Abbreviations. . . . . . . . . . . . . . . . . . . . . . . . .xvii Executive Summary. . . . . . . . . . . . . . . . . . . . . . . . xviii Chapter I. Introduction. . . . . . . . . . . . . . . . . . . . . 1 A. Objectives of the Survey . . . . . . . . . . . . . 1 B. Specifications of the Survey . . . . . . . . . . . 1 Chapter II. Sampling Design and Implementation. . . . . . . . . . 3 A. The Frame. . . . . . . . . . . . . . . . . . . . . 3 The Basic Frame . . . . . . . . . . . . . . . . . 3 Limitations of the Frame . . . . . . . . . . . . . 4 1. Structure Undercoverage/Overcoverage. . . . . 4 a. Permit lag universe. . . . . . . . . . . 4 b. Time lag between permit issuance and chance of entering sample . . . . . . . . . 4 c. Nonpermit issuing TAR ED's . . . . . . . 4 d. Incompleteness of permit universe. . . . 4 e. Undercoverage of special places-mobile homes . . . . . . . . . . . . . . . . . . . 4 f. Other structure misses . . . . . . . . . 5 g. Structure overcoverage . . . . . . . . . 5 2. Within Household Misses . . . . . . . . . . . 5 3. The Effect of the CPS Undercoverage on the employment Statistics . . . . . . . . . . . . . . . . 5 Attempts to Strengthen the Frame . . . . . . . . . . . . . 7 B. Procedure for Sample Selection . . . . . . . . . . . . . . 7 The Sample Selection Procedure . . . . . . . . . . . . . . 7 1. The Selection of the Primary Sample Units (PSU's) . . . 7 2. Selection of Sample Households. . . . . . . . . . . . . 8 3. Rotation of the Sample. . . . . . . . . . . . . . . . . 9 4. Replacement of PSU's. . . . . . . . . . . . . . . . . . 9 Potential Sources of Error in CPS Sampling Procedure . . . 9 C. Quality Control of Sample Procedure. . . . . . . . . . . .11 Selection of Sample PSU's. . . . . . . . . . . . . . . . .11 Within PSU Sampling Process. . . . . . . . . . . . . . . .11 Chapter III. Observational Design and Implementation . . . . . . . .13 A. Basic Data Collection Procedure . . . . . . . . . . . .13 Listing and Sampling. . . . . . . . . . . . . . . . .13 ix Page 1. Listing, in Address, Permit, and Cen-Sup Segments or Groups of USU's. . . . . . . . . . . . . . . . . . . . . .13 2. Listing in Area Segments. . . . . . . . . . . . . . . .13 3. Listing in Special Place Segments . . . . . . . . . . .14 4. Updating. . . . . . . . . . . . . . . . . . . . . . . .14 Conducting the Interview. . . . . . . . . . . . . . . . . . . .14 1. General Procedure . . . . . . . . . . . . . . . . . . .14 2. Mode of Interview . . . . . . . . . . . . . . . . . . .15 3. Noninterviews . . . . . . . . . . . . . . . . . . . . .15 Potential Sources of Error in the CPS Data Collection Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . .15 1. Listing by Observation in Area Segments . . . . . . . .15 2. Problems with the Year Built Procedure. . . . . . . . .16 3. Determination of Race of Household Head for Type A Noninterview Units. . . . . . . . . . . . . . . . .16 4. Effect of Mode of Interviewing on the Employ- ment Estimate. . . . . . . . . . . . . . . . . . . . . . .16 5. The Effect of Proxy Respondents on the Data . . . . . .17 B. Questionnaire Design and Instructions. . . . . . . . . . .18 The Questionnaire and Interviewer Instructions . . . . . .18 Potential Sources of Error in the Questionnaire Design . .20 C. Data Collection Staff. . . . . . . . . . . . . . . . . . .21 Organization and Description of the Data Collection Staff. . . . . . . . . . . . . . . . . . . . . . . . . . .21 Potential Sources of Error Associated With Interviewers . . . . . . . . . . . . . . . . . . . . . . .22 D. Training of the CPS Interviewers . . . . . . . . . . . . .28 The Training Procedure . . . . . . . . . . . . . . . . . .28 1. Initial Training. . . . . . . . . . . . . . . . . . . .28 2. Refresher Training. . . . . . . . . . . . . . . . . . .28 3. Special Training. . . . . . . . . . . . . . . . . . . .28 4. Determining when an Interviewer is Trained. . . . . . .28 Limitations in the Training Procedure. . . . . . . . . . .28 E. Development of Quality Control Design. . . . . . . . . . .29 Coverage Check of Sampling Units . . . . . . . . . . . . .30 Questionnaire Checks . . . . . . . . . . . . . . . . . . .30 Interviewer Observation. . . . . . . . . . . . . . . . . .30 Reinterview. . . . . . . . . . . . . . . . . . . . . . . .30 1. The Basic Procedure . . . . . . . . . . . . . . . . . .30 2. Reconciliation. . . . . . . . . . . . . . . . . . . . .34 3. Results from Reinterview. . . . . . . . . . . . . . . .31 Limitations of the Quality Control Procedure. . . . . . . . . .32 1. Quality Control of Telephone Interviewing . . . . . . .32 2. The Reinterview Procedure . . . . . . . . . . . . . . .32 a.Independence. . . . . . . . . . . . . . . . . . . .33 b. Reinterview nonresponse rate . . . . . . . . . . .33 c. Coverage check . . . . . . . . . . . . . . . . . .33 d. 80 Percent vs. 20 ]Percent Sample. . . . . . . . .34 Chapter IV. Data Processing. . . . . . . . . . . . . . . . . . . . .37 A. Mcrofilming/FOSDIC . . . . . . . . . . . . . . . . . . . . .37 The FOSDIC/Mcrofilming Procedure. . . . . . . . . . . . . . . .37 x Page Potential Sources of Errors in the FOSDIC/Microfilming, Procedures. . . . . . . . . . . . . . . . . . . . . . . . . . .37 1. Results from the Data Acceptance Runs . . . . . . . . .39 2. Results of Test Run of Blank CPS Questionnaires . . . .39 3. Variations in FOSDIC Scanning of Monthly Quality Control Samples. . . . . . . . . . . . . . . . . . . . . . . . . 4.1 4. CPS/FOSDIC Study. . . . . . . . . . . . . . . . . . . .41 B. Editing and Imputation . . . . . . . . . . . . . . . . . . .46 Procedure for Editing and Imputation for Missing Values . . . .46 Potential Sources of Errors Arising from the Editing and Imputation Procedure. . . . . . . . . . . . . . . . . . . . . .48 C. Quality Control. . . . . . . . . . . . . . . . . . . . . . .48 Quality Control of Microfilming/FOSDIC Procedure. . . . . . . .48 1 . Design of the Questionnaire. . . . . . . . . . . . . .50 2. Selection of Paper. . . . . . . . . . . . . . . . . . .50 3. Printing. . . . . . . . . . . . . . . . . . . . . . . .50 4. FOSDIC Scan . . . . . . . . . . . . . . . . . . . . . .51 5. Cameras . . . . . . . . . . . . . . . . . . . . . . . .51 6. Film. . . . . . . . . . . . . . . . . . . . . . . . . .51 7. Production Scanning (FOSDIC). . . . . . . . . . . . . .51 Quality Control of Editing and Imputation . . . . . . . . . . .51 Chapter V. Estimation. . . . . . . . . . . . . . . . . . . . . .53 A. Weighting Procedure. . . . . . . . . . . . . . . . . . . . .53 Specification of Weighting Procedure . . . . . . . . . . .53 Potential Sources of Error Caused by the Weighting Procedure. . . . . . . . . . . . . . . . . . . . . . . . .57 B. Specification of Estimation Procedure. . . . . . . . . . . .59 Composite Estimation . . . . . . . . . . . . . . . . . . .59 1. Description. . . . . . . . . . . . . . . . . . . .59 2. Potential Sources of Error in the Composite Estimation Procedure . . . . . . . . . . . . . .61 Seasonal Adjustment. . . . . . . . . . . . . . . . . . . .65 1. Description. . . . . . . . . . . . . . . . . . . .65 2. Source of error in Seasonal Adjustment . . . . . .66 a. Model Selection . . . . . . . . . . . . . . .67 b. Trend-Cycle Moving Average. . . . . . . . . .67 c. Final Seasonal Factor Moving Average Curve . . . . . . . . . . . . . . . . . . . .67 d. Forecasting Seasonal Factors for the Current Period . . . . . . . . . . . . . . . . . . . . .67 3. Estimating the Error in Seasonal Adjustment. . . .68 C. Specification of Estimation Procedures for Errors. . . . . .69 Estimation of Errors . . . . . . . . . . . . . . . . . . .69 1. The CPS Variance Estimation Procedure. . . . . . .69 2. Generalizing Estimated Sampling Errors . . . . . .70 3. Simple Response Variance and Response Bias . . . .70 Potential Problems in the Estimation of Errors. . . .72 D. Quality Control of the Estimation Procedure. . . . . . . . .76 Quality Control of the Weighting and Estimation Procedure . . . . . . . . . . . . . . . . . . . . . .76 xi Page Quality Control of the Keyfitz Estimate and Variance Program . . . . . . . . . . . . . . . . . . . . . . .76 Chapter VI. Analysis and Publication . . . . . . . . . . . . . . . .77 A. Description . . . . . . . . . . . . . . . . . . . . . .77 B. Potential Sources of Error in Analysis and Publication.77 Chapter VII. Conclusion . . . . . . . . . . . . . . . . . . . . . .79 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81 xii List of Figures Figure Page 1. CPS Rotation Chart. . . . . . . . . . . . . . . . . . . . . . . .10 2. Excerpt from CPS Questionnaire. . . . . . . . . . . . . . . . . .19 3. Excerpt from CPS Questionnaire Showing Index Marks and Marking Circles . . . . . . . . . . . . . . . . . . . . . . . . . . . .38 4. Control Chart for Percentage of Film Associated Pre-Computer Edit Rejects (Computed from January 1974-December 1975 Rejects). . . . . . . . . . . . . . . . . . . . . . . . . . . .40 5. Control Chart for Percentage of Spurious Pickups Per Total Items Tested Prior to Processing (January 1976-December 1976) .43 6. Control Chart for Percentage of Spurious Pickups Per Total Items Tested during Processing (January 1976-December 1976) . . . . .44 7. CPS Noninterview Adjustment Cells . . . . . . . . . . . . . . . .54 8. Noninterview Adjustment in Mixed-Race Households. . . . . . . . .58 xiii List of Tables Table Page 1. CPS Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Labor Force Estimates Adjusted for CPS-Census Undercoverage Compared to March 1973 Estimates at Two Stages of Estimation. . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3. Persons Employed as Percent of Labor Force for Two Stages of March 1973 Estimation Compared to Percentage Adjusted for Census-CPS Undercount. . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4. Effect of Omitted Persons on Labor Force Estimates, Comparability and Poverty Neighborhood Assumptions, 1967 Monthly Averages . . 7 5. Percent Telephone Interviews by Month in Sample-1976 Average. . .17 6. Percent Telephone Interviews by Month in Sample for Employed Persons-December 1973 . . . . . . . . . . . . . . . . . . . . .17 7. Total Employed as Measured by Three Procedures in Methods Test. .18 8. Percent Difference in Total Employed as Measured by Self and Proxy Respondent Procedure. . . . . . . . . . . . . . . . . . . . . .18 9. Frequency of Rewording of Labor Force Questions . . . . . . . . .20 10. Rotation Group Indices for Employment Items for Two Time Periods, 1968-69 (TI) and 1970-72 (T2). . . . . . . . . . . . . . . . . . . .21 11. Summary of Characteristics of CPS Interviewers, March 1975 . . .22 12. Number and Distribution of Total, Noninterview and Interview Units in A and C Design CPS Sample. . . . . . . . . . . . . . .23 13. Distribution of CPS Interviewer Type A Noninter-view Rates, Average July 1974-June 1975 . . . . . . . . . . . . . . . . . .23 14. Distribution of CPS Interviewer Questionnaire Edit Error Rates, Monthly Average July 1974 Through June 1975 . . . . . . . . . .24 15. Distribution of CPS Interviewer Production Ratios, Averages July 1974 Through June 1975. . . . . . . . . . . . . . . . . . . . .25 16. Distribution of Interviewers' Monthly Gross Coverage Error Rates in the Current Population Survey April 1973-December 1974. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 17. Distribution of Interviewers' Gross Content Error Rates in the Current Population Survey April 1973-December 1974. . . . . . .25 18. Responsibility for Differences Between Responses From CPS Original Interview and Reinterview: 1959 to 1966. . . . . . . .26 19. Estimates of the Average Ratio Over Six Months of the Correlated Component of Response Variance to the Sampling Variance for Employment Items, and the Estimated Standard Deviation of the Estimated Ratios . . . . . . . . . . . . . . .27 20. Tolerance Table of Acceptable Limits . . . . . . . . . . . . . .31 21. Summary of Percent of Persons Employed as Measured in the Original CPS Interview for the Reinterview Subsample and as Measured by the Reinterview After Reconciliation, 1956-1976 . .32 22. Annual Interviewer Error Rates . . . . . . . . . . . . . . . . .32 23. Noninterview Misclassification Rates. . . . . . . . . . . . . .32 24. Percent Net Change in Area and B Segments: April to September 1966, October 1966, November 1966 to May 1967, and June 1967. .33 25. Reinterview Classification of Units Originally Classified as Noninter-view . . . . . . . . . . . . . . . . . . . . . . . . .34 26. Percent Difference Between Reinterview and Original . . . . . .35 xiv Page 27. Distribution of Questionnaires Rejected by FOSDIC by Reason for Rejection, January 1976 to December 1976. . . . . . . . . .39 28. Number and Percentage of Spurious Pickups Detected on Print Sample Documents Tested on FOSDIC during CPS Processing (January 1976-December 1976). . . . . . . . . . . . . . . . . .42 29. Variation in the FOSDIC System . . . . . . . . . . . . . . . . .45 30. Some Results of the FOSDIC Experiment. . . . . . . . . . . . . .45 31. Measured Reflectance of Ten Questionnaire Positions. . . . . . .45 32. Results of Test on Opacity and Brightness of CPS Questionnaires. . . . . . . . . . . . . . . . . . . . . . . . .46 33. The Possible Item Combinations Used in Determining the Existence of A Legitimate Person's Record . . . . . . . . . . .47 34. Summary of Errors Due to Omissions or Inconsistencies by Item Number, February-December 1975. . . . . . . . . . . . . . . . .49 35. Average Frequency With Which A Value From the Same CPS Record is Used for Imputation for Item Nonresponse . . . . . . . . . .50 36. CPS First-Stage Ratio Adjustment Factors, November 1974-March 1975. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54 37. Civilian Noninstitutional Population and Labor Force by Sex and Age, Using Old and New Method of Estimating Civilian Population; January 1974. . . . . . . . . . . . . . . . . . . .55 38. Civilian Noninstitutional Population and Labor Force for Negro and Other Races by Sex and Age Using Old and New Method of Estimating Civilian Population; January 1974 . . . . . . . .55 39. Employment Status by Sex and Age Using Old and New Method of Estimating Civilian Population, January 1974. . . . . . . . . .56 40. CPS Second Stage Ratio Adjustment Factors for Total Population by Age, Race and Sex. . . . . . . . . . . . . . . . . . . . . .57 41. Maximum, Minimum and Average Weights for Records in 13 Relationship Categories, March 1975 . . . . . . . . . . . . . .57 42. Mixed-Race Households in March 1975 CPS Sample . . . . . . . . .59 43. Separate Second Stage Ratio Adjustment Factors Applied to Persons of "Other Races" by Age' Sex, and Month in Sample, March 1975. . . . . . . . . . . . . . . . . . . . . . . . . . .60 44. 1975 Average of Unbiased, Ratio, and Composite Estimates of Selected Employment Items . . . . . . . . . . . . . . . . . . .61 45. Ratios of the Unbiased and Ratio Estimates to the Composite Estimate for Selected Employment Items. . . . . . . . . . . . .62 46. Rotation Group Bias Indexes in the Current Population Survey for Employment Status by Sex, Five-Year Averages, 1968-1972 . .63 47. Comparison of Estimated Mean Square Errors for Ratio and Composite Estimates for 1975. . . . . . . . . . . . . . . . . .64 48. Original, Current, and Historically Seasonally Adjusted Data for June,1974 . . . . . . . . . . . . . . . . . . . . . . . . . . .68 49. Components of Variance, CPS Composite Estimates of Level, Annual Average, 1975 . . . . . . . . . . . . . . . . . . . . . . . . .69 50. Variance of Composite Estimate of Level and Variance Ratio for Selected Estimators-Averages, 1975. . . . . . . . . . . . . . .70 51. Standard Errors for the CPS Estimates of Monthly Level- Employment and Unemployment . . . . . . . . . . . . . . . . . .71 52. Percent in Class (Reinterview Estimate After Reconciliation), Net Difference Rate (After Reconciliation) and the Index of Inconsistency (Before Reconciliation) by Quarter, 1974-1976 . .72 xv Page 53. REL-MSE of Estimates of Variance Based on Taylor Approximations. . . . . . . . . . . . . . . . . . . . . . . . .73 54. Serial.Correlation in Sampling Error . . . . . . . . . . . . . .73 55. Sampling Errors for CPS Seasonally Adjusted Series . . . . . . .73 56. Between PSU Variances for Selected Employment Items for 1975 . .75 xvi Key Abbreviations 1. AHS Annual Housing Survey 2. BLS Bureau of Labor Statistics 3. Cen-Sup sample Census Supplemental sample 4. CPS Current Population Survey 5. DPD Data Preparation Division 6. ED Enumeration District 7. FOSDIC Film Optical Sensing Device for Input to Computers 8. NSR nonself-representing 9. PSU primary sampling unit 10. SMSA standard metropolitan statistical area 11. SR self-representing 12. TAR tape address register 13. USU ultimate sampling unit xvii Executive Summary An objective of this error profile is to make a comprehensive list of the survey operations with the documentation of what is known about each survey operation as a potential source of nonsampling error. When possible, the magnitude of the error is given. When no data are available about the source of error and possible impact, this is also noted. Each stage of the survey from the choice of the sampling frame to the analysis of the data in a final publication is described. Associated with each stage of the survey is a description of the process and then a discussion of what is known about possible errors or biases arising from the process. The stages examined are as follows: 1. Sampling design a. frames b. sample selection c. quality control of sampling process 2. Observational design a. data collection procedure b. questionnaire design c. data collection staff d. interviewer training e. quality control of field work 3. Data preparation design a. data input operations b. cleaning, editing and imputation c. quality control of data processing 4. Production of estimates a. weighting procedure b. estimation procedure c. quality control of estimation procedure 5. Analysis and publication Certain areas of the survey operations have received a great deal of attention and much is known about the limitations inherent in these operations. One such area is the sample design and implementation. This area is discussed in the first section of the paper. The coverage bias resulting from the sampling frame not completely covering the entire universe of households is estimated to be of the order of 3 percent. However, in addition to missed households, there are missed persons within enumerated households. These missed persons are mo re likely to be young, male, and to be black or of races other than white. There is an adjustment procedure to take care of part of this bias. The full impact of the undercoverage in the 1970 census and the additional undercoverage in the CPS is not known. The next survey operation discussed is the actual fieldwork including the design of the data collection procedure, the design of the questionnaire, the data collection staff, the training program for interviewers, and the quality control program for the data collection. Several potential sources of error and their effects are discussed. For example, the effect of using proxy respondents in the collection of labor force data is discussed. Several years ago some experiments were conducted outside of the CPS to measure the effect of using proxy respondents. These measures are given, though they may not be strictly applicable to the CPS. Also, the impact of xviii interviewer variability is discussed, along with the limited amount of information available. A recent experiment conducted to evaluate the interviewer training procedures is described. As is true of almost every data collection process, there is the potential for several errors. However, the studies designed to measure the errors give fragmentary results. Although the fieldwork on this survey is probably studied more than any other government survey, the knowledge of the effects of nonsampling error arising from the data collection procedure is sketchy. Another major survey operation, that of data processing, includes the data input, and the cleaning, editing, and imputation procedures. In this area, extensive information is available about the data input process and an estimate of any errors of the process is made. The cleaning, editing, and imputation process is described. This process is intended to alleviate the impact of errors arising from the fieldwork. Therefore, one would not expect the biases arising from these errors to be additive with.the biases arising from fieldwork. Although these procedures may reduce the overall bias, they may also induce other biases. Some of the ways the cleaning, editing, and imputation impact on the final data are described, but no quantifiable information is available. Finally, the estimation procedure is discussed. Not only is the estimation procedure for employment reviewed, but also the estimation procedure for the variances of the employment statistics, and the estimation procedure for some of the nonsampling errors. Limited information is available on the impact of the estimation procedures. Nonsampling errors may also occur in the analysis and publication of the data. A brief discussion of this phase of the survey operation is included. Again, not much is known about the impact of errors in this procedure on the many uses made of these data. Though the Bureau makes frequent use of a mean-square error model to describe the combined effects of sampling and nonsampling errors on estimated means and totals, no research studies have been done that permit the estimation of the bias term. e do not know how the nonsampling errors arising from different sources fit together. In some cases, they are clearly additive; in other cases. they may not be. It is quite possible that some of the errors go in different directions so that some may have the effect of overstating employment while others have the effect of understating employment. Thus, at the end of the description of nonsampling errors, we are left with the important question: how do these errors interact and, what is the magnitude of the bias term in the mean-square error? This error profile, even with its limitations, can be used as a framework for a systematic approach to evaluate the different potential sources of error. It may also be used as an illustration of the need for controlled experiments to enable us to quantify the errors and learn more about their interaction. xix Introduction A. Objectives of the Survey Prior to the 1930's, there w ere no direct measures of the number of jobless persons. Because of the mass unemployment during the economic depression of the early 30's, the need for statistics became urgent, and widely conflicting estimates based on a variety of indirect techniques began to appear. As a result of dissatisfaction with these methods, experimentation with direct surveys of the population began. In March 1940, the Works Progress Administration initiated the Sample Survey of Unemployment on a monthly basis. This survey was the forerunner to the present program. The primary purpose of the Current Population Survey (CPS) is to obtain estimates on a monthly basis of employment, unemployment, and other characteristics of the general labor force, of the population as a whole, and of various subgroups of the population. This, report focuses on "employment". B. Specifications of the Survey The Current Population Survey is restricted to the civilian no- ninstitutional population age 14 and over. Although the official tabulations have been restricted since.1967 to data for persons 16 years of age and over, the CPS labor force questions, including those concerned with "employment", are asked of persons 14 years of age and over. The target population encompasses every person 14 years of age and over who is not institutionalized or in the military. The CPS utilizes a household sample which represents the universe of all households in the United States and also includes those persons living in nonhousehold units, such as dormitories, flophouses, budhouses, and the like. In using a household surveY, an implicit assumption is made that each person 14 years of age and over will be uniquely associated with either a household or one of the nonhousehold units mentioned. Because of the many uses of the employment data no single definition is appropriate for every situation. The criteria used in classifying persons as employed are as follows (see BLS Report No. 463-Series P-23): 1. All those people who worked, for at least one hour, as paid employees or in their own business, profession, or on their own farm; 2. All those people who worked 15 hours or more as unpaid workers in a family-operated enterprise; 3. All those persons who did not work but bad jobs or businesses from which they were temporarily absent because of illness, vacations, labor-management disputes or other reasons. No distinction is made whether or not they were paid by their employers for the time off and whether or not they were looking for other jobs. Employed persons, even if they hold more than one job, are counted only in the job at which they worked the greatest number of hours. Employed citizens of foreign countries, excluding those who live on the premises of an Embassy, are counted in the total. Excluded are persons whose only activity was work around their own homes or volunteer work. In addition to data on employment by age, sex, race, and marital status, the CPS provides data on many other characteristics. For example, separate data are provided for wage and salary workers, the self-employed, unpaid family workers, and total agricultural and nonagricultural employment. In this report, we shall limit the discussion to data on employment by age, sex, and race. We shall focus on estimates of the monthly level of employment. In,order to provide comparable data on a month-by-month basis, stringent guidelines must be followed. Although the survey provides a measure of employment for a given month, data are collected for a specific week. Each month, during the calendar week containing the 19th day, the CPS interviewers ask a series of standard questions on economic activity relating to the preceding week (the calendar week containing the 12th day of the month). All clerical and machine editing and computations are completed in the following two weeks, and the employment figures are released during the first week of the following month. Because of the importance of these estimates, the accuracy of the data must be 1 at a high level. The coefficient of variation on the level of employment has been approximately 0.2 percent for many years. The most pressing constraints on the survey are the timing and the need to present comparable statistics over time with great accuracy. 2 CHAPTER II Sampling Design and Implementation A. The Frame The Basic Frame The frame for this survey is derived from a variety of sources with the main source the 1970 Decennial Census. In the CPS, extensive use is made of the 229,000 enumeration districts (ED's) defined in ad- vance of the census; these are large geographic areas, each containing about 350 housing units on the average. There were three types of ED's in the census, each type identified by the manner of forming the address register. (An address register is a listing by address of each housing unit in the ED.) These are as follows: 1. Tape address register (TAR) ED's, approximately 95,000 in number, in which the address register was created from a computer tape copy of a commercial mailing list and cor- rected by the Post Office and enumerators. 2. Prelist ED's, approximately 26,000 in number, in which the address register was constructed by a listing procedure conducted in advance of the census and corrected by the Post Office and enumerators. 3. Conventional ED's, about 108,000 in number, in which the address register was prepared by the enumerator during the census enumeration. A 1970 census ED is referred to as an address or list ED if the conditions listed below are satisfied. 1. The ED is a TAR ED. 2. The ED is a prelist or conventional ED satisfying both "a" and "b". a. at least 90 percent of the 1970 census addresses within the ED are recorded with complete street name and house number; b. the ED is located in an area which issues building permits. In address ED's the CPS sample is selected from the census address registers' and the resulting sample is referred to as an address sample. All other 1970 census ED's are referred to as area ED's and the enumerator lists the structures in the sample area segments in these ED's about a month before the initial interview. The Census Supplemental sample, referred to as the Cen-Sup sample, is used to cover housing units in address ED's at addresses missed in the census or inadequately described in the address register. To obtain this sample, a sample of address ED's was selected, and a probability sample of blocks or areas within these ED's was selected and canvassed. The resulting addresses were then matched to the census. The complete addresses of unmatched units and those with inadequate addresses were stored in the computer and are sampled on a rotation basis for representation through the life of the current design. These units represent less than one percent of the CPS sample. The frame is further supplemented by the permit universe, consisting of jurisdictions where building permits were required as of January 1, 1970 and where suitable records are maintained. A self- weighting sample of permits issued as of January 1, 1970 or later is then selected from this universe on a periodic basis. Thus new construction is represented in address ED's and permit issuing area ED's by a sample of building permits from permit offices. In nonpermit issuing area ED's, new construction is covered by interviewer listing. The following table (ignoring frame deficiencies) shows the percent of the total CPS sample that comes from each part of the CPS frame as of 1976. Table 1. CPS Frame Percent of Part of Frame CPS Sample Census Address Sample including 2 percent special place1. . . . . . . . . . . . . . . . . . . . . . . . . .65 Area. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24 Supplements to Census Permit Universe . . . . . . . . . . . . . . . . . . . . . . . .10 Cen-Sup Sample. . . . . . . . . . . . . . . . . . . . . . . . . 1 1 A special place ED is an ED consisting entirely of places such as transient hotels, boarding houses, and mobile home parks where the occupants have special living arrangement Special places are also within other ED's. 3 Limitations of the Frame It is known that the frame used for CPS sampling does not fully represent the target population. However, frame deficiencies (excluding within household coverage losses) represent less than 3 percent of the population, though it is concentrated in certain types of units. 1. Structure Undercoverage/Overcoverage a. Permit lag universe In ED's where building permits are issued, housing units completed after the census for which permits were issued before January 1, 1970 are not included in the CPS frame. These units are referred to as the permit lag universe. There is an estimated total of 598,000 units for which permits were issued prior to January 1, 1970 that were completed after April 1970 (MacKenzie, 1977). b. Time lag between permit issuance and chance of entering sample Because of sampling efficiency and data preparation time, there is at present a 5-month lag between the time a permit for new con- struction is issued and the time the unit has a chance of entering the CPS sample. Thus for a short period of time there are units in the new construction universe that may not be represented in the CPS sample. In a study by Linebarger (1975), it was estimated that ap- proximately 12 percent of the units for which building permits were issued were interviewable 4 months after date of issuance; however, this is a cumulative figure so most of these units would not have been interviewable for the entire 4 months. c. Nonpermit issuing TAR ED's A small number of the TAR ED's (approximately 47-50 or about 0.3 percent of all TAR ED's) are in nonpermit areas. Thus new construction which should be represented in the permit universe, is not represented in these ED's (Baer, 1973 and Boisen, 1971 d. Incompleteness of permit universe The permit universe used to select the sample of new construction units is not complete. One of the reasons for this incompleteness is illegal construction; i.e. construction for which the required building permits were not obtained. The undercoverage in the permit universe was estimated to be roughly 2.3 percent for 1976. e. Undercoverage of special places-mobile homes Mobile homes located in address segments are another potential source of coverage loss. Presently, in the CPS there is no general procedure for identifying or representing mobile homes in new mobile home parks, or new mobile homes at large in address ED's at addresses which were nonexistent in the 1970 Census; the permit universe includes regular housing units only. In addition to new mobile homes, the coverage problem of mobile homes in address ED's extends to those occupied at the time of the census but missed in the. census and to those vacant at the time of the census and not counted because of census rules. In area segment ED's in permit areas, new construction has a chance of selection from both the area segment and the permit uni- verse. Therefore, in these ED's new construction units are deleted from area segments by an unduplication procedure referred to as the "Year Built Procedure". However, mobile homes are not included in the permit universe. Therefore, special instructions are specified to include them in the area segment sample. A coverage problem occurs when CPS interviewers apply the "Year Built Procedure" to mobile homes. This can occur when a mobile home "looks like" a regular housing unit. The effect of this error has not been studied yet; however, it probably can be assumed to contribute only a small error to the CPS employment estimate. The coverage improvement program in the October 1976 Annual Housing Survey (AHS) located approximately 300,000 mobile homes previously missed for the period April 1970 October 1976. This improvement program has not yet been included in the CPS so these mobile homes which are eligible for inclusion in the CPS represented mobile homes missed by CPS (MacKenzie, 1977). Though concern is greater for mobile homes, other special places including transient hotels, boarding homes, etc. present some of the same problems as the mobile homes. 4 f. Other structure misses Other problems with the frame in address ED's that are not addressed by the -Census Supplemental sample include coverage of homes moved to a site that did not have an address in the 1970 Census and structures used for nonresidential purposes at the time of the Census and converted to residential use after the census. g. Structure overcoverage Coverage errors sometimes result in overcoverage also, but far less frequently. One example is a new structure built in the same location, with the same address as the old sample structure, and containing the same number of units as the old structure. If the interviewer fails to recognize that the unit is new and interviews at this address, then the units at this structure have two chances of coming into sample. Because it is a new structure, it is also represented in the permit universe in addition to being represented in the regular address sample. This, like many of the errors, occurs infrequently; in addition, the address is usually a one-unit structure rather than multi-unit, which further reduces the impact on the sample. 2. Within Household Misses Within household misses are believed to account for a large percentage of the total undercoverage; however, information on both the extent and the causes of this problem are limited. It is estimated that because of missed structures less than 3 percent of the target population is not included in the frame. Table 40, which shows the ratio of independent estimates of population to those of the Current Population estimates, indicates a coverage problem exceeding this amount. For white males and females the ratios are 1.049 and 1.023, respectively, while for males and females of black and other races the respective ratios are 1.155 and 1.075. (These ratios indicate an undercoverage of 4.9 percent, 2.3 percent, 15.5 percent, and 7.5 percent, respectively.) Further, the independent estimates of the population do not reflect the undercoverage of the census so the within household coverage problem is even greater than indicated by this analysis. Another study. indicated that an estimated 64 percent of blacks missed in the census and 42 percent of per- sons of white and other races missed in the census were missed within units enumerated as occupied in the census or in occupied units enumerated as vacant in the census (Jones and Blass, 1975). Though knowledge of the extent and causes of within household misses is admittedly very limited, there is some knowledge on the subject. Analysts of census undercounts and ethnographers cite that concealment and oversight are two reasons why respondents give incomplete, rosters to interviewers. Both reasons apply more to men than to women and more to poor persons than to nonpoor persons (Korns, 1977). Two small ethnographic studies conducted in the late 60's in (a) an Hispanic neighborhood and (b) a black and Hispanic neighborhood found that many of the households reported in surveys as female headed were actually male-headed. The survey respondents, generally- female, failed to report these men because of fear of loss of economic security. Further, illegal immigrants have strong incentives to conceal their presence in the household and persons loosely attached to households may be unintentionally left off household rosters by the respondents (Korns, 1977). In a Census Bureau study a sample of 710 young men 20-29, mostly black and in poor neighborhoods in an urban area were matched to the 1970 Census; the Bureau found that 23.5 percent of the men were missed or probably missed in the census. The report by Novoa (1971) explores the reasons for these misses which included both oversight and concealment. In addition to within household misses, persons with no attachment to any address present a coverage problem for CPS. 3. The Effect of the CPS Undercoverage on the Employment Statistics Each month CPS estimates of employment are weighted up to independent estimates of the population which are the result of census data carried forward to the present by considering births, deaths, etc. (see Section V.A.). In regard to CPS undercoverage, the accuracy of the CPS employment estimate is affected by the following problems: (a) the independent estimates of the population used to adjust the CPS estimates &f; employment are too low because they are not adjusted for the census undercount which varies by age-sex-race groups and because illegal aliens are not included in the count; 5 (b) the CPS sample misses people it is designed to cover (Korns, 1977). Korns (1977) has done extensive research into the effect of the CPS undercoverage on the employment estimate as part of his comparative study of the cyclical behavior of the CPS measure of employment and that of the Labor Department's payroll (or establishment) survey. He presents evidence where missed persons in enumerated households experience larger cyclical fluctuations in their employment ratios than covered persons of the same age, race, and sex; this suggests that missed persons can have a noticeable effect on employment estimates. A paper by Hirschberg, Yuskavage, and Scbeuren (1977) discusses the impact of alternative adjustment procedures on some selected labor force estimates obtained in the March 1973 CPS. Their three alternative survey estimates are presented in Tables 2 and 3: (a) initial-the March 1973 survey estimate before any adjustment for coverage; (b) standard-the published March 1973 estimate which has been weighted up to the independent estimates of the population; and (c) extended-the survey estimate which has been adjusted to correct for the March 1973 undercoverage. The adjustment procedure makes use of both demographic analysis and administrative records. Under the assumption that the contentions advanced by Hirschberg, et al, are plausible, Table 2 would indicate that the March 1973 published estimate (standard) of total employed persons was understated by 2.5 million. It also shows that the published estimate accounted for only 50 percent of the actual CPS undercoverage. (The difference between the unadjusted initial estimate and the extended estimate was 5.0 million.) Table 3 shows the effect of population undercoverage on the percent of persons employed for total persons and by sex and race. The published estimate of the percent of total persons employed is overestimated by 0.4 percent; Table 2. Labor Force Estimates Adjusted for CPS-Census Undercoverage Compared to March 1973 Estimates at Two Stages of Estimation1 (in millions) Persons Aged 16 Initial Standard Extended2 and Over (1) (2) (3) In the Labor Force. . . 84.7 87.3 90.3 Employed 80.3 82.8 85.3 Unemployed 4.5 4.5 5.0 1 Hirschberg, Yuskavage, and Scheuren, 1977. 2 Average of two methods of undercoverage adjustment demographic and administrative. Table 3. Persons Employed as Percentage Labor Force for Two Stages of March 1973 Estimation Compared to Percentage Adjusted for Census-CPS Undercount1 Persons Aged 16 and Over Initial Standard Extended2 Total 94.7 94.8 94.4 White males 95.6 95.6 95.3 White females 94.5 94.8 94.4 Black and other nonwhite males 92.1 92.0 91.2 Black and other nonwhite females 89.5 89.8 89.3 1Hirschberg, Yuskavage, and Scheuren, 1977. 2 Average of two methods of undercoverage adjustment demographic and administrative. for black and other minority race males the comparable figure is 0.8 percent. Johnston and Wetzel (1969) explored the effect of the 1960 Census undercount on the labor force estimates. The authors provided two alternative sets of "corrected" labor force estimates. In the first set, an assumption was made that the missed persons had the same labor force status as their peers (persons of the same age, sex, and race group). This is called the "comparability" assumption. In the second set omitted persons were assumed to have labor force status comparable to people of the same age, sex, and race but living in urban poverty areas. This is called the "poverty neighborhood' assumption. (These assumptions are at odds with those of Korns (1977) whose research supports the premise that the labor force status of missed persons differs from that of persons counted in the CPS of the same age, sex, and race.) Table 4 shows the effect of omitted persons on the labor force estimates of population coverage by using the Johnston and Wetzel study. This study focuses on the effect of the independent estimates of population used to adjust the CPS, for undercoverage. Though the official estimates of level of employment under the comparability and poverty neighborhood assumptions are understated by 2.8 million and 2.7 million, respectively, the employment rates under both assumptions are 96.2 percent as compared to an official estimate of 96.1 percent. Differences between the Hirschberg et al study and the Johnston and Wetzel study result from changes in the CPS over time, in the labor force, and in the undercoverage between the two time periods. However, a basic difference between the two studies results from the fact that Johnston and Wetzel in their study only looked at that portion of the CPS coverage problem which would be accounted for if 6 Click HERE for graphic. the independent estimates were corrected for census undercoverage (Hirschberg, Yuskavage, and Scheuren, 1977). Attempts to Strengthen the Frame Much research has gone into coping with these problems. Already in the Annual Housing Survey several supplemental universes are used in address ED's. These are listed below (Jones, 1976 and Montie and Schwanz, 1977): 1. The Woodall Universe derived from a private commercial list of mobile home parks created during the period January 1970- December 1974; 2. The Windshield Survey which consists of canvassing sample tracts by automobile to locate mobile home parks; 3. The Successor Check which provides coverage of certain types of mobile homes at large, houses moved to the present site, and structures converted from nonresidential use since the census; 4. Permit Lag universe, which provides coverage of new construction for which permits were issued prior to January 1, 1970, but for which construction was not completed until after April 1, 1970;' 5. Within structure check for SMSA's in address ED's. Theoretically, this is not needed in CPS since the interviewer is supposed to list/update units at the first interview and at intervals thereafter. (Of course, within structure misses can occur in CPS due to inadequate listing by the interviewers.) Another coverage improvement procedure still in the planning stage is the use of the records of mobile home dealers--Dealers' Survey--to obtain new mobile homes outside of parks. The Permit Lag universe, the Windshield Survey universe, and the Woodall universe created for AHS will be used to supplement the basic CPS sample, with implementation to begin in late spring/early summer 1978. Thus, many of the problems with the frame will be corrected in the future. Within household coverage losses are more of a dilemma since it is felt that at least some of this coverage loss results from a deliberate misrepresentation of the total household composition by the respondent. Since the studies regarding within household undercoverage were limited in scope, there is still some uncertainty about some of the reasons for this significant coverage problem and whether adequate solutions can be found to remedy it. B. Procedure for Sample Selection The Sample Selection Procedure After the establishment of the frame, the next step in the design of any survey is the selection of the sample. The selection of the sample in the CPS, which is a multi-stage cluster sample, involves the selection of the primary sampling units (PSU's) and the selection of the sample households within these units. A brief description of these procedures is described in this section. A more thorough ex- planation is presented in Technical Paper No. 40. 1. The Selection of the Printary Sampling Units (PSU's) 7 The CPS in which data for national estimates are collected from approximately 47,000 eligible. households, is redesigned after each decennial census in order to utilize the most recent census data available. These data from each county or county equivalent (independent cities, parishes, etc.) in the United States are used in the definition of the PSU's, restratification of these PSU's, and the selection of the ultimate sampling units. As of March 1973, the CPS was operating completely within the framework of the redesign based on the 1970 Census. In the 1970 redesign 461 PSU's, primarily counties or groups of counties, were selected from 376 strata' Of these 461 PSU's, 156 were designated self-representing (SR); i.e., the strata and PSU's are equivalent. The other 305 PSU's were selected from 220 strata with more than one PSU in each stratum and are referred to as nonself- representing PSU's. The CPS actually consists of two independent probability samples, designated the A and C samples. The nonself-representing (NSR) portion of the A sample consists of one PSU selected from each of 220 strata; the nonselfrepresenting (NSR) portion of the C design sample consists of one PSU selected independently of the A sample from each of 110 paired strata. The 156 self-representing PSU's are considered both A and C sample PSU's. The use of two independent samples in the CPS makes possible unbiased estimates of some of the survey errors and the use of one sample for surveys other than the CPS; e.g. the A sample is used for the Health Interview Survey (HIS) while a subset of the C sample is used for the Quarterly Household Survey (QHS). The nonself-representing PSU's were assigned probabilities of selection that maximized the retention of sample PSU's from the old design sample. The procedure was developed by Keyfitz (1951) and extended by Perkins (1970 and 1971). These probabilities were then used in a Goodman-Kish (1950) controlled selection procedure to arrive at the final selection of sample PSU'S. The controlled selection procedure, which is a probability sampling procedure' was used to control the number of sample PSUs selected from each State to approximately the number expected to, be selected from the State and to control the number of PSU's to be retained in the sample to the expected number. The controlled selection program was run independently for the A and C sample PSU's and for each region. 2. Selection of Sample Households The probability of selection of each household in the CPS national sample, essentially a self-weighting sample, is dependent on the predetermined total sample size. The rate is reduced at intervals in order to keep a constant sample size over the decade. At the present time (1977) the selection rate is approximately 1 in 1500. Basically, two stages were used to select the units to be enumerated in the CPS each month from the sample PSU's. First, a sample of ED's defined for the 1970 census was selected from each PSU. These ED's were selected systematically from a sort order of ED's approximating an additional stratification by city size and geography. The probability of selection of an ED was proportionate to its 1970 population. The second step in the selection process involved the selection of ultimate sampling units (USU's)--clusters of approximately four, usually contiguous, housing units within the designated sample ED's. The procedure for the selection of these USU's varied somewhat according to whether the sample ED was designated as an area ED or address ED. In sample area ED's each housing unit was located on a map and the ED was then divided into chunks or blocks containing approximately two to five USU'S. The sample USU's were designated with probability proportionate to 1970 population, and the chunk containing the sample USU was defined as the sample segment. When that segment is scheduled to enter the sample, the interviewer will visit and list all the units in the chunk. Within segment sampling instructions will then be applied to the segment to designate the sample household units. In TAR ED's already existing Census Tape Address Registers were corrected to incorporate changes the census enumerator had noted during the census interview procedures; in other address ED's, the tape address register was generated from the handwritten address registers. In an operation called "segmenting", 8 a computer' program was used to form the USU's from the census housing units listed on the revised tape address register, designate the sample USU, and prepare the Est of units for interview. 3. Rotation of the Sample USU's selected for sample do not remain in sample for the entire decade since it is felt that this would put too much of a burden on these household respondents. Thus several CPS samples must be generated for use during the decade. Each CPS sample consists of eight approximately equal systematically selected subsamples known as rotation groups. These rotation groups are introduced into the sample once a month for 8 months using a 4-8-4 rotation scheme; i.e., each sample USU is in sample 4 months, out 8 months, and then in 4 more months. Under this scheme each month there is a 75 percent month-to- month overlap and a 50 percent year-to-year overlap. An example of the CPS rotation schedule is given in Figure 1. 4. Replacement of PSU's The CPS sample PSU's are used for surveys other than the CPS, but households are in only one sample. Thus, in small PSU'S, the USU's can easily be exhausted before the decade is over. To handle this situation a system of replacement of PSU's was developed. If a stratum contained at least one PSU without enough USU's for survey needs during the decade, then the rotation clusters or groups of PSUs were formed such that each cluster had enough USU's to last the decade. The full sample for each replacement PSU is introduced in one month across all samples and rotation groups Potential Sources of Error in CPS Sampling Procedure In the development and implementation of the CPS sampling procedure, the faithfulness to the execution of the design was a major concern. However, as with any complex survey design it was, for all practical purposes, impossible for the process to be error-free. There can be nonsampling errors associated with the sampling procedure as with any other part of the survey. These nonsampling errors in the sampling process and its implementation are discussed below. Generally, the errors involve a very small percentage of the total sample households and were accepted because time and cost made it im- practical to do otherwise. 1. In area ED's, before the ED's were divided into chunks for sampling (discussed in Section II.B.), the location of each housing unit had to be determined. Because of time and cost considerations, it would have been impractical to have the field staff visit every ED with any problems in the location of housing units; an alternative was to set a limit on location problems. If the location of at least one-fourth of the units or 50 housing units (whichever was less) could not be determined, the field staff visited the ED and determined the location of all the current housing units. Thus, in a few ED's with housing units whose location could not be determined, some of the chunks or blocks might not have received their correct measure of size; these chunks would then have been selected with a slightly larger or smaller probability of selection than they should have been. However, this particular "problem" is reflected in the estimate of variance. 2. In the original segmenting program an error was detected after about half of the address ED's in Phase I areas had been segmented. (Phase I areas consisted of those areas which were self-representing in both the current and previous design and represented approximately 57 percent of the total population.) The error involved combining two addresses in a block with identical house numbers, but i different street names such that the units at the second address would not have a chance of being interviewed. An estimate of the total lost housing units in all affected ED's was 1.656. Because of the cost involved in resegmenting the affected ED's and the small percentage of units affected-an estimated 0.1 percent of the housing units in the 3,500 affected ED's it was decided to accept the segmentation in the ED's (Shapiro, 1972). 3. An in-house Bureau memorandum (Fasteau. 1973) summarized the evaluation of the overall quality of the final machine readable address file which was the output from the CPS segmenting process. The evaluation involved the selection of a sample of TAR, prelist, and 9 conventional ED's for analysis. Some of the findings were as follows: a. The percent of defective address units in the final file processed through the system and used as input for sample designation was estimated as 0.31 percent. b. Errors left in the records which could contribute to sampling bias or cause serious problems for the enumerators were estimated at only 0.09 percent or less than one-third of all errors. c. The total number of units from the census first count tapes was about 0.5 percent higher than the estimated total number of units used for sample selection. The overall quality of the segmenting process was considered very good. 4. The introduction of replacement PSU's into the sample is completed in one month; i.e. all households in the PSU are in sample for the first time the first,month a replacement PSU is in sample, with households in sample for the first or second time in the second month. (See rotation chart, Figure 1, Section II.B.) Thus the normal rotation pattern of 4-8-4 is not followed for the PSU's; in fact, for the first 15 months the replacement PSU is in sample, the time in sample of the rotation groups does not follow the regular pattern. Considering the effect of rotation group bias (the expected value of the response is not the same for each rotation group and appears related to the number of times the units in particular rotation groups have been in sample), this procedure introduces a bias into the sample (Boisen, 1973). However, the bias is expected to be minimal since there are only 18 NSR PSU's originally designated for sample scheduled for replacement in 1973-1983 and no more than four at any one time. C. Quality Control of Sampling Procedure Selection of Sample PSU's There was no formal quality control procedure attached to the selection of the sample PSU's; however, various checks were made to ensure that the procedures were performed correctly. Though there were a number of "rules" by which the PSU's were redefined and restratified, much of this procedure was subjective, and therefore it was difficult to designate any particular outcome as "correct". However, the stratification was reviewed, discussed, and revamped sufficiently to catch any gross "incorrect" procedures. In order to ensure that no PSU was included in more than one stratum or left out entirely, the stratum populations were totaled and compared with the total published counts for States, etc. The Keyfitz probabilities were reviewed by PSU by the clerical staff and differences reconciled. Several patterns in each run (region and sample A or C) in the controlled selection program were reviewed to see if the program was actually selecting patterns according to specified controls, and the probabilities adding to 1.000. Within PSU Sampling Process Before the within PSU sampling process was begun, a program was written which checked the census published counts for each county with those on the census first count tapes used in the sampling process. Any differences were reconciled. As previously mentioned, the sample ED's were selected by computer program and certain,hand checks were devised and compared to the computer output. The comparison usually involved, within each "work unit", such checks as the random starts and tallies of hits by CPS sample and rotation group. The total counts for the "work unit' were compared to those calculated. Since the SR PSU's had the same within PSU sampling rates, it was not considered necessary to check each PSU. Thus for SR PSU's there were hand checks at ED level by SMSA/non-SMSA and region for each work unit (several States sampled within one computer run). In addition, two to four PSU's were hand sampled and compared to computer output, record by record. (A record consists of the sample ED for each survey hit.) For the NSR PSU'S, which had different within PSU sampling rates, these checks were made for each PSU. After the ED's were screened clerically to determine whether they were area, address, permit or nonpermit, etc. ' these data were punched so that the ED's could be computer edited. Any ED that failed the edit was verified or investigated. Some of the edits included (1) the identification of impossible codes in the permit, area/address or special place code fields; (2) identification of an address type ED which was nonpermit; (3) an identification of a TAR ED in a nonpermit area; and (4) identification 11 of differences of housing and population counts from the address register and first count tape (Waksberg, 1971). The street address, block number, and house number for non-TAR address ED's were coded and punched in the Jeffersonville office twice independently and the records were computer matched and clerically reconciled at. the Bureau. This procedure was referred to as address keying. In TAR ED's additions, deletions, and changes needed to update the tape address register were keyed twice dependently and reconciled. In addition to this check, a stratified sample of TAR ED's and prelist and conventional ED's were selected for analysis. The analysis involved an evaluation of the coding process, clerical review and correction, and related computer processing utilized to correct and update the ED's. From the analysis of this sample of ED's, it was concluded that the overall quality of the addresses was good (Fasteau, 1973). The computer program used in the segmentation process in address ED's generated a summary of the segmenting results. It showed the number of units which came into the segmenting process and the number of USU's formed by size of USU. The number of USU's (measure of size assigned to the ED before the selection of the sample ED's) was multi- plied by four and compared to the number of units on the computer output of the segmentation process. If the counts differed by 10 percent or more, the ED was reviewed and the results reconciled. The sample area ED's were not prepared for sample selection by computer as were the address ED's. Before the ED was prepared for sampling by the procedure described in II.B above, the number of measures obtained from the census address register was compared to the number of measures on the area ED sample list generated from the then prepared ED summary records. Differences greater than 10 percent were reconciled. 12 Observational Design and Implementation This chapter reviews the entire data collection procedure and involves a discussion of the work of many divisions within the Bureau of the Census. The potential sources of nonsampling error are many, but little is known about the existence or size of any errors. Discussed in this chapter are the following elements: basic data collection procedure, the questionnaire design, the data collection staff, the training of CPS interviewers, and the quality control program for the data collection procedure. Each of these areas is reviewed in turn, potential sources of error are pointed out, and any pertinent material on the measurement of such errors is given. A. Basic Data Collection Procedure Listing and Sampling 1. Listing in Address, Permit, and Cen-Sup Segments or Groups of USU's Address segments which consist of regular addresses selected from the 1970 Decennial Census listings, permit segments, and Cen-Sup segments are listed basically by, the same procedure. For such a segment the interviewer is provided with a segment folder which contains the address listing sheets. The folder is used to record the segment identification and the survey procedures to be applied to the segment in a given month, record the transmittal of questionnaires and control cards for completed interviews and noninterviews, etc. Take-all addresses are those at which every unit is designated for interview and correspondingly non-take-all addresses are those at which only a sample are designated for interview. At non-take all addresses, after listing all the units at the address, the interviewer must verify the listing with the building superintendent, manager, etc. At take-all one-unit addresses the interviewer must verify with a household member that there are no more units at the address; at take-all multi-unit addresses' the interviewer must verify the listing with one of the respondents. If the number of units listed for an address is different from that reported, in the census (indicated on the address listing sheet), the interviewer is instructed to determine, if possible, the reason for the discrepancy in the number of units. Where the difference is excessive, specific instructions are provided to the interviewer who generally checks with the office before interviewing. Otherwise, the interviewer will interview each unit at take-all addresses and those units which fall on lines designated for the sample units during the computer sampling at non-take-all addresses. 2. Listing in Area Segments For each area segment the interviewer receives a segment folder which contains the area segment listing sheet (providing segment, PSU, and geographic identification), a segment map and, possibly, one or more grid maps. In listing area segments, the interviewer uses a grid and segment map to locate the segment and determine its boundaries. Within each segment the interviewer must list every structure in the segment, identifying (a) each housing unit, (b) each special place, and (c) each nonresidential structure. In general, the interviewer is required to list by observation. If a structure appears to contain more than one housing unit and the interviewer is unable to determine the number of units in the structure and their location by observation, he/she is to inquire. In apartment-houses with numbered or lettered apartments the interviewer is allowed to list by observation or inquiry. If the listing is by observation, the interviewer must verify the listing with a knowledgeable person such as the manager or, a longtime tenant. Regular housing units, including those constructed after April 1970 in area sample segments within jurisdictions which issued building permits as of January 1970, are represented in permit segments. If the area is one in which there is ..considerable" new construction activity, the interviewer must inquire at each listed unit at time of listing the year the structure was built; these regular units built after April 1970 will not be represented in the area sample. In areas of low new construction activity, inquiry is not made, and new construction units are identified later in completing the control card during the interview 13 and deleted from the sample as Type C noninterviews (see page 16). This procedure is referred to as the Year Built Procedure. The listing forms are sent to the Regional Office. where the sampling takes places. 3. Listing in Special Place Segments Take-all special places are listed at the time of interview on address listing sheets. For each nontake-all special segment, the interviewer, receives the special place listing sheet. Staff units in special place segments are houses, apartments, etc. occupied or intended for occupancy by resident employees and their families; unclassified units are houses, apartments, etc. occupied or intended for occupancy by persons in certain types of special places, e.g. guests in motels/hotels. The interviewer lists Only staff or unclassified units, using his/her instruction manual as a guide. If there is a usable register and more than 100 staff or unclassified units in the special place, the interviewer records the count on the listing sheet. For other special places, the interviewer uses the complete listing method; i.e., he/she lists each staff or unclassified unit on a separate line of the listing sheet. For take- au addresses, the interviewer interviews all units on the address listing sheet; for non-take-all special places the interviewer interviews all units that the office has transcribed to the listing sheet. 4. Updating Updating is the terminology used to describe the checking of the listing and, when required, the adding of new or missed units to the present listing sheet and the recording of changes in units which are already listed. 'ne rules for updating are as follows: Address, Cen- Sup, and take-all special places in special place segments are updated during the first and fifth month of interview if they have not been updated in the previous eight months. Area segments and non-take-all special places are updated the months prior to the first and fifth months of interview if they have not been updated in the previous eight months. Permit segments are not updated. Conducting the Interview 1. General procedure The week containing the 19th day of the month is designated as interview week and an views are supposed to be conducted on Monday through Saturday of that week. For households that are difficult to enumerate, interviews are conducted on the Monday or Tuesday of the following week. In all cases, the reference period .is the week containing the 12th day of the month, called "survey week". Thus, all questions on work activity refer specifically to work activity during the survey Week. Before the interviewer visits the households, an introductory letter describing the CPS and announcing the forthcoming visit is sent to the households scheduled for interview for the first and fifth time if they, have an address to which mail can be delivered. At the first and fifth time a household comes into sample, the interviewer must inquire whether the respondent received the respondent letter and, if not, the interviewer will furnish the respondent a copy. According to the provisions of the 1974 Privacy Act, the respondent must be told that the survey is voluntary; this is clearly stated in the letters. If requested, the interviewer must explain the provisions of the Privacy Act and give a limited explanation of the sampling procedure and uses of the data. Though almost any adult household member 14 years of age or older is eligible to act as respondent, the interviewer is encouraged to interview the most knowledgeable household member, usually the household. bead or spouse. As a last resort the interviewer is allowed to interview a nonhousehold member such as a neighbor, friend, or mother-in-law provided that (1) it is not the first or last time the household is in sample; (2) the individual is at least 14 years old; and (3) the individual is knowledgeable about the family, either having lived in the household during survey or interview week, or having spent a lot of time there. The interviewer is provided a field control card for each unit scheduled for interview. At the initial interview, the interviewer records on the control card the names of all persons at each household (including visitors, if they have at least spent the night before the interview there and are present at time of interview) and determines the usual residence and relationship to household head of each person listed., In addition he/she enters information on date of, birth and age, race, sex. etc. for each person who is determined to be a 4 household member by CPS definitions. At each subsequent visit the listing is updated. The questionnaire is completed for all household members 14 years old and older. Further, the interviewer is instructed to ask the questions exactly as worded on the questionnaire and in the prescribed order. (A facsimile of the questionnaire is shown in Figure 2.) The interviewer is instructed to check the completed questionnaires carefully before sending them to the Regional Office. All completed questionnaires are mailed to the Regional Office on a daily basis where they are reviewed. 2. Mode of Interview Personal interviews are required for households in sample for the first time and for the fifth time. Interviewers also are instructed to conduct personal interviews at households in sample for the second time; however, if the interviewer does not contact the household on the first visit, he/she is permitted to conduct the interview by telephone. At households which have telephones and where the householder has consented to be interviewed by telephone, telephone interviews are permitted all other times. Some telephone interviews are conducted from the Regional Office in sample areas containing the Regional Office and in other areas if the need arises. These interviews are conducted by the clerical staff or by interviewers for any households except those in sample for the first, second or fifth time. This interviewing is done only in spe- cial circumstances such as when an interviewer has an unusually large or difficult workload, when an interviewer is sick and cannot be replaced, or other times when it seems expedient to do so. Only about 300-400 interviews of this type are conducted each month. There is a reluctance to encourage interviewing from the Regional Offices because it tends to hurt the interviewer. Since in general those households that cooperate are more subject to be telephoned, it tends to decrease his/ her production rate. 3. Noninterviews The interviewer may encounter three types of noninterview situations: Type A-those households eligible for the survey for which the interviewer was unable to complete the interview; Type B-vacant units, vacant sites, or units occupied by persons ineligible for the survey; and Type C-units demolished, converted to permanent storage or business use, moved from site, or found to be in sample by mistake. Only the Type A noninterviews affect the reliability of the data. There are four types of Type A noninterview households-the "no one home", the "temporarily absent", "the refusal" and all "other" Type A noninterviews. The "no one home" Type A's are those whose household members cannot be found at home by the interviewer after repeated calls but who are not away for any extended period of time. The households which are "temporarily absent" are those whose household members are away on vacation, business trip, etc. and will not be available for interview during the survey period. "Refusal" households are those which are contacted but whose members refuse to respond. "Other" Type A households include those which could not be reached because of impassable roads, those with death in the family, and any other Type A's which cannot be classified into the other three categories. For Type A noninterviews, race of the household head and farm/nonfarm status are required for noninterview adjustment (dis- cussed in Section V.B.). The interviewer is instructed to acquire information needed for determination of farm/nonfarm status from neighbors if it is not previously recorded on the control card. The interviewer is instructed to "determine" race of the household head by observation or his/her knowledge of the neighborhood. Type B noninterview households are visited each month to determine if any have become eligible for interview. Type C noninterview units are not visited again. Potential Sources of Error in the CPS Data Collection Procedure The potential sources of error in the CPS data collection procedure are discussed below.,Some of these errors are systematically investigated as part of the CPS reinterview program. Some of the problems interviewers have in listing were discussed in Section II.A., entitled "The Frame"; others are discussed later in this chapter in Section III.C., entitled "Data Collection Staff". 1. Listing by Observation in Area Segments Interviewer listing in area segments is accomplished by observation. Inquiry is made only if the interviewer is uncertain about the number of 15 living quarters a structure contains. An alternative procedure to this method of listing in area segments is to knock on every door for address information. Most of the listing errors occur in area segments (Schreiner, 1977); the use of the "knock on every door" procedure could result in more accurate listing in these areas. But the cost could be prohibitive and could result in undue respondent burden. In the fall of 1975 a rural listing test was conducted in nine counties (two in Louisiana, three in Mississippi, and four in Arkansas) to investigate the feasibility of a mail census in rural areas. Three procedures were used: Procedure I in which the lister tried to obtain address information by observation or from neighbors, inquiring at a housing unit only when necessary. When he/she did stop at a unit, the lister tried to get address information for other units nearby. Procedure 2 in which the lister knocked at every door. If no one was home, address information was obtained from a neighbor or by observation. A single callback was allowed as a last resort. Procedure 3 in which the lister knocked at every door. If no one was home, return visits were made in order to obtain address information from the householder. Neighbors and observation were used as a last resort. Listing by Procedure I is closest to that used in area segments in the CPS at present. In Arkansas no coverage difference was detected between any of the listing procedures. However, in Louisiana and Mississippi both Procedures 2 and 3 achieved statistically significant coverage improvement over Procedure 1, but only Procedure 2. appeared to obtain enough additional coverage to offset the increased cost per net listing (Dinwiddie, 1977). These results could have implications for the CPS. 2. Problems with the Year Built Procedure It is difficult at times for a respondent to determine the year a structure was built, particularly when he/she was not the first owner of the housing unit or when the respondent is renting rather than buying. Thus the Year Built Procedure which is used to determine whether a unit was built after April 1970 in permit issuing area segments does not always perform its intended function. Units built after April 1970 that the respondent mistakenly believes were built before that time are represented in both the permit sample and the area sample; those units built before April 1970 that the respondent states were built after April 1970 are not represented at all. 3. Determination of Race of Household Head for Type A Noninterview Units The determination of race of household head of Type A noninterview households will not always be accurate. Those households which are in sample for the first time and are eligible for interview but for which interviews cannot be obtained are more likely to be subject to this error than households in sample for the other months. This error in the determination of race of the Type A noninterviewed households will a ct the noninterview adjustment factors discussed in Section V.A. 4. Effect of Mode of Interviewing on the Employment Estimate Are there differences in the data collected by means of personal interviews from data collected by telephone interviews? If there are differences, which mode of interviewing yields data that are more nearly correct? Before the use of the telephone in CPS was instituted, a test in a limited number of PSU's was conducted to determine its effect on the data. This test, conducted in the early 1950's, showed no appreciable difference in the labor force data obtained by the two methods of interviewing, personal visit and telephone (Hanson, 1976). However, the test conducted at the time was not a completely controlled experiment, the sample size was small, and the results for today's purposes are outdated. Not only has telephone interviewing increased, but the attitudes of respondents toward surveys have probably changed over the years. Because of the wide use of telephone interviewing in the CPS, there is growing concern about its possible effects on the data, and because of this, studies are now being planned to learn more about possible effects. Tables 5 and 6 present data on the amount of telephone interviewing in the CPS. Table 5 shows the average percent of interviewing by telephone by month in sample for 1976 and Table 6 presents the percent of telephone interviewing by month in sample for employed persons in December 1973. 16 Table 5. Percent Telephone Interviews by Month in Sample-1976 Average1 Percent Month in Sample of all Interviews 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44.5 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10.3 3, 4, 6, 7, 8 . . . . . . . . . . . . . . . . . . . . . . . .76.0 1 Based on Bureau of the Census monthly CPS enumerator computer runs. Table 6. Percent Telephone interviews by Month in Sample For Employed Persons-December 1973 1 .2 Percent Month in Sample Telephone Interviews 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55.5 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81.5 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82.5 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14.2 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .78.9 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83.2 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82.8 Total. . . . . . . . . . . . . . . . . . . . . . . . . .60.2 1 Minutes of the BLS--Census Labor Force Committee Meeting of March 27, 1974. 2 The percent of interviews conducted by telephone may be somewhat higher in this month because of the energy crisis and because interviewers have traditionally conducted more interviews by telephone in December because of the Christmas holidays. It can be seen from the tables that telephone interviewing is extensive and occurs frequently even in months when. personal interviews are supposed to be conducted. Respondents in sample for the fifth time, coming back from a vacation from the survey of 8 months, are supposed to be interviewed personally and yet about 10 percent are interviewed by telephone. In months when telephone interviewing is permitted, over three-quarters of all interviews are conducted by telephone. The telephone interview rates are even higher for employed persons. At the present time there is no evidence that personal interviewing and telephone interviewing yield different results on employment questions. A recent study to assess differences it. response related to personal visit contrasted with telephone interviewing in the National Crime Survey showed differences in the number Of reported victimizations depending on the mode of interview (Woltman and Bushery, 1977). We have no evidence that the same thing would be true of employment reports. However, it is recognized that the use of the telephone may cause a different respondent to be interviewed. Data show that there is an increase in the number of "other relatives" who are respondents in later months in sample. To the extent that "other relatives" may not be as knowledgeable as the head and/or spouse of the household about the labor force status of all household members, the telephone data may not be as accurate. A well-designed study could provide answers to the questions raised. 5. The Effect of Proxy Respondents on the Data In the CPS the interviewer is instructed to interview the most knowledgeable household member, generally considered to be the household head or spouse. Technically, though, almost any adult household member 14 years of age or older is eligible to act as the respondent. Thus in the CPS, proxy respondents are frequently used. In the CPS only about 20 percent of the males and 55 percent of the females are interviewed for themselves. Groups which are largely responded for by proxies are working men and young men and women in school. Between February 1965 and June 1966 a Methods Test was conducted outside of the regular CPS with the purpose of testing new methods for the CPS. One thing studied in the test was the selection of best respondent for individual household members. Two different studies were made. In the first, three procedures were compared. The three procedures were as follows (Deighton and Zinter, 1965): Procedure 1: This procedure was similar to the present CPS procedure in that any responsible household member was accepted as the respondent for the entire household. However, unless it would have appeared odd to the respondent, the interviewer was requested to talk to only one respondent. Procedure 2: Each adult household member was to be interviewed for himself/herself, unless the interviewer judged that some person did not have the ability to give accurate responses. Procedure 3: An advance form containing important labor force questions was sent to each household in the test with a request that each adult household member fill the form personally. The interviewer was then to transcribe this information to the questionnaire and ask the household respondent the remaining questions about the household members 17 Table 7. Total Employed as Measured by Three Procedures in Methods Test1 Percentage of persons employed Procedure Total Males Female 1 (Household respondent)....... 55.5 74.2 39.7 2 (Self-respondent)............ 57.2 75.8 41.5 3 (Advance form)............... 57.3 75.8 40.9 1 Deighton, 1967. A comparison of the results was provided in a memorandum by Deighton (1967). Table 7 shows the results for the employed category. A rough approximation of sampling errors indicates that the difference in the percent of total persons employed as measured by the household respondent and the self-respondent procedures is significant. A second experiment reported by Williams (1969) also took place in which 4,500 households in three PSU's were to be interviewed in two consecutive months. In the first month, each person was to b e interviewed for himself/herself. In the second month, there were to be two respondents, each of whom would report for himself/herself and also for the rest of the household. The second month interviews also contained a retrospective interview. The experiment was actually conducted with about 70 percent of the originally selected households. About 12 percent of the households were one-person households and so were not eligible; another 12 percent were noninterview; and for 6 percent the inter- viewer had probably "curbstoned", i.e., contrived the information. By comparing the record for a person interviewed for himself/herself the first month with the retrospective interview from the second.month with a proxy respondent, an estimate of the joint effects of recall bias and proxy respondent was made. By comparing the second month interview of a person responding for himself/herself with the interview for the same time period provided by another household respondent, the effect of the proxy respondent was estimated. For the employment item, the results were as follows: Table 8. Percent Difference in Total Employed as Measured by Self and Proxy Respondent Procedure1 Procedure Effect All Males Females Joint effect, recall, proxy...... -4.1 -1.8 -8.4 Proxy alone...................... -1.2 - .8 -2.0 1 Williams, 1969 and Jones and Aquilino, 1970. The negative sign indicates that there was an understatement of the number of employed. Sampling errors given in Williams (1969) indicate that the -4.1 percent difference in total employed is sig- nificant. The smaller percentages are clearly not significant. The results above show the effect of a total proxy respondent procedure and thus are upper bounds on the effect of proxy respondents in the CPS procedure. B. Questionnaire Design and Instructions The Questionnaire and Interviewer Instructions The basic CPS questionnaire has been in use since 1961. However, there have been changes in the employment criteria over time, some that affected the questionnaire and vice versa. For example, before January 1967, persons who had jobs but did not work during the survey week and were looking for other jobs were classified as unemployed and the questionnaire reflected it; as of January 1967, persons with jobs were classified as employed even if absent from their work during the survey week and looking for jobs. The questionnaire in use as of January 1967 has a skip pattern within the labor force questions so that these individuals can be recorded as employed. The questionnaire effective as of February 1976 has further changes in the description of job or business and the classification of government employees into Federal, State, or local government employees. It should be noted that question 20 (see Figure 2 for portion of questionnaire containing labor force questions) has instructions for the interviewer to probe about unpaid work if the household contains a farm or business owner. Without the additional probe, the respondent might consider his/her contribution to family, business as "work around the house" and he/she would not be included in the employment count. Unpaid workers who worked on a farm or in a family business at least 15 hours a week are considered employed. Before January 1967, question 20 did not contain the added note to probe for this type of employment. Beginning in January 1970, the labor force supplement for non- labor force persons (item 24a-24e---see Figure 2) was asked in the fourth and eighth month panels; from January 1967 to December 1969 this set of questions had been asked in the first and fifth month panels. This change was made because 18 there was evidence that the use of this question in the first and fifth month panels added unemployed persons to these panels which already had higher numbers of unemployed than the average over all panels. The fourth and eighth month panels usually had the lowest numbers of unemployed. The interviewer's manual contains explanations of the purpose of the labor force questions and instructions for their execution. The labor force questions begin with question 19. In question 19, "What was . . . doing most of last week, working or something else?," the responses are arranged in a priority order, i.e. the interviewer is instructed to mark the first category which fits one of the respondent's responses. For example, if the respondent replies, "going to school and looking for work", the interviewer will. mark looking for Work. The following categories under "Working" are given in the Interviewer's Reference Manual (1976) but in somewhat more detail. 1. Paid work or wages or salary which includes cash wages, salary, commissions, tips, or meals, living quarters, etc. received in place of cash wages. 2. Work for profit or fees in the respondents own business or profession, or farm. 3. Work without pay in family business or family farm which actually contributed to the operation of a farm Or unincorporated business run by a member of, the same household related by blood, marriage, or adoption. 4. Exchange work or share work on farms. 5. Jury duty if the person is paid for jury duty. 6. National guard duty (not in the active Armed Forces by Presidential Order). There are two different methods used to tell the interviewer how to proceed from item to item directional arrows or italicized instructions in parentheses. The interviewer is instructed to proceed in numerical order (in the absence of either directional arrows or italicized instructions) and is cautioned not to skip questions unless told to do so. For cases in which the interviewer receives the answer to questions other than those he/she asks, the interviewer must always verify the responses to the additional questions when he/she encounters them. Potential Sources of Error in the Questionnaire Design Do the questions make sense to respondents? Do they know what is being asked of them? Do the interviewers feel comfortable in asking the questions as worded or do they frequently reword questions? Do other questions on the interview schedule impact on the classification of labor force status? Some limited data are available to answer these questions. In September and October 1969, Census staff members undertook some systematic observation of CPS interviewing as part of a questionnaire-research program. Members of the Washington staff were the observers. In January 1970 additional observation was undertaken, but with regular CPS observers. Specifically, the staff/regular observers observed the interviewers' wording of the questions, question explanations, use of probing, etc. as one indication of questions that could be sources of problems. An hypothesis was that if a question is rarely under,stood or replied to as initially asked, the interviewers would be likely to reword the questions to get replies. Such rewording could possibly alter the meaning of the question. Table 9 shows the results for three questions concerned with employment. The interviewer did not accept the first answer given for these three questions only 5 to 8 percent of the time; however, the percentage of times that these three questions were reworded was greater. Question 19 was reworded 8 percent of the time in the first period and 14 percent of the time in the second. Whether this rewording of the questions led .to less accurate answers was not within the scope of the study. In a Mock Interview Study conducted by the Response Research Staff, (see Section III.D.) further information was gathered on problems inter- Table 9. Frequency of Rewording of Labor Force Questions1 Percent of times Percent of times an interviewer question not asked did not accept as written first answer given CPS Question 1st 2nd 1st 2nd Period Period Period Period 19. What was... do- ing most of last week?......... 8 14 8 6 20. Did ... do any work at all last week, not counting work around the house?..... 13 14 7 6 21. Did ... have a job or business from which he was temporarily absent or on lay- off last week? ... 18 9 3 5 ______________________________________________________________________ 1 Rustemeyer and Rothwell, 1969 and 1971. 20 viewers have with specific questions. This information will be presented in a comprehensive study report not yet released by Rustemeyer. Additional questions on the questionnaire can affect the estimates of employment. The labor force supplement questions (24a- 24e) are asked of persons not in the labor force in part of the sample each month. In 1968-69, the questions were asked of persons in sample for the first and fifth times. In 1970 these questions were shifted to persons in sample for the fourth and eighth times. The questions may make a difference in the number of persons classified as employed, at least for women. The data are shown in Table 10. The rotation group indices shown are the average number of persons classified as employed in the particular month in sample divided by the average over all months in sample and then multiplied by 100. The indices arc higher for months one and five in the earlier time period and for months four and eight for the later time periods. The sampling errors of the differences of the indices is about .4 for mates and .7 for females. Though these differences are not statistically significant for the data shown, a third time period is now available which has data for 1970-1976. The differences for females are "significant". Thus, the data show that the additional questions on the interview schedule may result in more persons being classified as employed. These people should probably be classified as employed. Therefore, the questions as presently worded, without the additional questions, seem to result in a slight downward bias in the number of employed. Whether these additional questions should be included in the survey Table 10. Rotation Group Indices for Employment Items for Two Time Periods, 1968-69 (TI) and 1970-72 (T2)1 Month in Sample Characteristic All Persons 16 and over 1 4 5 8 Civilian labor force ... T1 102.3 99.5 100.8 99.0 Civilian labor force ... T2 101.6 100.3 100.0 100.0 Employed ............... T1 101.6 99.8 100.4 99.3 Employed ............... T2 101.1 100.3 99.9 100.1 Males Employed ............... T1 100.9 99.8 100.2 99.7 Employed ............... T2 100.7 100.2 99.9 100.2 Females Employed .............. T1 102.8 99.7 100.8 98.6 Employed .............. T2 101.9 100.3 100.0 100.1 1 Based on annual averages from gross change tables produced monthly, quarterly, and annually by the Bureau of the Census. every month can only be answered after further research and experimentation. Possibly the additional respondent burden of answering these questions each month could have a detrimental effect on the data. The CPS frequently has supplements to the basic labor force interview, It is possible that these supplements may affect the quality of the data. However, the labor force questions are asked first, so they are probably not changed because of more questions being asked. The March supplement, however, is very long and it has been noticed that the noninterview rate is higher in March. C. Data Collection Staff Organization and Description of the Data Collection Staff The data collection staff works from 12 Regional Offices under the overall supervision of the Chief of the Field Division. Regional- Offices were originally defined to equalize the workload over all programs. A Regional Director supervises each Regional Office. The CPS is the responsibility of the Demographic Program Coordinator who has a CPS Program Supervisor on his/her staff. There are three to five office clerks in each regional office who work essentially full time on the CPS. Each Regional Office has between 70 and 150 interviewers available for the CPS, or about one interviewer for every -55 households assigned for interview. Each office usually has 12 or more Supervisory Field Representatives who assist the CPS Program Supervisors in on-the-job training, observation and reinterview of the interviewer staff. The Supervisory Field Representatives also carry out CPS assignments as the need arises. Each interviewer is a part- time employee who works out of his/her home. About 30 percent of the interviewers leave the staff each year, though this is not evenly distributed throughout the country. As a result, the Regional Offices are always recruiting and training new interviewers. To be accepted as a CPS interviewer a person must pass a "Field Employee Selection Aid" test which covers material on reading, arithmetic, and map reading. Each interviewer usually lives in the PSU in which there is CPS work, and must have an automobile available for interviewing assignments. (Interviewers in large cities where public transportation 21 is readily available are not required to have automobiles.) In most cases, new interviewers are paid at GS-3 levels and after one year of experience are eligible for payment at the GS-4 level. As of October 1977 those salary levels are $3.81 and $4.28 an hour respectively. They are paid mileage for the use of their own cars while on the job and are paid for classroom and home study training as well as time spent in interviewing and traveling. As part of a research project to analyze relationships between interviewers' attitudes, expectations, and characteristics and their ability to collect income information in the CPS March supplement, a questionnaire was sent to each CPS interviewer who worked in March 1975. There were 1,015 interviewers who worked and 867 of them filled out and returned a questionnaire. A small number of those who filled out a questionnaire did not sign it, so that it was not possible to link the responses from that questionnaire with other data. However, for 802 of the interviewers, the linking of records from the March 1975 CPS and from Field Division records was made. Based on that subset of the interviewers, certain items of information are available. Table 11 summarizes this information. The table shows clearly that, in general, the CPS interviewers are middle-aged, married women who are well-educated and have family incomes above the national average. Many of them had been CPS inter- viewers for a long time. Over 28 percent of the 802 had been CPS interviewers for more than six years; about 20 percent had been CPS interviewers for one year or less. Potential Sources of Error Associated with Interviewers Interviewers have the opportunity to affect the data in numerous ways. They may fail to collect the data from certain households and thus induce a nonresponse bias. They may change the meaning of questions. They may record the answers incorrectly. They may make up information. They may misunderstand the concepts involved. Interviewer errors can increase the variance of survey statistics and can also increase the bias. To the extent-that interviewers do not understand the concepts, ask the questions, and record the answers in a uniform way, there is an increase in the variability of survey statistics. This can be important for small areas, but usually is negligible for large areas. To the extent that all inter- Table 11. Summary of Characteristics of CPS Interviewers, March 1975 Number of Percentage of Characteristics interviewers interviewers Total ..................... 802 100.0 Age: 25 to 34 ........ 67 8.4 35 to 44 ........ 221 27.6 45 to 54 ........ 345 43.0 55 and over ..... 166 20.7 Not reported .... 3 .4 Sex: Male ............ 26 3.2 Female .......... 772 96.3 Not reported .... 4 .5 Marital status: Married ......... 688 85.8 Widowed, divorced, or separated .... 103 12.8 Never married ... 9 1.1 Not reported .... 2 .2 Family income: Under $5,000 .... 30 3.7 $5,000 to $9,999. 109 13.6 $10,000 to $14,999 161 20.1 $15,000 to $19,999 202 25.2 $20,000 to $24,999 176 21.9 $25,000 or more .. 115 14.3 Not reported...... 9 1.1 Educational attainment: Did not complete high school....... 26 3.2 High school graduate ......... 410 51.1 College 1 to 3 years .......... 235 29.3 College graduate.. 128 16.0 Not reported ..... 3 .4 viewers, or a large set of them, behave in a way different from what was planned, they add a bias, The bias can be important at all levels of aggregation. The degree of nonresponse can also add to the bias. The quality of an interviewer's work is measured in many, different ways at the Census Bureau. For example, interviewers may not get an interview at each assigned unit. Sometimes this is because the unit is vacant, no longer exists, or because of other such circumstances. These are cases in which either the sampling unit was no longer there (Type C noninterview) or because there was no eligible person in a household (Type B noninterview). Then there are cases in which there was an eligible person in the household but the interviewer did not interview the person because of "no one home", "refusal", or some other reason. This latter type of noninterview is known as a Type A noninterview, and an interviewer's Type A rate is one measure of the quality of the interviewer's work. The Type A rate by itself is not an adequate measure of performance. A low Type A rate could be obtained by making up interviews at difficult-to- interview households. However, 22 Table 12. Number and Distribution of Total Noninterview and in A and C Design CPS Sample1 (Each month. average 1975)2 Number of CPS Sample Design units 3 Percents 1. Total sample units designated......... 57,667 100.0 2. Type C noninterviews detected in previous interview months........... 2,589 4.3 3. Units assigned for interview (1-2).... 55,078 95.5 4. Type C noninterviews detected in current interview month............. 480 0.8 5. Type B noninterview................... 7,589 13.2 6. Occupied units (households) eligible for interview (3-4-5)...... 47,009 81.5 100.0 7. Type A noninterview, total............ 1,982 4.2 8. No one home........................... 0.9 9. Temporarily absent.................... 0.9 10. Refusal.............................. 2.2 11. Other, occupied...................... 0.2 12. Completed interviews, occupied units. 45.027 95.8 13. Total persons interviewed, age 14.... 99.281 1 Hanson, 1976. 2 Figures include an additional sample of about 1800 households with Spanish head interviewed in March. 3 Housing units and group quarters listing units. classification of noninterviews is one of the items checked in the reinterview in an attempt to-control possible fabrication. A Type A rate of over 5 percent is in the "needs improvement" category. Table 12 shows the average monthly distribution of the total units, the noninterviewed, and interviewed units for the CPS sample averaged over the 12 months of 1975. Table 13 shows the distribution of Type A rates by interviewers for the period July 1974-June 1975. Table 13 shows that over 72 percent of the interviewers had Type A rates of under 5 percent and only about 5 percent had Type A rates of I 0 percent or higher. It is possible that Type A rates are not exclusively a function of an interviewer's performance, but rather a function of the type of area in which he/she works. A report by Inderfurth (1972) summarizes the results of two studies at the Census Bureau that refute that argument. In the first study, PSU's were divided into five strata by type of area (highly urban to rural), and every fifth PSU within a stratum was selected for a sample. For the sample PSU's the number of total households and the number of Type A households were listed for each segment. The segments in the sample PSU's were divided into two groups: those with zero Type A rates and the remainder. In the analysis, all of the segments with nonzero Type A rates were used and one-fifth of the segments with zero Type A rates were used. Certain demographic characteristics, such as proportion of persons of races other than white and median income, were determined for each segment. It was found that the segments with zero Type A rates were almost identi- Table 13. Distribution of CPS Interviewer Type A Noninterview Rates. Average July 1974-June 19751 Interviewers Noninterview Rating (percent) Total interviewer assignments2........ 992...... 100.0 Interviewers rated3............................. 98.6 Excellent 51.5 0-0.9..................................... 26.8 1.0-1.9..................................... 9.1 2.0-2.9..................................... 15.6 Satisfactory 21.1 3.0-3.9..................................... 11.4 4.0-4.9..................................... 9.7 Needs improvement4 16.8 5.0-5.9..................................... 7.3 6.0-6.9..................................... 5.6 7.0-7.9..................................... 3.9 Unsatisfactory 9.2 8.0- 8.9.................................... 2.7 9.0- 9.9.................................... 1.6 10.0-10.9................................... 1.5 11.0 and over.............................. 3.4 1 Hanson, 1976. 2 Average number of interviewer assignments per month. 3 Interviewer must have 6 months' CPS experience to be rated. 4 On basis of local conditions (e.g., bad weather) or if interviewer is directed to complete another interviewer's assignment with large number of unconfirmed refusal, the Regional Director may accept a type A rate in excess of five percent as Satisfactory. cal to those segments with nonzero Type A rates with respect to the demographic characteristics. The conclusion was that demographic characteristics associated with a segment were not good predictors of Type A rates. In the second study, 29 PSU's were selected which were single- assignment PSU's, but in which more than one interviewer had worked over a certain time 23 period. A hypothesis was stated that the mean of the differences in Type A rates for the sets of two interviewers would be no greater than the mean of differences in randomly 'selected assignments from the universe of CPS assignments. The assignments for each of the two interviewers in the 29 PSU's were identical. Therefore ' if the Type A rates were more alike than could be expected by chance, then the Type A rates were attributable to the characteristics of the PSU. The mean difference in Type A rates for the 29 PSU's was 2.8 while the mean difference in randomly selected PSU's was 2.4. The conclusion was that the Type A rates were functions of the interviewer, not the type of area. Another type of rate that is viewed as a measure of quality is the error rate. Error rates are computed in two different ways, depending upon whether an interviewer has attained QE (qualified edit) status. For a non-QE interviewer, the error rate is defined as the ratio of two numbers. In the numerator is a total defined as the sum of the number of errors found by the computer. Added to this is the number of errors from the regional office edit, from which is subtracted the number of entries not applicable. The denominator includes the sum of the total records and the total interviewed households. If an interviewer maintains an error rate of 7.5 or less for each of 3 consecutive months, he/she attains QE status. If the error rate goes above 7.5 percent for 3 consecutive months or if it exceeds 12.5 percent for any 1 month, the interviewer becomes non-QE. For the QE interviewers, no regional office edit is performed, so the error rate has the same denominator, but the numerator is just the number of computer-detected errors adjusted for entries that are not applicable. Table 14 shows the distribution of CPS interviewer error rates, averaged over the period July 1974-June 1975. Notice that 8..) percent of the interviewers were in the unsatisfactory range and about half of the interviewers had very low error rates and were rated excellent. An analysis of error rates was made in 1964 for CPS. Graphs for monthly Type A and error rates were tabulated for interviewers trained in 1962 or 1963. These graphs were compared with those for experienced interviewers (those trained before 1962 with more than 24 months on the job). It was found that the Type A rates did not fall to the level of experienced interviewers for about 22 months,and the error rates did not come down to the level of experienced interviewers for about 20 months. The Table 14. Distribution of CPS Interviewer Questionnaire Edit Error Rates, Monthly Average July 1974 Through June 19751 Edit error rate2 and rating Interviewers for interviewer (percent) Total interviewer assignments3........ 992...... 100.0 Interviewers rated............................... 98.6 Excellent 48.3 0.0-0.4................................... 13.9 0.5-1.4..................................... 19.6 1.5-2.4..................................... 14.8 Satisfactory 32.0 2.5-3.4..................................... 10.4 3.5-4.4..................................... 7.5 4.5-5.4..................................... 6.0 5.5-6.4..................................... 4.6 6.5-7.4..................................... 3.5 Needs improvement 10.1 7.5-8.4..................................... 3.0 8.5-9.4..................................... 2.5 9.5-10.4.................................... 1.7 10.5-11.4................................... 1.6 11.5-12.4................................... 1.3 Unsatisfactory 8.2 12.5-14.4................................... 1.8 14.5-16.4................................... 1.3 16.5 and over............................... 5.1 11.0 and over............................... 3.4 1 Hanson, 1976. 2 Number of errors per 100 questionnaire pages. 3 Average number of interviewer assignments per month. mean error rates for the CPS were also tabulated by regional office and varied considerably. Another measure of interviewer performance is a production ratio. The production ratio is the ratio of the minutes allowed for an assignment to the minutes reported for the assignment. The number of minutes allowed for the assignment is determined by the type of area and other characteristics of the PSU, and the activities necessary for Completion of the assignment (number of telephone calls, personal visits. etc.). Production ratios are used for the following purposes: (1) to hold down costs. (2) to maintain a certain level of efficiency, in the program, (3) to help analyze the activities of individual interviewers, and (4) to assist supervisors in identifying, interviewers who need corrective action. An interviewer production ratio less than 0.8 based on the work of one quarter is considered below standard. Table 15 shows the distribution of monthly CPS interviewer production ratios, averaged over the period July 1974 through June 1975. Notice that 7.5 percent of the interviewers had production ratios of less than 0.8. It could also be argued that the work of those interviewers with production ratios of 1.50 or more should also be analyzed. High production ratios could signal that the assignment was done too rapidly and corners 24 Table 15. Distribution of CPS Interviewer Production Ratios, Averages July 1974 Through June 19751 Production ratio2 and rating Interviewers for interviewer (percent) Total interviewer assignments3........ 992...... 100.0 Interviewers rated............................... 88.3 Excellent 35.1 1.50 or more................................ 3.1 1.40-1.49................................... 2.8 1.30-1.39................................... 5.5 1.20-1.29................................... 9.7 1.10-1.19................................... 14.0 Satisfactory 45.7 1.00-1.09................................... 17.5 .90- .99................................... 7.5 .80- .89................................... 16.4 Needs improvement 11.8 .70- .79................................... 5.1 Unsatisfactory 5.1 less than .70.............................. 2.4 14.5-16.4................................... 2.4 1 Hanson, 1976. 2 Ratio of standard allowable time per assignment to the actual interviewer time. 3 Average number of interviewer assignments per month. were cut or that the standards set for that area should be reviewed. However, as noted in the table, high production ratios are labelled "excellent" and 35 percent of the interviewers are in that category. In 1966 an analysis of production ratios was undertaken. Monthly production ratios were studied for all interviewers who had up to 72 months service. It was found that it took 47 months to reach a pro- duction ratio of 1.09. The mean production ratio for interviewers with over 24 months service was 1.04 and it took approximately 37 months to reach that mean. The CPS reinterview program,is described in more detail in Section III.E. of this chapter. One of the results of that reinterview program is to provide indices of the quality of coverage by interviewers and of the quality of the content. The principal index of coverage is the "interviewer gross coverage error rate" which is the sum of the erroneous omissions and inclusions divided by the total number of listings before the reinterview. Table 16 shows the distribution of the monthly gross coverage I error rates averaged over the period April 1973 to December 1974. Notice that about 10 percent of the interviewers accounted for 87 percent of the coverage errors. Indeed, only 16 percent of the interviewers were found to make coverage errors. Table 17 shows the distribution of the gross content error rates averaged over the period April 1973 through December 1974. The numerator is the number of errors in content divided by the number Table 16. Distribution of Interviewers Monthly Gross Coverage Error Rates in the Current Population Survey April 1973-December 19741 Interviewer checked Gross errors Gross coverage error rate Cumulative Cumulative (percent) Number percent Number percent Total ......... 3543 --- 2177 --- 0.............. 2901 100.0 0 100.0 0.1- 0.9 .............. 63 16.0 81 100.0 1.0- 1.9 .............. 128 14.2 201 96.3 2.0- 2.9 ............... 77 10.5 175 87.1 3.0- 3.9 ............... 56 8.2 161 79.0 4.0- 4.9 ............... 46 6.6 198 71.6 5.0- 5.9 ............... 39 5.3 156 62.5 6.0- 6.9 ............... 28 4.2 122 55.3 7.0- 7.9 ............... 16 3.4 73 49.7 8.0- 8.9 ............... 15 2.9 94 46.3 9.0- 9.9 ............... 16 2.5 139 42.0 10.0-14.9 ............... 26 2.0 138 35.6. 15.0-24.9 ............... 15 1.2 105 29.3 25.0 and over............ 27 0.8 534 24.5 1 Hanson, 1976. Table 17. Distribution of Interviewers' Gross Content Error Rates in the Current Population Survey April 1973-December 19741 Interviewer checked Gross errors Gross coverage error rate Cumulative Cumulative (percent) Number percent Number percent Total ......... 3,459 --- 5,901 --- 0.............. 1,563 100.0 0 100.0 0.1- 0.9 .............. 143 54.8 147 100.0 1.0- 1.9 .............. 489 50.7 698 97.5 2.0- 2.9 ............... 386 36.6 876 85.7 3.0- 3.9 ............... 243 25.4 771 70.9 4.0- 4.9 ............... 192 18.4 795 57.8 5.0- 5.9 ............... 126 12.8 548 44.3 6.0- 6.9 ............... 114 9.2 614 35.0 7.0- 7.9 ............... 50 5.9 319 24.6 8.0- 8.9 ............... 57 4.5 395 19.2 9.0- 9.9 ............... 17 2.8 103 12.5 10.0-14.9 ............... 59 2.3 437 10.7. 15.0-24.9 ............... 16 0.6 156 3.3 25.0 and over............ 4 0.1 42 0.7 1 Hanson, 1976. of items to be completed. About half of the interviewers accounted for all of the content errors. An attempt was made to attribute content differences to the original respondent, the reinterview respondent, the original interviewer, the reinterviewer, or to material shortcomings. The identification of a source is not always possible. The results of the process for the years 1959-1966 are shown in Table 18. Though the table shows that the interviewers were charged with the errors at least 25 percent of the time, there are no data that show the number of errors by type of error that were chargeable to the interviewers. Thus, one does not know 25 if errors in reporting of employment status were more or less likely to be errors charged to the interviewers. There are practically no data available that show how any of the measures of interviewer performance mentioned to this point--Type A rate, error rate, production ratio, gross coverage or content error rate--affect the estimates of employment. Thus, we have very little idea of whether interviewers with high error rates, for example, tend to misclassify employment status. One other kind of measure of interviewer performance is obtainable by means of an interviewer variance study. In a well- designed study, the assignments of two or more interviewers are interpenetrated throughout the same area. Thus, since the interviewers are working in the same area, differences between them do not reflect the type of area, but legitimate interviewer differences. Though the Bureau of the Census has carried out interviewer variance studies for the 1950, 1960, and 1970 censuses of population, and for the 1975 National Crime Survey in eight cities, there has never been an interviewer variance study in the CPS. Such a study would provide estimates of interviewer variability for many different items and would provide information on which items and would provide information on which items were not being handled in a uniform manner by interviewers. However, a study was carried out at the Bureau of Tepping and Boland (1972) that does shed some light on the extent to which variability among CPS interviewers affects the employment estimates. During the last 6 months of 1966, the CPS and the Monthly Labor Survey (MLS) were conducted concurrently and independently. They had separate field staffs and were separately supervised. Although the samples were different, the field procedures were basically the same, and in 143 PSU's both surveys operated concurrently. It seemed a reasonable assumption that in each of these PSU's the two surveys provided independent estimates with the same expected values for a number of different characteristics. Of course, this is an approximation since, in general, the CPS interviewers were more experienced and the two questionnaires were slightly different with respect to certain items. Estimates of the total variance, the sampling variance, and the correlated component of response variance were computed. It is the latter component which reflects the degree to which interviewers do things the same way. If each interviewer interpreted and carried out the instructions in exactly the same way, this component would be zero. To the extent that interviewers express their "personal bias" this term can be quite large. If this component is large, it is an indication that interviewers are having a serious impact on the survey data. The authors showed the ration of the correlated component of response variance to the sampling variance, and concluded that ratios of 0.4 or greater represented substantial contributions by the interviewers. In the study, assumptions had to be made. Assumptions were almost always made in such a way that the ratios would be understated rather than overstated. Thus, the authors felt that the results presented were conservative estimates of interviewer effect on CPS data. For a complete description of the mathematical model the reader is referred to the report. Table 19 summarizes the results of this experiment for employment characteristics. The ratios were averaged for sets of PSU's and variances of the averages were also computed. Also whenever the estimated correlated component was negative, it was replaced by zero. The estimates are shown for all 143 PSU's and separately for three groups. Group 1 consisted of the 42 largest PSU's according to the 1960 population. Each had over one-half million people. Group III contained the remaining 55 PSU's. Table 18. Responsibility for Differences Between Responses From CPS Original Interview and Reinterview: 1959 to 19661 (In percent) Chargeable 1959 1960 1961 1962 1963 1964 1965 1966 Interviewer............. 24.0 29.9 28.9 30.6 27.1 24.8 24.4 24.7 Reinterview............. 6.8 8.8 4.9 5.0 5.2 3.9 4.6 4.5 Original respondent..... 22.8 20.4 20.4 21.3 21.7 17.9 18.5 21.8 Reinterview respondent.. 19.7 20.8 25.9 24.7 23.8 26.5 27.9 25.5 Not ascertained......... 26.7 20.0 20.8 18.5 22.2 26.9 24.6 23.5 1 U.S Bureau of the Census, the Current Population Survey Reinterviews Program, January 1961 through December 1966, Technical Paper No. 19, 1968. 26 Table 19. Estimates of the Average Ratio Over 6 Months of the Correlated Component of Response Variance to the Sampling Variance for Employment Items. and the Estimated Standard Deviations of the Estimated Ratios1 Group I GroupII.2 Group III.2 All Item r o.r r o.r r o.r r o.r At work full time...... 0.68 0.17 0.70 0.16 0.98 0.29 0.80 0.13 At work part time...... 0.35 0.13 0.69 0.17 0.72 0.17 0.60 0.09 With job, not at work.. 0.53 0.18 0.64 0.16 17 17. 7.1 6.5 Unpaid family worker... 0.20 0.07 0.09 0.05 85. 85. 33. 33. Employed agriculture... 0.42 0.28 0* 0.06 0* 0.07 0.08 0.9 Employed non- agriculture............ 0.64 0.15 0.18 0.17 1.2 0.25 0.90 0.12 1 Tepping and Boland, 1972. 2 Group 1 consisted of the 42 largest PSU's each with 1960 population over 0.5 million; Group 11 consisted of the next 46 PSU's, each with a population between 0.3 and 0.5 million persons; and Group III contained the remaining 55 PSU's. * Indicates that a negative estimate was replaced by zero. Though the item "employed" was not one of those studied explicitly, the categories shown in Table 19 are part of the total employed. Indeed, the category "employed non-agriculture" accounts for about 96 percent of the total employed. The results for the employed non-agriculture" show that the item is subject to substantial interviewer effects in all three types of PSU's. For people at work full time, there would seem to be little difficulty in classification, but the results show that there is considerable interviewer variability for that item over all PSU's and for each group separately. The estimates of between-PSU variability produced monthly for the CPS for the nonself-representing PSU's are not large. These estimates would contain interviewer variability components. As was pointed out by Banks and Shapiro (1971) the estimated between-PSU variances are not very reliable. Based on 3-year averages, the estimated between - PSU variances ranged from about 3 percent to 26 percent of the total variance. However, negative variances were common. All of Group 1, most of Group 11, and part of Group III consisted of self representing PSU'S, so would not have interviewer variability measured in between- PSU variances. The impact of interviewer variability can affect the survey data in a variety of ways. First, the estimates of sampling variability that are computed regularly do not reflect the interviewer variability for the self-representing PSU's. The published estimates would reflect the interviewer variability in the nonself-representing PSU's. But for the largest PSU's, those represented in Group 1, the estimates of variability are too low. Second, estimates of employment are released for areas below the national level, including states, selected SMSA's and 14 central cities of large metropolises. Those estimates, specifically for the central cities, are based on the work of only a few interviewers and could be biased as well as having a large variance. For example. estimates for the city of Washington, D.C. are published. These estimates are based on the work of three or four interviewers. The interviewer impact could be substantial. Finally, the Mock Interview Project, conducted in 1975, gave some indication, in a controlled setting, of the types of errors made by interviewers. This project will be described in more detail in the next section of this report. However, in this project, it was found that interviewers were correct more often in their written work than in their verbal work. Thus interviewers were more likely to record answers correctly than they were to ask questions correctly. The way in which this selected group of interviewers probed for additional information was judged acceptable about 80 percent of the time (Rustemeyer, 1977). In summary, we do have considerable information available about the interviewing staff. We know their Type A rates, error rates, and production rates. We have seen that there may be considerable inter- viewer variability in the employment classification. However, we do not have a way of combining the information to measure the full impact of the interviewers on the employment statistic. We can use the limited interviewer variability study to give some indication of the increase in variance that comes about because of interviewers. However, we do not know how different Type A rates, error rates, and production rates affect the bias of the statistic. These kinds of rates give an indication that where the Bureau of the Census is able to identify a potential troublespot, good control is kept over the operator, of the survey. Unfortunately, the correlations between the measures used and the actual data quality are inconsequential. D. Training of the CPS Interviewers The Training Procedure The CPS training of interviewers is a mixture of self-study, classroom training, and on-the-job instruction. The training is a continuous process and consists mainly of these steps: 1. Initial Training a. New interviewers recruited for the survey are given special intensive training the first 3 months they are on the job. The program includes approximately 1 day of advance home study exercises and about 2 days of classroom study. There is another « day of instruction on payroll and administrative matters. The classroom study includes the use of lectures, discussions, audio-visual aids, and mock interview exercises. It includes comprehensive instructions on the completion of survey forms with special emphasis on the completion of the CPS questionnaire and the labor force concepts. The supervisor or Supervisory Field Representative observes the interviewer during her first 2 days of interviewing. b. Before the 2nd month's interviewing assignment, the interviewer completes approximately 1« days self-study exercises. He/she then is observed for 1 day by the supervisor or Supervisory Field Representative. c. Prior to the 3rd month's assignment, a self study exercise is completed and a final review test is administered. d. The interviewer completes a self-study exercise before the 4th-6th month assignment and is observed at least once during this period. 2. Refresher Training Prior to each monthly enumeration, interviewers are given home study exercises usually consisting of questions concerning labor force concepts, survey procedures, and listing. Three times a year the interviewers participate in group training sessions. 3. Special Training As part of the CPS Quality Control Program, interviewers are observed and reinterviewed (see Section III.E.) on a regular basis. Special training is administered to those interviewers whose work performance is found by these procedures to be unsatisfactory in one or more areas. 4. Determining When an Interviewer is Trained "An interviewer is presently considered to be trained when: -- He/she achieves a satisfactory Type A Rating, which measures understanding of the survey and ability to sufficiently explain it to respondents; -- He/she achieves a satisfactory Error Rating, which measures technical knowledge in, properly handling the data documents; -- He/she achieves a satisfactory Production Standard, which measures efficiency and, consequently, a major part of the survey costs; and -Reinterview, Observation and Testing Check reports on his/her work indicate that he/she is visiting the appropriate households and conducting interviews properly." (Love, 1974). Limitations in the Training Procedure In March, 1974 an Advisory Committee on Current Program Training was formed to review the training given to CPS interviewers and make recommendations, with,special emphasis given to the initial training of the CPS interviewer. Though the committee found that much of the training was good. it felt there were areas where it needed improvement. These areas included: a. determining when an interviewer is trained; b. training procedure and materials; c. the training schedule. The committee noted that there could be a vast difference between visible and concealed interviewer error. In a paper written about the Government Social Survey of Britain (Harris, 1952) it was stated that only about 12 percent of the errors made by interviewers who were tested could be considered visible errors--errors detected at the coding stage. e.g. omissions or inadequate information, items written in the wrong place. The rest could be attributed to invisible errors such as altering the scope of the question, failure to probe for additional information, overprobing after it is clear that the respondent has nothing further to add, incorrect recording of information given by respondents, and so forth. Though the study was conducted in Britain in 1952 and the percentages cannot be applied to the CPS interviewers, the concept could have some relevance to CPS interviewers. There was contrary opinion as to whether training could be improved to detect "invisible" errors. To detect these invisible errors in initial training, it was suggested by some committee members that 28 interviewers conduct mock interviews with trained Bureau staff members and that these sessions be recorded so that defective interviewing could be recognized. Other committee members felt that observation and reinterview serve the same purpose. A study of this suggested addition to the CPS training program was performed in the summer and fall of 1975 under the leadership of the Response Research Staff; it is referred to as the Mock Interview Project (MIP). While interviewers were sampled for this study, the situations were contrived to "represent" those they encounter in the field. Therefore, inferences cannot be made from the results of this study about the extent of labor force misclassification' However, the study gives some indication of the kind of situations that may present problems to the interviewers and suggests the need for better training and supervision which reinforces training. Briefly, a sample of 225 interviewers was selected from interviewers at three levels of experience. Of the 225 interviewers, 114 were experienced, most of them with more than 12 months interviewing experience; 72 had just finished classroom training but had no CPS interviewing experience; and 39 had completed all classroom training and had 2 to 3 months of CPS interviewing experience. Five scripts covering a variety of labor force situations were used, though not all were used with each group of interviewers. For example, the group having, 2 to 3 months experience were tested with a script containing a potential coverage problem, whereas the other groups were not. The mock interviews were tape recorded and the interviewer performances coded. The most common error found in the British study was "failure to probe". In the study conducted by the Response Research Staff the most common error of experienced interviewers, was. to "alter the scope of the question"; the most common type of error made by the inexperienced interviews was what the British called "invisible recording errors". Of the total errors made by new interviewers at the end of their training, 18 percent were visible, and of those errors made by experienced interviewers, only nine percent were visible errors. Thus, in both the British study and the study conducted by the Response Research Staff, the invisible errors dominated. One aspect of the study report concerns itself with the impact of interviewers' inappropriate behavior on the Employment Status Recode (ESR) and labor force classification. As far as labor force misclassification is concerned, Rustemeyer (1977) reported that 36 percent of the experienced interviewers, 67 percent of the inexperienced interviewers and 61 percent of those with 2 or 3 months of experience made errors that prevented labor force classification or resulted in misclassification. Two examples of hypothetical persons misclassified in the MIP that would affect the employment count are listed below: 1. A volunteer church worker was incorrectly classified in 13 percent of the classifiable interviews as employed rather than not in the labor force. 2. A young man with a job but on extended sick leave was in correctly classified in 20 out of 223 classifiable mock interviews as either unable to work (18 interviews) or unemployed (two interviews) rather than with a job but not at work. The study results showed statistically significant but low correlation between the test results just described and Type A rates (another performance criterion) only for experienced interviewers.- The other correlated performance criteria were for the error rate of first assignments of the interviewers who were inexperienced when tested and their ability to provide information for correct labor force classification. With these exceptions, however, there was no statistically significant correlation between other criteria by which interviewers are evaluated and their ability to obtain information to correctly classify respondents on the test. Thus no other criteria can be considered as substitutes for the test. Further recommendations by the Advisory Committee on Current Program Training included a revision of the training schedule to cover topics according. to their order of importance ta the interviewer's job; improvement in the classroom mock interview exercises; and expansion of the training in the area of interviewing techniques (Love, 1974). E. Development of Quality Control Design The various phases of the data collection operation are subject to regular quality control procedures. These procedures, including the address coverage check, questionnaire checks, interviewer observer, etc. are discussed below. 29 Coverage Check of Sampling Units The Regional Office reviews the listing sheets on which the area segments and special place segments have been listed. The listing sheets used in the area segments are used for sampling and then returned to the interviewer for conducting the interviews. In the special places, the listing sheets are sampled by the Regional Office and the sample units are transcribed to another form and sent to the interviewers for conducting the interviews. For address segments, one copy of the listing sheet remains in the Regional Office and a second copy is sent to Jeffersonville, where listing patterns and differences between the expected number of units and actual units are reviewed; if necessary, corrective action is taken. Area and special place sample reports are reviewed in Washington to see that the sampling instructions have been, applied correctly. Questionnaire Checks Interviewers check for omissions or errors before transmitting questionnaires to the Regional Office. There a check-in and edit operation is performed by the office clerk to see that questionnaires have been received for each sample unit assigned, at FOSDIC markings are correct, that the correct forms have been filled, and that questionnaires are in good condition. During this review errors are corrected, if possible, before the forms are transmitted to the Data Preparation Division for further processing. In the Regional Office the work of new interviewers, and of experienced interviewers whose work is below acceptable levels, is reviewed item by item. In the Washington office the questionnaires of all interviewers are computer edited item by item, and interviewer total error rates are calculated. Interviewers are informed of their errors and ratings before the next month's enumeration (Hanson, 1976). Interviewer Observation Interviewers are observed in the following three situations: (a) as part of the initial training of interviewers; (b) as systematic once a year observation; (c) as special observation when there is evi- dence that the interviewer's performance is below a satisfactory level. The observer accompanies the interviewer for a minimum of six hours during an actual work assignment and notes the interviewer's performance in areas such as probing, recording answers, establishing rapport with the respondent, etc. The result of this observation is a discussion with the interviewer and a written evaluation which is provided to the interviewer. Reinterview 1. The Basic Procedure The prime objective of the reinterview survey is to evaluate interviewer performance, instructions, and procedures with a secondary objective to measure content and coverage error. Reinterview is con- ducted by a Supervisory Field Representative or a member of the supervisory staff. It is begun as soon as possible after the original interview and completed by Saturday of the week containing the 26th of the month. About one-sixth of the interviewers are subject to reinterview checks each month and about one-third of the USU's in his/her - assignment are sampled; thus, the monthly reinterview sample consists of about one in 18 of the monthly CPS sample units. Each interviewer is generally reinterviewed twice a year as part of the reinterview program. The reinterviewer is instructed to follow the same procedures as the interviewer. To preserve the independence between the original interview and the reinterview, the following rules for the prepare on of reinterview material are followed: A person who is to conduct reinterview may not (a) prepare the materials for the reinterview survey for the households he/she is to reinterview; (b) observe the preparation of the reinterview materials for the household in his/her reinterview assignment; or (c) see or edit the original questionnaires for the households he/she is to reinterview. The reinterview procedure involves three basic checks: (a) listing coverage check; (b) within household coverage check, and (c) content check. For certain types of units, the listing coverage check may be conducted by telephone at the same time as the content reinterview. For single unit structures, this check is conducted with the household respondent. For multi-unit addresses, the respondent may be the manager or other person in charge of the building. Listing checks in area segments and special place segments are always conducted by personal visit. Types A and B noninterviews may be contacted by telephone when feasible, but all checks on Type C noninterview must be made by personal visit. 30 In general, the reinterview respondent is selected in the following order according to availability 1. the person for whom the information is being obtained; 2. the original respondent as indicated on the control card; 3. another responsible member of the household, 14 years of age, or over who is qualified to respond for the occupants and who has their permission to do so. The reinterviewers are expected to carry out the following procedures: (a) refrain from looking at the reconciliation record containing the original responses until the interview is completed; (b) make sure the respondent understands the time frame of Is survey week"; (c) make no changes on the reinterview questionnaire as a result of 'information obtained during reconciliation. 2. Reconciliation Data for 80 percent of the units in the reinterview sample are reconciled. By design, the reinterviewer is not supposed to know whether reconciliation, is required for a particular unit until reinterview is completed for that unit. If "omit" is stamped inside the reconciliation form, then the reinterviewer, concludes the reinterview; otherwise the reinterviewer must reconcile the data. When the answers to any questions are not identical or equivalent, then the reinterviewer must ask the respondent enough questions to determine the reason for the differences and which information is more nearly correct. The reinterviewer prepares a summary report for each interviewer whose work is reviewed. The report includes the interviewer's rating, i.e., acceptable or nonacceptable. Table 20 shows the acceptable level of errors for coverage of housing units and persons and for errors in labor force items. Column a shows the total units for which errors from all sources could have been made; column b gives the maximum number of errors that can be made in listing, identifying the sample units, and obtaining the within household coverage for an interviewer's work to be considered acceptable; and column c shows the maximum number of content errors an interviewer can make on the labor force items and still have acceptable work (The CPS Reinterview Survey, 1975). Table 20. Tolerance Table of Acceptable Limits1 Acceptable Level of Errors Number of ____________________________________ Units (Base) Coverage Errors Content Errors (a) (b) (c) 1-14 ............. 1 1 15-20 ............. 1 2 21-22 ............. 2 2 23-32 ............. 2 3 33 ............. 2 4 34-42 ............. 3 4 43-50 ............. 3 5 51-52 ............. 4 5 53-62 ............. 4 6 63-72 ............. 5 7 73-78 ............. 5 8 79-84 ............. 6 8 85-95 ............. 6 9 96-98 ............. 7 9 99-109 ............. 7 10 109-110 ............. 7 11 111-120 ............. 8 11 121-130 ............. 8 12 131-144 ............. 9 13 145 ............. 9 14 146-154 ............. 10 14 155-160 ............. 10 15 161-166 ............. 11 15 167-180 ............. 11 16 181-190 ............. 12 17 191-200 ............. 13 15 201-215 ............. 13 19 216-220 ............. 14 19 221-230 ............. 14 20 231-250 ............. 15 21 251-270 ............. 16 22 271-275 ............. 17 22 276-290 ............. 17 23 291-300 ............. 15 23 1The CPS Reinterview Survey, 1975. 3. Results from Reinterview Specific to the number of persons classified as employed in the CPS, the reinterview provides information on how many persons were classified as employed in the reinterview. If one is willing to accept the reinterview as a standard, then the difference between the original interview and the interview can be used as a measure of bias. Table 21 shows the results in the two estimates of employment annually from 1956 through 1976. For several years,, the employment rate in the reinterview was about 0.2 percent lower than the original interview. However, in more recent years .the difference between the percentage of persons employed as measured by the reinterview and the original interview has increased; in 1976 the reinterview was 0.7 percentage points lower. When applied to an employment figure of 80 million, these differences account for between 160,000 to 560,000 persons. Though many of the estimates shown in Table 21 would be within sampling error, the consistent direction of the difference signals a potential problem. 31 Table 21. Summary of Percent of Persons Employed as Measured in the Original CPS Interview for the Reinterview Subsample and as Measured by the Reinterview after Reconciliation, 1956-19761 Percent of persons in labor force employed Reinterview Year Original reinterview minus original 1956 ... 96.3 96.1 -0.2 1957 ... 95.8 95.8 -0.0 1958 ... 93.2 93.0 -0.2 1959 ... 94.4 94.2 -0.2 1960 ... 94.6 94.4 -0.2 1961 ... 93.1 92.8 -0.3 1962 ... 94.1 94.5 -0.0 1963 ... 94.4 94.0 -O.4 1964 ... 94.8 94.3 -0.5 1965 ... 94.9 94.7 -0.2 1966 ... 96.1 95.8 -0.3 1967 ... 96.2 95.8 -0.4 1968 ... 96.3 96.0 -0.3 1969 ... 96.3 95.9 -0.4 1970 ... 94.9 94.5 -0.4 1971 ... 94.1 93.7 -0.4 1972 ... 94.7 94.4 -0.3 1973 ... 95.0 94.7 -0.3 1974 ... 94.5 93.9 -0.6 1975 ... 91.8 91.2 -0.6 1976 ... 92.5 91.8 -0.7 1 Based on data tabulated quarterly and annually by the Bureau of the Census in regular reinterview tabulations. Table 22. Annual Interviewer Error Rates1 Gross Error Rate Net Error Rate Household Household Year Listing Composition Listing Composition 1973 ........ 1.13 0.46 -0.52 -0.23 1974 ........ 0.99 0.44 -0.14 -0.23 1975 ........ 0.87 0.44 -0.17 -0.17 1976 ........ 1.15 0.42 -0.32 -0.18 1 Moye, 1976a and Schreiner, 1977. Table 22 shows the results of the reinterview listing check for 1973-76. The gross error rate is the sum of the erroneous omissions and inclusions divided by the total number of listings/household members before the reinterview, whereas the net error rate is. the difference between the erroneous inclusions and omissions divided by the total listings/ household members. A negative net error rate de- notes an understatement in the original survey. The annual gross and net listing error rates showed a decrease between 1973 and 1975; however, they both increased again in 1976. The interviewers erroneously missed an estimated 0.73 percent of the units listed in the original interview and erroneously added an estimated 0.41 percent of the units resulting in a net error rate of - 0.32 percent. The household composition gross and net error rates in 1976 were 0.42 percent and -0.18 percent. respectively. Neither of these rates differed significantly from the rates for 1973-1975. These error rates are representative of a percentage of eligible units and/or persons that are not represented in the CPS sample and could affect the employment level. though admittedly the effect would be small. Table 23 shows the noninterview misclassifications for the years 1973-76. There was an annual misclassification rate of 2.4 percent for 1976 which was significantly different from the 1974 rate only. The table shows that there are more Type A noninterviews that are misclassified as B's or C's than the reverse. This means that there is then a deficit in the number of households eligible for interview which has a small effect on the CPS weighting and thus the estimate of employment. Limitations of the Quality Control Procedure 1. Quality Control of Telephone Interviewing Quality control of the CPS data collection procedures does not extend to one important area. Although 62 percent of the CPS is done by telephone, there is currently no classroom training, on the-job training or observation for that type of data collection. Nearly all telephoning is done from interviewers' homes and Bureau rules prohibit Bureau staff from going to interviewers' homes. Thus the only quality control on 62 percent of the interviews is the reinterview program. (A telephone self study, however, is currently being written in the Training Branch). 2. The Reinterview Procedure The CPS reinterview procedure, used as a quality control mechanism on the observational design and implementation, is itself subject to limitations in Table 23. Noninterview Misclassification Rates (Percentages)1 Misclassification 19732 1974.3 1975.3 1976.3 Total........... 2.7 3.3 2.9 2.4 A's as B's........... 1.4 2.1 1.8 1.6 B's as A's........... 0.7 0.4 0.2 0.4 C's as B's........... 0.45 0.7 0.6 0.4 Other................ 0.15 0.1 0.3 0.03 3 Moye, 1976a and Schreiner, 1977. 2 B's as C's, C's as A's, and A's as C's. 3 Base-total noninterviews. 32 design and implementation. These limitations, discussed below, should be kept in mind when use is made of the results from the reinterview. a. Independence For best results, the reinterview and original visit to a household should be obtained under the same conditions, or in other words, the reinterview should be independent of the original visit. However, this is not possible. The reinterview is conducted approximately 1 week after the original visit and it is highly probable that the reinterview respondent has been conditioned by his/her previous response or that of the original respondent. b. Reinterview nonresponse rate For content coverage checks, interest is in the difference between the original and reinterview results. Therefore, for reinterview, the nonresponse households consists of those households selected in the reinterview for which no response was obtained in either the original interview, the reinterview or both interviews. For the first quarter of 1976, the reinterview nonresponse rate was calculated as 9.8 percent. An additional 1.6 percent of the sample households were not matched. For all practical purposes, the nonresponse rate was 11.4 percent. To the extent that these house- holds differed from the matched households, the reinterview results are biased. c. Coverage check The reinterview survey accounts for only a small percentage of the total missed persons. The estimated average rate of net coverage error in the count of persons, as determined the reinterview in 1974, was approximately 0.22 percent. However, the independent estimates of population in 1974 indicate an undercoverage in 1974 of approximately 3.9 percent. In the summer of 1966, it was suggested that the reinterview survey. might not be finding missed persons because of the emphasis on checking content in the reinterview. As a consequence, an experiment was conducted in October of 1966 and in June 1967, in which the entire reinterview was devoted to coverage. These results were compared to the coverage check results of the reinterview periods preceding October 1966 and June 1967. The experiment was referred to as the intensive coverage check. Table 24 summarizes the coverage comparisons. The October intensive coverage check indicated that 1.74 percent of the listed units were missed as compared to 0.83 percent in the preceding 6 months. The comparable figures for June and the preceding 7 months were 1.26 and 0.78, respectively. The number of persons missed in properly interviewed units was estimated as 0.80, in October and 0.36 for the preceding 6 months. A significant source of coverage error seems to be due to the misclassification of noninterviews. Misclassification Type A noninterviews as Type B or C results in a loss of persons; misclassifying Type B or C noninterviews as Type A results in the erroneous inclusion of persons (Technical Paper No. 19, 1968). Table 25 shows the comparison of the reinterview classification of units to the original classification for October 1966 and the preceding months and June 1967 and preceding Click HERE for graphic. 33 months. For the period April-September 1966, the reinterview found that 3.5 percent of the units originally classified as Type B's and 4.7 percent of the units originally classified as Type Cs should have been classified as Type A's. Correspondingly, the October 1966 reinterview found these percentages to be 10.1 percent and 12.1 percent, respectively. For June and the 7 months preceding June, the table shows the two periods of the classification of the interview status of all units reported in either. the original interview or the reinterview. The discrepancy between the percentage of Type B's and C's that should have been classified as Type A's was very small with a larger percentage found during the first period. Fewer noninterview - misclassifications and people missed in properly interviewed units between the June intensive coverage check and previous months were found than between the October check and previous months. A possible explanation is that the June interviewer group training covered the classification of noninterview units and the rules of residency for occupants of units. From the results of the study, it was felt that the reinterview would do a better job of coverage if more attention were given to this aspect of the program; further, it was felt that the interviewers needed more training on residence rules, listing, etc. One point to keep in mind is that even in the intensive coverage check, the reinterview did not account for all of the undercoverage. In October 1966, the independent estimates of population indicated an undercoverage of 2.3 percent compared to the 1.7 percent estimated in the intensive coverage check, and in June 1967, the comparable figures were 3.8 and 1.0. d. 80 Percent vs. 20 Percent Sample As previously stated, the reinterview survey consists of a 20 percent unreconciled sample and an 80 percent reconciled sample. Past reinterview results have shown a consistently lower rate of difference in the 80 percent sample before reconciliation than in the 20 percent unreconciled sample. These significant differences are attributed to the reinterviewers not following the rules of the reinterview procedure. There are several possibilities: (1) the reinterviewer may be looking at the results of the original interview before recon- ciliation; (2) the reinterviewer may be changing the reinterview questionnaires after reconciliation, or (3) the reinterviewer is detecting the serial number pattern on the questionnaires after con- ducting a few reinterviews and is subsequently not doing as good a job on those households he/she knows will not be reconciled. Table 26 shows the differences between the two samples for the years 1973-1975 for six labor force categories. The category "employed" is not presented. but the six categories shown do indicate a problem in the reinterview procedure which might also affect the data on the quality of the employment estimate. In summary, the reinterview survey is a useful tool for evaluating interviewer performance, its primary purpose. Each interviewer is rated on his/her performance in the areas of both content and coverage. If the reinterview indicates a need. the interviewer is retrained and observed before his Click HERE for graphic. 34 Click HERE for graphic. 35 next regular assignment. It is felt that the reinterview program is instrumental to maintaining interviewer control by discouraging "curbstoning", etc. However, though the measurement of the overall quality of the coverage and content in CPS is a secondary function of the reinterview survey, it is still used for this purpose. Users of this information should be aware of the limitations of the reinterview program in its secondary role and use these results with caution. CHAPTER IV Data Processing Processing of the data is an integral part of the CPS survey procedure, and its proper operation has a large effect on the accuracy of the data. This chapter includes a description of all the major data processing operations from the time the questionnaires are received in Jeffersonville, through the data input operation- microfilming and FOSDIC-and the editing and imputation of missing data. Potential sources of error associated with these major steps in the survey procedure are discussed. Questionnaires are shipped from the Regional Offices on a daily basis to the Data Preparation Division (DPD), Jeffersonville, Indiana. The shipments are logged in and the count of documents as recorded on the transmittal are verified. The documents that represent interviewed units are assigned industry-occupation codes by the clerical staff and verified. Since no coding is done for employment by age, race, or sex, coding has no effect on the statistic and thus will not be discussed here. A. Microfilming/FOSDIC After coding, the questionnaires are microfilmed, which is an integral part of the Bureau of the Census' Film Optical Sensing Device for Input to Computers (FOSDIC). The major part of the two operations, microfilming and FOSDIC, are conducted in DPD. Output from the FOSDIC operation is relayed to Washington in work units. At the receipt of each work unit, the data tapes containing this information are subjected to a computer data acceptance run which checks the readability of the tapes and rejects questionnaires because of poor FOSDIC marks, missing questionnaire pages, etc. These errors are then corrected in DPD and the data from these questionnaires are enclosed in a later shipment. The FOSDIC/Microfilming Procedure FOSDIC can be described as a process which uses a machine capable of "reading" information from a microfilm copy of an appropriately designed schedule and transferring this information to magnetic tape for processing on electronic computers. Basically, FOSDIC operates as follows: A beam of light locates itself on the microfilm by centering on an index mark (a black square printed on the FOSDIC schedule). A series of circles is in the vicinity of each index mark, and marks in these circles convey the information recorded by the person who fills the document. The scanning beam moves from the center of an index mark to each of the circles associated with the mark, thus enabling it to identify which circles have been marked. A code indicating which circles are filled is recorded by FOSDIC on magnetic tape (McPherson and Volk, 1962). FOSDIC consists of four separate units: 1. The scanning unit which controls the beam both horizontally and vertically and decides where a mark has been made. It also electronically adjusts FOSDIC to optimize the prob- ability of correctly reading each frame (questionnaire page). This series of operations is referred to as calibration. 2. The program unit which "tells" the machine what to do, including the order in which the questions are to be scanned and the distances to travel to find the questions. 3. The tape unit which holds the magnetic tape, drives it at a constant rate, records impulses on tape, and at the end of a run, rewinds the tape for computer use. 4. The console which contains all the operating switches, recording dials, cathode ray tube, and oscilloscope for testing the strength of electrical impulses. In addition the camera equipment, film, and film development used in microfilming are all an integral part of the CPS FOSDIC procedure. Potential Sources of Errors in the FOSDIC/Microfilming Procedures Several variables play a role in the microfilming, and FOSDIC procedures and adherence to standards can determine the success or failure of this aspect of the data processing. These variables include, but are not limited to, the quality of the paper and the printing used in the questionnaires; the uniformity of the index marks and marking circles on the questionnaire (Figure 3 which is a copy of one page of the CPS questionnaire shows these marks); and of course, the proper operation of the FOSDIC and 37 microfilming equipment. The potential sources of error associated with the quality of these variables are discussed below.' The data acceptance runs, the quality control program, and the CPS/FOSDIC study all have contributed to the knowledge of errors in the system. 1. Results from the Data Acceptance Runs Table 27 shows a distribution of questionnaires rejected by FOSDIC during the data acceptance runs by cause of error for the year 1976. FOSDIC will reject an entire questionnaire if one questionnaire page does not conform to standards; i.e., in scanning, if it is found that a questionnaire page does not conform to size, a page was missed in microfilming, the index marks are displaced, etc. Table 27 shows that in January, out of 72,172 total questionnaire forms, 1,074 or 1.49 percent of the questionnaires were rejected. Of these 1,074 questionnaires, 379 or 0.5 percent were rejected because of FOSDIC/filming errors. It is quite possible that some of the interviewer-associated rejects are actually FOSDIC related; it is not always possible to distinguish the difference. Figure 4 shows the documents rejected because of FOSDIC/microfilming errors (unbroken line) as compared to the January 1974-December 1975 average (dotted line) and the upper control limit (dashed line). In February and September, the percentage of total documents rejected exceeded the upper control limit-in February, because of misprinted documents and in September, because of FOSDIC. Otherwise, the percentage of documents rejected was lower than the previous 2- year average. These errors in FOSDIC/filming detected in the data acceptance runs that are associated with filming problems, bad index marks, etc. can be corrected; for example, if the questionnaire is "bad", data are transcribed to another questionnaire and questionnaires with missed pages are remicrofilmed. The errors that represent a threat to data accuracy are the "invisible errors", i.e. errors that cannot be detected. An example is a FOSDIC pickup of an incomplete erasure as - a mark. In addition, FOSDIC itself is subject to a certain amount of measurement error; it is possible with the same tolerance levels for reading marks, that it can get different readings for the same marks. The error is, however, small. Because of the tight time schedule, final processing is allowed to continue when the number of missing documents or uncorrected documents is reduced to 75 or less, generally less. To the extent possible these documents are appropriately treated as Type A, B, or C noninterviews. 2. Results of Test Run of Blank CPS Questionnaires A sample of blank questionnaires is selected each month as part of the quality control program 39 and run through the FOSDIC procedure. The sample documents are filmed and tested prior to and during processing in Washington and in Jeffersonville. Table 28 gives the results for January 1976 through December 1976 of the number of spurious pickups detected on the sample documents. The tests also specify the number of failed calibrates, failed indices, and drops. For example, in the Washington December test of 425 documents run before processing, there were no failed calibrates, one failed index, in addition to six spurious pickups. In the test performed during processing, there were 27 failed calibrates, two failed, indices in addition to the four spurious pickups. The high number of failed calibrates was attributed to problems with FOSDIC rather than the questionnaires. One FOSDIC drop was detected in the 100 percent PRERA (Preliminary Edit Recode and Allocation) Error Listing. These numbers, when considering the total number of items checked, are quite small. For instance, in the December preprocessing test of the 425 documents, 262,650 items were tested; thus the six pickups represented .0023 percent of the items tested. However small the percentages are, these are the types of error in the FOSDIC procedure that are not likely to be detected and therefore become a part of the data output. Figure 5 shows the percentage of spurious pickups detected in tests conducted prior to processing in Washington and Jeffersonville as compared to both an average based on past performance and on an upper control limit of pickups; Figure 6 shows the same information for tests conducted during processing. The Jeffersonville test exceeded the upper control limit in May because of filming. In June the Washington percentage of pickups exceeded the upper control limit because of FOSDIC problems. Obvious problems with FOSDIC or microfilming are investigated and corrected. 3. Variation in FOSDIC Scanning of Monthly Quality Control Samples The original film of several months' quality control samples in 1973 and 1974 were run several times on the FOSDIC equipment. Using the number of errors (pickups and drops) for each run, the observed and expected variances were calculated. (ne expected variances were actually the average variances over several months of runs.) Table 29 shows the results of the experiment. For four of the months the observed and expected variances were found to be significantly different; thus, it was concluded that more than random variation was in the system (Linebarger, 1974). 4. CPS/FOSDIC Study Between 1973-75, a CPS/FOSDIC Study was conducted to identify specific sources of variations in the system. The study included processing through the standard CPS/FOSDIC facilities. measurement of variables associated with paper quality, film density, etc. Several aspects of this study are discussed below. One major aspect of the CPS/FOSDIC Study involved the reading of two identical pages (both containing labor force data) of 300 CPS questionnaires. The two pages were filled in identically for each of the 300 documents so that there were then 600 identical pages of information. These 600 identically filled pages in the experimental process came from the Current Population Survey documents of November 1973, January 1974, and April 1974 and were marked by one individual. The questionnaires were selected in a purposive manner to incorporate the broadest range of fluctuation for each of the crucial variables. Filming was done during November 1974, and FOSDIC readings were conducted from January through April 1975. Five different cameras filmed the 600 pages producing 15 tape reels, three different FOSDIC readers read the microfilm reels four times each. A total of 36,000 readings was expected but because of loss of one microfilm reel and five questionnaires, there were 32.997 observations instead. There were 22 or 23 marks on each questionnaire depending on the month of the questionnaire, 39 blank questions. and 62 read areas on each questionnaire page. Out of the 32,997 attempted reading, the errors observed are shown in Table 30 So few FOSDIC system errors occurred that it was not possible to carry out the originally planned regression analysis of causes of the errors. However, some of the causes of errors could be pinpointed. For example, it was detected that nine of the drops occurred because of a single index probably mislocated by the FOSDIC reader and 30 of the total pickups occurred because of two large spots of scratched emulsion on two of the 14 reels of microfilm used in the experiment. One microfilm questionnaire page was read without error four times and then failed calibration Table 29. Variation in the FOSDIC System1 Variance Expected Month Observed Variance F-Statistic June 1973......... 24.50 11.22 2.18 October 1973...... 16.33 31.19 0.52 November 1973..... 481.00 34.25 14.04* December 1973..... 217.33 24.15 9.00* January 1974...... 3306.33 10.19 324.52* February 1974..... 84.00 8.82 9.53* 1 Linebarger, 1974. * Significant at 95 percent level. eight times in succession, accounting for 30 percent of the failed calibrates, one cause of skipped pages. Table 30. Some Results of the FOSDIC Experiment Problem Area Number Percent of Total Drops of marks........... 22 .0029 Pickups of blanks........ 44 .0034 Drop-and-pickups......... 1 .00013 Skipped read areas....... 260 .013 Skipped pages............ 27 .082 Source: Boisen, 1975. Some of the major findings resulting from this aspect of the CPS/FOSDIC Study were that (1) basically the system as operated during the experiment was under control with system error so slight that improvement could be impossible; (2) quality control procedures should be extended to the marking of CPS questionnaires and that (3) further investigation might pinpoint some nonrandom and significant sources of error that result in failed calibrates and missed indices (Boisen, 1975). The reflectance, opacity, and brightness of the questionnaire paper affect the readings of marks by FOSDIC. Thus, another aspect of the CPS/ FOSDIC study concerned the investigation of the quality of the CPS paper. Two hundred and eight blank CPS questionnaires were analyzed by the National Bureau of Standards where measurements were made on the reflectance, opacity, and brightness of ten positions on each of the documents. Six measurements were made on a righthand page and four on a left-hand page. If the opacity of the paper is below standard, FOSDIC could produce spurious pickups from markings on the other side of the page or if the reflectance or brightness is not at the required level, FOSDIC could "drop" marks that are made lightly because it is unable to distinguish the marked circles from those that are unmarked. Table 31 shows the average reflectance of the ten positions over the 208 documents. High reflectance is expected of white paper, with or without printing on the reverse side, and of unfilled marking guides; low reflectance is expected of black index marks. There is a statistical difference between positions one and two and between three and five. The explanation for this is that some bleed-through from the index occurs on the reverse side of the page. If bleed-through reduces the reflectance level too-much, spurious pickups can result (O'Reagan, 1973). Table 32 shows the results of the measurements of opacity and brightness that were performed on the documents. A large percentage of the documents were outside specifications, particularly with regard to the brightness scale. The paper was somewhat inferior to that of the March 1973 documents which exhibited poor FOSDIC results in relation to other months. Whereas the use of paper outside specifications does not automatically insure errors, it no doubt does increase the probability of drops and pickups. Further tests were performed in the CPS/ FOSDIC study. One such test was on the correlation of film density and false pickups. Film from the March 1973 CPS (which was a particularly bad month for FOSDIC) was used in the study. The conclusion was that a lowered upper limit for acceptable density of film would result in fewer false pickups (Linebarger, 1973). Other tests were performed on camera illumination and Jeffersonville microfilming. The illumination level during microfilming was in good control and the Jeffersonville film development showed good performance. However, tolerances in Jeffersonville microfilming were sometimes overridden by more liberal verbal specifications due to some lack of Table 31. Measured Reflectance of Ten Questionnaire Positions1 Average Description Reflectance 1. White paper with no printing on reverse 75.68 2. White paper with black index on reverse 68.39 3. Marking guide with no printing on reverse 67.43 4. Marking guide with black index on reverse 64.59 5. Black index 8.26 6. Marking guide with no printing on reverse 66.83 7. Marking guide with black index on reverse 65.27 8. Black index 7.89 1 O'Reagan, 1973. 45 Table 32. Results of Test on Opacity and Brightness of CPS Questionnaires1 Outside Specifications Number Opacity Brightness of Month-Year Reading Number Percent Number Percent January 1970...... 5 0 0.00 5 100.00 February.......... 5 0 0.00 5 100.00 March............. 3 0 0.00 0 0.00 April............. 6 0 0.00 0 0.00 May............... 3 0 0.00 3 100.00 June.............. 6 0 0.00 6 100.00 July.............. 4 0 0.00 4 100.00 August........... 5 0 0.00 5 100.00 September......... 5 0 0.00 0 0.00 October........... 3 0 0.00 1 33.33 November.......... 3 0 0.00 3 100.00 December.......... 5 0 0.00 3 60.00 Total 1970... 53 0 0.00 35 66.04 January 1971...... 5 3 60.00 4 80.00 February.......... 3 0 0.00 3 100.00 March............. 2 0 0.00 1 50.00 April............. 4 0 0.00 4 100.00 May............... 6 0 0.00 2 33.33 June.............. 6 3 50.00 6 100.00 July.............. 4 4 100.00 2 50.00 August............ 3 0 0.00 3 100.00 September......... 5 0 0.00 5 100.00 October........... 6 1 16.67 0 0.00 November.......... 4 1 25.00 1 25.00 December.......... 1 0 0.00 1 100.00 Total 1971... 49 12 24.49 32 65.31 January 1972...... 6 0 0.00 6 100.00 February.......... 6 0 0.00 6 100.00 March............. 1 0 0.00 1 100.00 April............. 6 0 0.00 0 0.00 May............... 6 0 0.00 5 83.33 June.............. 6 0 0.00 0 0.00 July.............. 6 0 0.00 0 0.00 August............ 6 0 0.00 0 0.00 September......... 6 0 0.00 0 0.00 October........... 6 0 0.00 0 0.00 November.......... 6 0 0.00 0 0.00 December.......... 6 0 0.00 0 0.00 Total 1972... 67 0 0.00 18 26.87 January 1973...... 6 1 16.67 0 0.00 February.......... 6 1 16.67 0 0.00 March............. 15 4 26.67 0 0.00 April............. 6 1 16.67 0 0.00 May............... 6 1 16.67 0 0.00 Total 1973... 39 8 20.51 0 0.00 1 O'Reagan, 1973. Specifications: Opacity-not less than 90. Brightness--78-81. understanding of the importance of tight controls (O'Reagan, 1974). In summary, though FOSDIC system error exists, it is small, and the use of the FOSDIC represents a gain in accuracy when compared with previous keypunching of the data which involved more human error. Still some variables that affect the system, such as the paper quality, deserve continued monitoring. B. Editing and Imputation Procedure for Editing and Imputation for Missing Values The first stage in editing is the determination of whether a household is an interviewed or noninterviewed household. Even within households that are classified as interviewed, there may be persons who 46 have very little information recorded on the schedule. To be considered a "person" there are 12 combinations of items that are - reviewed and data must be present for one of these combinations. These combinations are shown in Table 33. If data are not present for one of these combinations, the record is excluded from processing in the data acceptance runs; if data are present, all remaining items with missing values are imputed values. The imputation for labor force items is done by means of a "hot deck" procedure. The sample is partitioned into two sex, two race, and five age groups (14-17, 18-24, 25-44, 45-64, 65 and over). The basic assumption is that labor force status is highly correlated with sex, race, and age. However, in order to begin the imputation process, there must be entries for the age, sex, and race items. Therefore, an editing process takes place first. The editing pro- cedure is as follows: 1. The "relationship" item is reviewed. A "head" is assigned to each household without a person so designated. It is either the first listed male, married-spouse present or, if no such person is in the household, the first person listed of any type. If two or more "heads" are designated, only one is retained; similarly, only one "wife" is retained. Other persons with blank relationship are assigned codes based on a 10-category age, sex, hot-deck matrix. However, if the age and sex are also blank, the hot-deck matrix also has a code for these cases. 2. Sex is assigned after relationship has been edited. Editing is performed for consistency such as any head with "wife" present in the Table 33. The Possible Item Combinations Used in Determining The Existence of A Legitimate Person's Record1 Combination Number Item Numbers2 1 ........................ 19, 23e 2 ........................ 19, 24 3 ........................ 19, 26, 28 4 ........................ 19, 26, 30 5 ........................ 19, 28, 30 6 ........................ 23e, 26, 28 7 ........................ 23e, 26, 30 8 ........................ 23e, 28, 30 9 ........................ 24, 26, 29 10 ........................ 24, 26, 30 11 ........................ 24, 28, 30 12 ........................ 26, 28, 30 1 Bailar, Bailey, and Corby, 1977. 2 See Figure 2 for questions related to the given item number. household is assigned as "male." Heads without wives are assigned the sex code of the last head processed. Other relatives and nonrelatives are assigned as male and female alternately. 3. Race is assigned after relationship and sex have been allocated. If anyone in the household has an entry, missing entries are supplied from the first person listed who has an entry. If no one in the household has an entry, the race of the immediately preceding household with an,entry is assigned. 4. Marital status is the next item reviewed. Heads with wife present or persons listed as "wife" are assigned a code of "married, civilian spouse present". Other heads (with relatives) are assigned marital status from a hot-deck matrix based on sex and age or, ff the age is missing, from the entry for the last head of the same type and of the same sex. Persons other than heads with wives or other relatives are assigned marital status from a hot-deck matrix based on relationship, sex, and age. If age is missing, a code is assigned from the last person in the 'same sex and relationship group. 5. Age is the next item reviewed. For a husband-wife family, if the age group of one but not the other is known, the missing age is supplied from a hot-deck matrix based on the age group of one spouse. For a household with other relatives if the age of the head is known, missing ages for other relatives are assigned from a hot-deck matrix based on age group of the head, sex of other relatives, and marital status of other relatives. For nonrelatives, heads with no relatives in the household, and cases where the age of head (and wife, if any) are unknown, ages are assigned from a hot-deck matrix based on relationship, sex, and marital status. The editing and allocation continues for other sociodemographic characteristics, but none of the other characteristics bear on the labor force status entries. An employment status recode is now determined for each person. The possible recodes are as follows: 1. At work 2. With a job but not at work 3. Unemployed 4. Keeping house 5. Going to school 6. Unable to work 7. Other nonworker Codes 1 and 2 classify persons as being employed. The classification takes account of the priority order of the questions and certain special rules. Generally, two indications of work (or unemployment) are required before a person is recoded into the labor force. For example, if a person were coded as working, evidence would also be reviewed from hours worked, occupation, or some other item. Persons without any entries in the entire series of employment status items, formerly deleted from the sample, are now retained and assigned a set of values and an employment status recode from a hot- deck matrix based on sex, race, and age. Persons with missing entries in some of the employment status items are also assigned values from a hot-deck matrix. The process is begun with the storing in the computer of a "cold-deck". The colddeck matrix is based on past survey data and the values are used only as starting values. The colddeck matrix is updated periodically to reflect changes in the population. If the first record to be processed has complete labor force information, the labor force status for that record replaces the cold- deck value for that age, sex, and race group. If the first record has missing labor force information, the value from the colddeck is assigned. The hot-deck matrix is continually updated to reflect the most recently processed sample cases. Each missing value is assigned the code from t he most recently processed case in the appropriate subgroup. The same record can be used to assign missing values for many different records. Table 34 shows the number of times amputations were made for different labor force items. because of omissions or inconsistencies over the 11-month period from February through December, 1975. Considering that about 110,000 records are processed each month, the effect of imputation is probably quite small. It would be valuable t o have a frequency distribution for the number of times each record was used to impute values for, missing items. Such a distribution is not a by-product of the CPS processing, but some data are available. Table 35 shows the average number of times a record was used to impute missing data if the record were used at least three times. The data are for the same 11-month period, February through December 1975. Potential Sources of Errors Arising from the Editing and Imputation Procedure The success of the editing and imputation process depends on the correlation of labor force status with age, sex, and race. To the extent possible, values are allocated that use information for the household members. However, there are some flaws in the procedure. 1. There is no probability mechanism operating in the assignment of missing values. The value assigned depends on the sort order of the file. It is impossible to determine the exact probability that a person with missing values for labor force status will be imputed an "employed" status. 2 The same record can be used several times to supply missing information. Although the effect for the total U.S. may be negligible, the effect could be far from trivial for labor force data for smaller areas, such as SMSA'S. No data are available to evaluate such an effect. 3. The variances are estimated after imputation. The effect of imputation is to increase the number of records with the same value, thus decreasing the sums of squares. Thus, the variances are underestimated, though probably not by any appreciable amount. 4. The hot-deck procedure causes an increase in variance. It has been shown by Bailar, Bailey. and Corby (1977) that the variance of the hot-deck procedure is greater than that of simple random sampling with no nonresponse and also greater than an imputation method in which the mean of the subgroup is imputed for all nonrespondents. However, it is uncertain whether the use of the hot-deck procedure increases the mean square error. In summary, there is little evidence of the effect of the editing and imputation on data accuracy. As a major step in the processing of the data, more research should be conducted in these areas. C. Quality Control Quality Control of Microfilming/FOSDIC Procedure1 The quality control of the FOSDIC procedure involves the control of the design, paper specification ________________________________ 1 The majority of this section was taken from a Bureau of the Census memorandum written by J. Linebarger and W. Davie (1973). 48 Table 35. Average Frequency With Which A Value From the Same CPS Record Is Used For Imputation For item Nonresponse1 . 2 Average Average Item Value Frequency Item Value Frequency 20A 1 3.50 26 4 3.55 25 3.00 6 3.00 32 4.00 27 14 4.00 33 3.00 15 3.00 38 3.00 21 3.00 40 3.50 25 3.00 44 3.00 28 7.00 45 4.00 42 9.00 60 3.00 28 3 3.00 65 3.00 4 3.50 20C 111 4.00 29 1 6.10 115 3.00 2 5.10 212 3.00 3 3.75 213 3.75 30SX 2 3.00 21B 1 3.00 30VT 1 3.55 22B 1 3.00 2 4.00 22C 62 3.00 6 29.66 22D 1 3.00 31E 6 3.00 22F 1 3.21 31H 9 3.00 2 4.50 10 17.00 24A 1 3.03 11 3.80 2 3.33 12 3.17 3 3.40 31C 13 3.00 4 3.00 14 3.00 5 3.00 15 4.00 6 3.60 32 1 4.00 7 3.50 2 3.67 1 Bailar, Bailey, and Corby, 1977. 2 Only records which were used more than three times for imputation were included in the computation of the average. and printing of the CPS questionnaires; FOSDIC scan; cameras; film; and FOSDIC production scanning. The design specifications for most of the processes in the CPS FOSDIC are given in a manual by Greenough and Epley (1969) of the National Bureau of Standards. The design of the quality control program specifies the failure conditions/sample size/timing, the action to be taken when specifications are not met, and the type of records to be kept of problems in the system. Some of the procedures are outlined below: 1. Design of the Questionnaire The quality control for this procedure is intended to insure proper layout of documents. Upon receipt of the diapositive of the questionnaire from the Government Printing Office (GPO), measurements are to be made on the distance of the index marks from the marking guides, lines, etc, and thickness of the marking guides. If any measurements fail to meet specifications, then the questionnaire must be redesigned. 2. Selection of Paper To insure against the use of substandard paper, GPO must select a sample from each skid prior to printing and get an opacity reading. If the paper is substandard, GPO is expected to reject the paper. In addition, GPO selects a sample of 2/2000 sheets during printing and forwards this paper to the Engineering Division at -Census. If four or more out of every 60 sheets have an opacity reading under 90 percent, then the questionnaires should be rejected and reprinting re- quested. A record is kept by GPO and Census of all "paper" problems. 3. Printing The quality control of the printing is designed to insure against imperfections such as lightness and darkness, blurring, bleed-through, etc. A ten percent subsample of the 2/2000 sample sheets is selected at the Bureau for testing. A sample of ten marking guide circles per document are selected and tested for thickness; in addition a Print Contrast Ratio (PCR) is applied to the documents. If two or more circles fail the circle line thickness or two questionnaires fail the PCR test, a second sample is selected. If the second sample 49 is deficient, reprinting is requested. A record is kept of the printing problems. 4. FOSDIC Scan This procedure guards against obvious defects in printing and binding of,the questionnaire and provides a means of alerting proper authorities of possible production problems. The Demographic Surveys' Division selects a 0.5 percent (0.75 percent in March) of bound CPS questionnaires upon their receipt at the Bureau; these are scanned on FOSDIC to anticipate difficulties such as false pick-ups that may occur during regular processing. Problems are discussed with programmers and subject matter divisions and final accept/reject decisions are made by theme Records are kept of this procedure (see Table 28). 5. Cameras The quality control of camera equipment insures the proper functioning of camera focusing, tilt, etc. in order to keep remicrofilming to a minimum. A monthly "Step Test" is performed on each camera prior to CPS production work. The test is used to obtain the proper light setting based on checks on the focus, lens cleanliness, and shutter speed. In addition, a test referred to as the twice-daily dip test is performed at the beginning and middle of each shift every day on each camera during production. The items checked in this test are the light meter reading, alignment, spacing, focus, density, etc. Maintenance is performed on cameras in instances when design specifications are not met. If a camera fails the test on any items checked in the Dip Test, it is not approved for production. 6. Film The quality control of the film insures proper developing of film including proper density, contrast, etc. It is required that film be developed within 24 hours of exposure, and that the film be kept under refrigeration. If refrigeration is not available, then the documents must be refilmed. Checks are also made on the temperature, focus, and density. All records from rejected rolls of film are refilmed. 7. Production Scanning (FOSDIC) Quality control insures proper functioning of each FOSDIC machine. The procedures include the keeping of a general maintenance record, the running of a test deck each shift to measure the accuracy of readings; and the rejection and refilming of any roll of film from which failed calibrates or failed indices exceed 15 frames or 25 indices, respectively. If the test deck fails on any FOSDIC machine, then adjustments must be made on the machine and the entire roll of film rescanned. Basically, the CPS FOSDIC procedures are followed. However, because of time limitations, it is unusual for the Bureau to request reprinting of the documents because the paper or its printing falls below specification. Quality Control of Editing and Imputation There is no formal quality control procedure of the editing and imputation procedures. The basic computer program used to perform these procedures has been in use since 1967. Several test runs are made after any changes are made in the program. Output from the computer program includes a listing of errors by questionnaire and summarized over all questionnaires for each item. These listings are spot checked; if the error rate based on the number of inconsistencies or blanks for a questionnaire item per total number of records exceeds 0.5 percent, a check is made of the item to verify that the error rate is legitimate. If the errors are caused by bad microfilming or FOSDIC, then the questionnaire can be refilmed and rescanned. In addition there is a 100 percent verification of the error listing of the first 3000 questionnaires. This checks the data acceptance runs used to check all the data tapes, the operation of the cameras, and the FOSDIC machines. 50 CHAPTER V Estimation There are several steps in the preparation of the final estimates of employment. First is a weighting procedure to adjust for noninterview and undercoverage. Then, a series of estimates, each made at a different stage of the weighting procedure, is formed. Next, the composite estimate is computed. Finally, the Bureau of Labor Statistics handles the seasonal adjustment of the data. . Along with the estimates of employment, estimates of the sampling variances are also computed. Quarterly, some estimates of simple response variances and bias are made. A. Weighting Procedure Specification of weighting procedure The CPS design calls for a self-weighting sample. i.e., a sample in which each sample household has approximately the same Probability of selection. However, because of nonresponse and coverage problems, a reweighting of the records occurs before the estimates are produced. In the CPS there are five distinct steps in the reweighting process. These are as follows: 1. The reciprocal of the probability of selection is attached to the record for a given sample unit. 2. During listing it may, have been found that certain USU's contained far more units than were expected based on the census listing, so subsampling took place to make the inter- viewer workload manageable. The weights for the subsampled units are now, multiplied by a factor which inflates these units to reflect the actual number of units in the USU. This process is called duplication control. The maximum. factor used is four. When USU's are unusually large (100 or more units) and would thereby require greater subsampling, they are placed in the rare events universe for the rotation group in which they appear. They remain members of that rotation group for eight CPS samples, greatly reducing the subsampling rate. 3. The adjustment for total noninterview takes place next. For noninterview adjustment purposes. the CPS PSU's are divided into 72 clusters. These clusters were formed at the time of the latest redesign by grouping together PSU's with similar characteristics defined by the 1970 census. The clusters are classified by geographic region, and within a region, they are divided into clusters totally comprised of PSU's in SMSA's (Standard Metropolitan Statistical Areas) and those containing only non-SMSA's. The clusters. are further partitioned into race-residence cells for SMSA's and non- SMSA's. The cells for the SMSA clusters differ from those for the non SMSA clusters. The cells are shown in Figure 7. The household population is divided into the clusters shown at the top of the figure; the nonhousehold population (roomers in boarding houses, staff personnel in institutions, etc. ) are divided into the clusters shown at the bottom of the figure. Within each of the cells, noninterview adjustment factors are applied to the weights for the interviewed households. The adjustment is done separately for each of the eight rotation groups. The noninterview adjustment factor is defined as the inverse of the response rate, using unweighted counts, for the particular cell. The factor is the ratio of the total number of eligible sample households from which a response was required to the number of households actually interviewed. These factors are applied to each of the interviewed households in the cell except in cells where the ratio equals or exceeds 2.0. In this case, the race groups within a residence category are collapsed and a common factor is computed for both race groups. In Figure 7, classes with the same numerical subscript are collapsed. If the newly computed factor exceeds 3.0, a noninterview adjustment factor of 3.0 is used. The Procedure as described requires 3,456 separate noninterview adjustment cells for the household population (72 clusters x 6 race residence cells x 8 rotation groups) and either 48 noninterview adjustment cells for the non- household population if the cluster was in a 53 Figure 7. CPS Noninterview Adjustment Cells HOUSING UNIT POPULATION SMSA Balance of SMSA Central Race City Urban Rural White....................... a.1 a.2 a.3 Blacks and Others........... b.1 b.2 b.3 Non-SMSA Rural Race Urban Non-Farm Farm White....................... c.1 c.2 c.3 Blacks and Others........... d.1 d.2 d.3 NON-HOUSING UNIT POPULATION SMSA Non-SMSA Rural Central ___________________ Race City Balance Urban Non-Farm Farm White.............. e.1 e.2 g.3 g.4 g.5 Blacks and others........ f.1 f.2 h.3 h.4 h.5 non-SMSA or 32 if the cluster was in an SMSA. 4. The next step in the weighting process is the first stage ratio adjustment, a post-stratification technique. The purpose of the adjustment is to reduce the contribution to the variance arising, from the sampling of PSU'S. The firststage ratios are based on 1970 census data and the factors are applied only to the weights of records in nonself-representing PSU's. Separate ratios are applied for two groups of strata (SMSA and non-SMSA) for each of the four census regions. The adjustment factor is computed as the ratio of the 1970 census population in the race-residence cell for the given cluster to the estimate of this population based on the 1970 census population for sample PSU's in the same cluster. If a large ratio results, a maximum factor of 1.3 is used. The. factors computed are applied to each of the records in the given cluster. The first stage ratio adjustment factors which were used from August 1974 to March 1975 are shown in Table 36. 5. The final step in the weighting process is the second stage ratio adjustment which is an attempt to bring the sample estimates into closer agreement with independent estimates of the U.S. population by age, sex, and race. Table 36. CPA First-Stage Ratio Adjustment November 1974 Through March 1975 Region Ratio adjustment North cell Northeast Central South West ------------------------------------------------------------------ SMSA PSUs Central cities White ..... .92119 1.16007 1.03576 1.15735 Black and Others... 1.30000 1.00275 1.07429 1.09549 Balance urban White..... 1.30000 .84031 .93984 .91770 Black and others.... 1.30000 1.17715 1.04199 .77366 Balance rural White..... .93792 .91891 .93075 .93841 Black and others .80882 .79961 1.15307 .94946 Non-SMSA PSUs Urban White..... 1.00040 .97189 1.03693 1.01325 Black and others.... 1.06873 .91619 1.02939 1.20315 Rural nonfarm White...... .99076 .99492 .99624 .96469 Black and others.... 1.11226 1.23429 .95165 1.30000 Rural farm White...... .94886 1.02057 .89994 .97191 Black and others.... .79456 1.08839 1.00435 1.30000 1 Hanson, 1976. 2 First-stage factors require change with replacement of PSUs. (For an explanation of replacement of PSU's the reader should refer to Section II.B). These independent estimates are prepared each month by carrying forward the most recent census data and taking into consideration births, age cohorts, mortality and migration. Starting in January 1974. the methods used in preparing independent estimates of the civilian noninstitutional population were modified. The new procedure is called the "inflation- deflation" method. In this procedure. the 1970 Census population figure for a given race-sexage group is "inflated" to include the estimated net census undercount for the group, and the resulting adjusted census. figure is carried forward to each subsequent month and later age. Also, the current population estimates are "deflated" to reflect the estimated rates of net undercount in the most recent census for the ages at the estimated date by race, sex, and age. In this way the percent of net undercount in the census for any given single year of age would be preserved throughout the decade. 54 This procedural change bad a minor effect on overall levels for January 1974. It had its greatest effect on those age groups with relatively different undercounts in adjacent age groups, especially males 20-24 years of age and, particularly, males of black and other races 20-24 years of age. Tables 37, 38, and 39 which follow show the differences in the population estimates from the old method and the new method. The second stage ratio adjustment is done in two steps. First, a separate ratio adjustment is made for persons of races other than white. Blacks and "other races" are adjusted sepa- rately. The basic reason for this procedure is to insure that the effect of the second 55 ratio adjustment for "other races" is not weakened by the adjustment for blacks. Occasionally, the Population estimated from CPS, which is the denominator of the ratio, is zero or close to it. When this happens, cells are combined and a common factor is computed. The collapsing is as follows: Other minority races First combination, Black, by sex by sex Final combination 14-19 ....... 20-24 ....... 14-34 25-59 ....... 60-64 ....... 35-54 Males 14 and over 70 and over.. 55 and over Females 15 and over A second step in the second stage ratio adjustment is now made to the weights of records. Separate factors are computed by sex, by race, and by 17 age-groups, giving a total of 68 cells by rotation group. The age-sex-race groups are given in Table 40 as well as the second stage adjustment factors computed over all rotation groups in March 1975. These factors were not the actual factors used, since those factors are computed by rotation group, but these factors illustrate the coverage problem in certain age-sex-race groups. Table 40. CPS Second Stage Ratio Adjustment Factors for Total Population By Age. Race and Sex1 ALL ROTATION GROUPS, MARCH 1975 White Black and Other Races2 Age Male Female Male Female Total 1.04901 1.02342 1.15468 1.07532 14-15 1.01927 .99287 1.02939 1.01576 16-17 1.05079 .97006 .98710 1.10733 18-19 1.08621 1.02334 1.18278 1.14973 20-21 1.04730 1.02451 1.53855 1.18612 22-24 1.12071 1.13036 1.17701 1.07879 25-29 1.07204 1.02624 1.24781 1.06004 30-34 1.03480 1.00931 1.26153 1.13815 35-39 1.07660 1.03174 1.07273 1.13769 40-44 1.05811 1.02347 1.28741 1.10292 45-49 1.04750 1.00498 1.24003 1.03712 50-54 1.01799 1.01848 1.10833 1.07060 55-59 1.08018 1.02333 1.07344 1.02268 60-61 1.02980 .95590 .99989 1.02240 62-64 1.03221 1.071.19 1.14459 .92967 65-69 1.02640 1.03032 .99706 1.05466 70-74 .96956 .98466 1.00452 .85741 75+ .98257 1.04738 1.29061 1.10894 1 Bailar, Bailey, and Corby. 1977. 2 The factors for Black and Other Races indicate the seriousness of the undercoverage problem. These factors are not the actual adjustment factors used. It has been shown that the ratio estimates based on age-sex-race groups reduce the sampling variability of the estimates. The result of the weighting procedure is that records have weights that vary considerably, but it is hoped that this differential weighting will reduce both biases and variances. The maximum, minimum, and average weights for 13 relationship categories for March 1975 are shown in Table 41. An analysis of the weights that are outside a 2.5. range around the mean revealed that about 2/3 of the low weights resulted from the application of the duplication control procedure (a result of the Cen-Sup sample), and about 1/3 were the result of the separate second stage ratio adjustment for blacks and other races. About 55 percent of the high weights were a result of the duplication control procedure and approximately 37 percent were the result of the separate second stage ratio adjustment for blacks and other races. Potential Sources of Error Caused by the Weighting Procedure Very little is known about the effects of the weighting procedure on the estimates. Some implications of the weighting procedure follow: 1. Though the intent of the weighting procedure is to reduce the bias caused by nonresponse and undercoverage and to reduce the variance, the procedure itself is not unbiased. There is no known unbiased method of adjustment for nonresponse and undercoverage. The basic Table 41. Maximum, Minimum and Average Weights for Records in 13 Relationship Categories1 March 1975 Weights Relationship category Maximum Minimum Average Male head with relatives ... 7488.80 33.56 1645.03 Male head without relatives... 8006.21 206.62 1679.78 Wife of head ................. 7215.72 31.46 1604.91 Female head with relatives ... 8549.27 33.04 1621.36 Female head without relatives. 8288.90 144.76 1612.22 Male child related to head ... 6666.70 27.12 1617.02 Female child related to head.. 6597.57 30.52 1551.92 Male relative (over 18) ...... 7060.87 67.29 1695.01 Female relative (over 18) .... 6296.77 39.30 1625.48 Unrelated male child ......... 6365.42 206.62 1736.01 Unrelated female child ....... 4496.97 1153.54 1628.61 Unrelated male (over 18) ..... 3840.59 756.86 1695.63 Unrelated female (over 18) ... 4369.49 991.51 1638.46 1 Bailar, Bailey, and Corby, 1977. 57 assumption is that the characteristics of the nonrespondents and missed persons are similar to those of respondents with similar demo-graphic characteristics. In 1965, Palmer and Jones (1966) attempted an investigation of the nonrespondents, except for refusals, by means of an intensive field follow-up during the 3 weeks immediately following the CPS survey week in September. The results indicated that the noninterview adjustment procedures did not distort the estimated number of employed persons. However, the results of that study are inconclusive because (1) less than half of the nonrespondents were interviewed, and (2) no refusals were included in the study. It is not unreasonable to assume that the refusals have a different employment rate from the interviewed. The problem, if there is one, is more serious now because the nature of the noninterviews has been continually changing in the last few years. In 1970, the overall Type A rate was 4.0 percent. Of these, 28 percent were not-at-homes, 23 percent were temporarily absent, 39 percent were refusals and 9 percent were "other". In 1976, the overall Type A rate was 4.4 percent. Of these 19 percent were not-at-homes, 17 percent were temporarily absent, 59 percent were refusals, and 5 percent were "other". The percentage of refusals is growing, and we do not know the effect of the noninterview adjustment for those who refuse though we can place at least approximate bounds on the maximum possible effect. 2. The collapsing rule for combining racial groups when the noninterview factor equals or exceeds 2.0 is arbitrary. There is no clear evidence that indicates why racial groups should be combined rather than geographic groups within SMSA/non-SMSA groups. There is no evidence that a factor of 2.0 is necessarily the optimum point at which to collapse cells. 3. Extended use of the 72 noninterview clusters over a decade may not be effective. Also, the use of the same first-stage factors for nonself-representing PSU's that are in sample for a decade is less efficient at the end of the decade than at the beginning. However, there is evidence that the variance is not increased by the extended use of these factors. 4. There is evidence that the noninterview adjustment procedure does not work according to specifications in some very specific situations concerning mixed race households. Figure 8 illustrates the discrepancies (Corby and Bailey, 1978). Case I is probably the most usual kind of case. The noninterview factor for whites is 1.1 and that for blacks and other races is 1.2. No collapsing would be done. However, the race of the household is determined by the so-called "principal person", usually the wife of head if there is one, or the head. The factors are applied to the records of persons, depending on their race. Suppose there was a mixed race household in which the wife of the head was white and the remaining persons in the household were not white. The record of the white person would get an adjustment factor of 1.1; the records of the other persons in the household would get an adjustment factor of 1.2 even though they did not contribute to that adjustment factor. In Case II an adjustment factor of 1.1 is computed for the whites, but an adjustment factor of 2.0 is computed for blacks. and others. The two cells are collapsed and a factor of 1.25 is applied to the records of all persons in both cells. In mixed- race households, everyone would get the same weight. In Case III, the factor for whites is 2.0, .and there are no interviewed households in the black and other races group. Each of these is a reason for collapsing. In this case, the factor is 2.0 for each group. Thus, as in Case 11, in mixed race households, everyone would get the same weight. In Case IV, the factor for whites is 1.1, and there are no interviewed households in the black and other races group. This is a signal to collapse. However, the collapsing is not carried through. A factor of 1.1 is applied to whites, and a factor of 1.0 is applied to blacks and Figure 8. Noninterview Adjustment in Mixed-Race Households Color Case I Case II Case III Case IV Case V White... int.hh 10 int.hh 10 int.hh 2 int.hh 10 int.hh10 ni.hh 1 ni.hh 1 ni.hh 2 ni.hh 1 ni.hh 1 Black and int.hh 5 int.hh 2 int.hh 0 int.hh 0 int.hh 0 other races ni.hh I ni.hh 2 ni.hh 0 ni.hh 0 ni.hh 2 int.hh means number of interviewed households. ni.hh means number of noninterviewed households. 58 other races. Thus, in a mixed-race household the white persons would get a factor of 1.1 and all others would get a factor of 1.0. Finally. in Case V, the factor for whites is 1.1. and there are no interviewed house-holds in the black and other races group. The two groups are collapsed, and a factor of 1.3 is derived. However, this factor is applied only to the white group, and a factor of 1.0 is applied to the black and other races group. Though the two noninterviewed households contributed to the derivation of the factor, they did not share in the application of the weight. There are two problems with mixed race households. One is that the programming is not consistent. For example, in Case V, the factor of 1.3 should have been applied to both groups. Second, the derivation of factors for mixed-race households is based on the race of the "principal person" but is applied on the basis of race of the household individuals. Table 42 shows the frequency of occurrence of each case. In the March 1975 CPS, there we're 309 cases of mixed race households. Of these 309, 238 had two different factors applied within the households. Of the other 71 households in which each person received the same weight, 67 of them had no noninterviewed households within the respective cells. The noninterview situation described by Case 11 in which all household members will always receive the same weight is applicable to the other four households. The problem affects roughly 0.5 percent of the entire CPS population, so it has little effect on the total U.S. employment statistic. However, it could be more serious for area estimates, especially in areas where there are concentrations of racially mixed households. Table 42. Mixed-Race Households in March 1975 CPS Sample1 .2 Total Households Principal Person in Which Two Noninterview Blacks and Adjustment Cases White Other Races Factors Used I.......... 146 125 218 II......... 2 2 --- III........ --- --- --- IV......... 30 0 16 V.......... 4 0 4 Total... 182 127 238 1 Corby and Bailey, 1978. 2 Counted by duplication control--does not include group quarters. 5. There is no evidence that the separate ratio adjustment for blacks and others which i immediately followed by the ratio adjustment for all age, sex and race groups has a positive, - effect on the estimates. Indeed, the factor for the "other races" is highly variable. Table 43 shows the factors applied to "other races" by age and sex and by rotation group for March 1975. 6. The independent estimates which the Bureau provides every month and to which the sample data are adjusted during the second-stage ratio adjustment are prepared by carrying forward the most recent census data thus any error's in the census count are transmitted to the official employment figures. The effect of the census undercount on the employment statistic was discussed in Section II.A. In summary, good measures are not available on the impact of the weighting procedure on errors occurring in earlier stages of the survey. Though it is generally assumed that the weighting, procedures reduce the bias of the employment. statistic, little is really known about the reduction of bias accomplished by the procedures. There is more knowledge of the effect of weighting on the variance. B. Specification of Estimation Procedure Several estimates are made for the employment statistic. One is made directly after the adjustment for noninterview and is called the "simple unbiased estimate."' (Since the estimate is not unbiased, this choice, of a name is misleading.) An estimate is made after the first- stage ratio adjustment, and another is made after the second-stage ratio adjustment. Another is made that shows the effects of both ratio adjustments. Finally, a composite estimate is computed which takes into account the rotating nature of the sample. Of these estimates, only the composite estimate is published. Composite Estimation 1. Description With the CPS rotation system, it is possible to use composite estimation which can reduce the variance of, month-to-month change and level. 59 In general, the composite estimation procedure takes advantage of the fact that with the 75 percent overlap of segments in sample each month between two adjacent months, there is, for many items, a fairly high correlation between estimates for the current month and those of a previous month. Specifically, the composite estimate of persons employed is derived as the weighted average of two estimates for the current month. a. The first estimate X.1 is the ratio estimate for the current month. b. The second estimate X.2 consists of the composite estimate for the preceding month to which has been added an estimate of the change from the preceding month to the present month based on that part of the sample common to the two months. Each estimate is given a weight of one-half such that the composite estimate X=«(X.1) + «(X.2). The more specific notation is given below: 4 X.t=(1-k)X".t+j[X.t-1 + - (X.(t,t-1) - X.(t-1,t))] 3 where X.t = the composite estimate for the current month. X".t = the ratio estimate after the first and second stage adjustments for the current month. X.t-1 = the composite estimate for the preceding month. X.(t,t-1) = ratio estimate for month t based on the six rotation groups that are common to months t and t-1. X.(t-1,t) = ratio estimate for month i-i based on the six rotation groups that are common to t and t-1, and k, =« Table 44 shows the 1975 averages of the unbiased, first stage, second stage, combined first and second stage and composite estimates for selected employment estimates. Table 45, its counterpart, gives the ratios of each of these stages of estimation to the composite estimate for the same items. The composite estimate for most of the employment items was. lower than the first and second stage ratio estimates. The order observed most frequently from largest to smallest was second-stage ratio estimate, first and second stage estimate, composite estimate, unbiased estimate and first stage ratio estimate. 2. Potential Sources of Error in the Composite Estimation Procedure The rotation group bias, as it is called in the CPS, could also be called time-in-sample bias. The distinguishing characteristic of this bias is that the number of times a person is asked to respond in a certain survey has an effect on the pattern of response. In a survey such as the CPS where some of the respondents each month are in sample for the first time, some are in sample for the second time, and, finally, some are in sample for the eighth time, this kind of bias raises serious questions about the reliability of the data. It also limits the ability to do any kind of longitudinal analysis. Since the level of the characteristic of interest is quite different for different rotation groups, one wants to know which of these rotation groups produces an estimate which is more likely to be consistent over time. Some of the underlying phenomena that may contribute to rotation group bias are differing undercoverage by time-in-sample, differences in how interviewers administer the questionnaire by time-in-sample, and differences in how respondents answer by time-in-sample. The employment status items are affected by this bias, just as are many other statistics. Table 46 shows "rotation group indices" for employment status by age and sex. An index is computed by dividing the total for a rotation group by the average over all eight rotation groups and multiplying by 100. Thus, an index of 103.8 means that a rotation group had a total that was 3.8 percent above the average. Table 46 shows that people are more likely to be classified as employed in the first month than in later months. This is especially true of some age groups, such as males, 16 to 19 years of age, for which the index was 104.2 for the first month of interview. This means that 4.2 percent more males than the average over all rotation groups are; likely to be classified as employed. Since the standard error of the index is only 1.1, the index of 104.2 is not within sampling variability of 100.0. No data on the rotation group bias for employment items are available by race. The rotation group bias affects the estimators of employment in different ways, depending on whether the ratio estimator or the composite estimator is used. In a paper by Bailar (1975), it is stated that the expected value of the composite estimate for a sufficiently large sample size can be written in the following form: 8 4 E(y.t) = Y.t + äa, + - [(a.1 + a.s) - (a.1 + a.5)] 3 where a.1 = the bias associated with the rotation group in its i- th month in sample. (Y.t) = composite estimate for month t and Y.t = population parameter to be estimated. The composite estimator is unbiased only if 8 4 äa.1 + - [a.4 + a.8) - (a.1 + a.5)] = 0. 1=1 3 The expected value of the ratio estimator can be written in the form 8 E(y".t) = Y.t + äa.i 1=1 This ratio estimator will be unbiased only if 8 äa.i = 0. However, even if the ratio estimator 1=1 is unbiased the composite estimator will not be unbiased unless the sum of the biases from the households in sample for the fourth and eighth time are exactly equal to and in the opposite direction of the biases from households in sample for the first and fifth times. This condition does not exist in the CPS. 62 In comparing the ratio estimate with the composite estimate, we can take the 'expected value of the difference 4 ä[y"t - (Y.t)] = - -[(a.4 + a.8) - (a.1 + a.5)] 3 Bailar presented some empirical data comparing the mean square error of the composite estimate with the mean square error of the ratio estimate for the period January 1970 through July 1972. Table 47 shows similar comparisons for 1975 estimates for selected employment items. The data shown in columns (1) and (2) are the ratio estimates and composite estimates, respectively, averaged over the 12-month period. In columns (3) and (4) are the estimated variances of the ratio and composite estimates. These are also averages over the same time period. In column (5) is an estimate of the bias of the composite estimate under the assumption that the ratio estimate is unbiased. In the paper by Bailar, the bias of the ratio estimate was estimated from the reinterview data. Since the reinterview survey does not give an accurate estimate of the bias, we shall be more conservative than Bailar and assume the bias is zero. Column (6) shows the ratio of the estimated variance of the composite estimate to the estimated variance of the ratio estimate for estimates of level. For all items, the ratios were less than 1.00 indicating a gain in precision by use of the composite estimate. Column (7) shows the ratio of the mean square error of the composite estimate to the variance of the ratio estimate. Though a few of these ratios are larger than 1.00, indicating that the composite estimate is poorer for these items, the majority of the items have ratios somewhat less than 1.00. However, for employment items for which the rotation group bias is largest, the ratio is always over 1.00. When estimates of month-to-month change are made, the rotation group bias will not affect the estimates if the bias is constant from month to month. Some research underway indicates that this is probably not the case. At any rate, the estimated variance of month- to-month change is smaller for, the composite estimate than the ratio estimate for many, but not all, employment, items. The mean square errors presented here air not complete since they do not include the correlated component of response variance, bias from nonresponse adjustment procedures, etc. These might affect the estimates in different ways and possibly reverse the relationships shown here. However, based on the available data, employed items generally show a somewhat lower mean square error for the composite estimate than the ratio estimate. Some notable exceptions to this are total persons employed in nonagriculture, females employed in nonagri- culture, and teenagers employed in nonagriculture; these ratios are 1.05, 1.15 and 1.47, respectively. Seasonal Adjustment 1. Description The CPS employment statistic reflects a regular recurring seasonal movement which can be estimated on the basis of past experience. Thus, in order to make meaningful comparisons between monthly estimates of employment, this seasonal movement is removed. In the analysis of economic time series, four seasonal fluctuations are generally measured trend, cycle, seasonal variations and the irregular fluctuations. The long-term trend corresponds to a variation persisting over a period of time, that is long in relation to the cycle. In some cases, it is a steady growth, while in others the trend may move down-ward as well as upward. The cycle, usually referred to as the business cycle, is a quasi- periodic oscillation characterized by alternating periods of expansion and contraction. The seasonal variations represent the composite effect of climatic and institutional events which repeat more or less regularly each year. These three types of fluctuation are assumed to follow systematic patterns. However, the irregulars are unforeseeable movements related to events of all kinds. In general, they have a stable random appearance but, in some series, extreme values may be present. These extreme values or outliers have identifiable causes, e.g. floods, unseasonable weather, strikes; a nd, therefore can be distinguished from the much smaller irregular variations. Thus, the seasonal variations are distinguished from trend by their 65 oscillating character, from the cycle, by being confined within the limits of an annual period, and from the irregulars, by their systematic nature. For labor force series, the causes of seasonality are found in the main factors that, influence the demand for and supply of labor. The seasonal variations in employment and unemployment of adult males are strongly conditioned by the stages of the business cycle and the weather. The main reason for this is that while there is practically no seasonality in the labor supply of adult males, there is a large amount of seasonal variation in the demand side. This group is mainly absorbed by the primary sector (construction, agriculture, mining, fishing and forestry), where seasonality is mostly climatic with very large oscillation, and by the industrial sector, where seasonality is mostly induced by the seasonality in the primary sector but is also strongly affected by the stages of the cycle. On the other hand, seasonality for females and young males stems from demand as well as the supply of labor. Females and young males are mainly employed by the tertiary sector (services, financial, banking), where seasonality tends to be originated by institutional events (Christmas, Easter, Federal taxes, deadlines). Similarly, from the viewpoint of the supply, seasonal variations occur because this group tends to move in and out of the labor force in accordance with the school year. Because seasonality ultimately results from noneconomic forces (climatic and institutional factors), external to the economic system, its impact on the economy as a whole cannot be modified in a short period of time. Therefore, it is to the interest of decision makers to have the seasonal variations removed from the original series to obtain a seasonally adjusted series. In this manner, the variations of a seasonally adjusted series are due to variations only in the trend, the cycle, and the irregulars. Decisionmaking based on the raw data can lead to wrong policies, especially if the series is strongly affected by seasonal variations. The main reason for this is the fact that, on the average, the abso- lute month percentage change in the seasonal variation can be much greater than the corresponding changes in the irregular or trend- cycle. Results of several studies of selected economic 'indicators for the United States show that the average absolute month-to-month percentage changes in the.seasonal component run between three and seven times the average absolute percentage changes in the trend cycle or in the irregulars over the same time spans (Shiskin. 1973). The U.S. Bureau of the Census Method II X-11 Variant is currently used to seasonally adjust all the labor force series obtained from the Current Population Survey. This method is described fully in Shiskin, Young and Musgrave (1967) and the properties and limitations of its basic assumptions are discussed in Dagum (1976). Therefore, only a brief summary is presented here. The X-11 variant assumes that the main components of a time series follow a multiplicative or an additive model, that is, (1) 0.t = C.tS.tI.t (multiplicative model) (2) 0.t = C.t + S.t + I.t (additive model) where 0, stands for the original series; C.t, the trend-cycle; S.t, the seasonal; and I.t, the irregular. There are no mixed models in this program, such as 0.t = C.tS.t + 1.t or other possible combinations. This Census procedure, an adaptation of the standard ratio-to- moving-average method of seasonal adjustment, includes these steps: (a) calculating a 12-term centered moving average of the original data arranged in chronological order to estimate the trend-cycle; (b) dividing the trendcycle estimates into the original data to obtain a series of seasonal-irregular" ratios; and (c) for each month calculating a moving average of the .1 seasonal-irregular" ratios to estimate the seasonal factors for that month. In a second iteration, variable Henderson moving averages are used to estimate the trend cycle. To derive the national unemployment rate, each of three major labor force components (agricultural employment. nonagricultural employment, and unemployment) for four age-sex groups (male and female workers, 16-19 years and 20 years and over) are separately adjusted for seasonal variation and then added to give seasonally adjusted total employment and civilian labor force data. 2. Source of error in seasonal adjustment The X-11 program is designed to be flexible enough to be adequately applied to varying types of time series. In the application of the method, however, sources of potential error in the seasonally adjusted result are generated. These 66 sources of error include four which can be generally classified as:. -errors arising from misspecification of the additive/multiplicative model of the series; -errors arising from use of asymmetric filters in the computation of the trend-cycle moving average; -errors arising from the use of asymmetric filters in the computation of the final seasonal factor moving average curve; and errors arising in forecasting seasonal factors to be applied to current period data. These sources of error will be considered in turn: a. Model selection The X-11 program includes options for both multiplicative and additive models of the behavior of the seasonal component (S.t). In an additive model, the components of the series are assumed to be independent, and therefore, the seasonal effect is not affected by the level of economic activity conditioned by the stages of the cycle. On the other band, in a multiplicative model, the seasonal effect is proportional to the trend-cycle. If the true model is that of a constant multiplicative seasonality, an additive adjustment will produce seasonals that appear to vary with the trend-cycle. Reciprocally, if a constant additive seasonality is the norm, a multiplicative adjustment will produce factors that look unstable or moving. Despite the existence of a number of tests to determine the model behavior of the series, it is possible to misspecify the model as the behavior of the series may evolve from one model to another over time, and certain months may exhibit behavior which departs from the overall model which best fits the whole time series. When the model is misspecified, error is created which will lead to a misspecification of the contribution of the component on a current basis and to significant revisions as additional data are accumulated. The change of stable seasonality to a moving seasonality due to the wrong selection of the model would be of no consequence for the final output of the program if the program were able to handle moving seasonality with precision. However, the X-11 uses asymmetric filters for the estimates of the first and last 3 years which introduce systematic errors when the seasonals are moving. b. Trend-cycle moving average The estimation of the trend-cycle component of the aeries in the X-11 is based on a sophisticated, iterative procedure. A first approximation is obtained by use of a centered 12-term moving average; for the final estimate of the trend-cycle curve, Henderson moving averages are used. To obtain the, six missing values at either end of the centered 12-term moving average, the program assumes that the trendcycle is constant and thus repeats the first (last) available moving, average six times. The use of the Henderson filters enhances identification of the trend-cycle within the span of the filter (a 9-, 13- or 23-term Henderson moving average), but for the missing values at either end of these averages (4, 6, or 11 months, respectively) the X-11 program uses asymmetric filters that are capable of reproducing only a straight line. This is a serious limitation for turning points, causing error in the seasonal result when the current period is associated with a turning point. c. Final seasonal factor moving average curve A similar problem arises from utilization of a moving average applied to the final seasonal irregular ratios (differences) to obtain the final estimate of the seasonal component. In order to smooth out the seasonal-irregular ratios for each month, the standard option of the program requires at least 7 yearly observations.to use symmetric moving averages. The seasonal factor corresponding to the middle year (central) will be the only "final" seasonal factor in the sense that it will not change significantly when further observations are added to the series. The three moving seasonal factors at either end of this moving average are estimated using asymmetric weights, and the degree of error increases' in any departure from the central point. d. Forecasting seasonal factors for the current period The X-11 program provides a standard forecast of the seasonal factors to be used in the next 12-month period. These factors are computed by the formula: S.t+= S.t + 1/2 (S.t - S.t-1) Use of these factors, or alternatively, the decision not to use these factors in favor of the use of factors generated for the immediate past 67 year (S.t+.1 = S.t) as is the present practice with labor force data, creates another potential source of error. In both instances, an implicit assumption regarding the seasonality of the. coming period's original estimates is made. 3. Estimating the Error in Seasonal Adjustment The main interest in seasonal adjustment of labor force items is to obtain seasonally adjusted data on a current basis. These current estimates are approximations based on past experience. These estimates have a broader margin of possible error than the original data on which they are based, since they are subject not only to sampling and other errors in the estimation process, but, in addition- , are affected by the uncertainties of the seasonal adjustment process itself. In making judgements concerning the reliability of current seasonally-adjusted estimates, however, labor force analysts are constrained to consider only the sampling error of the original estimates, since good estimates of error arising from the seasonal adjustment process are not available. Though it is not possible to estimate the error arising from seasonal adjustment on a current basis with precision, it has been suggested that error can be detected on a retrospective basis. The historically, adjusted data for the period in which the weights of the seasonal adjustment program exhibit symmetrical characteristics may be considered the "best" estimates available. Seasonal factors applied to current data, whether "year-ahead" or "last year" factors, are computed in the asymmetrical portion of the moving average procedures. In the historical (central) period, seasonal irregulars are available on both sides of the year for which this seasonal factor is being determined, and the weights arc practically,symmetrical. Since the moving average is across more terms, the historical seasonally adjusted data have a smaller variance. In addition, the current month is seasonally adjusted by a factor based on historical experience, while the current "true" seasonality belongs to a period of time subsequent to that for which the factor was developed. Consequently, current basis adjustment will naturally have an irregular component uncorrelated with the seasonal factor applied to it (Kaitz, 1962). Though the X-11 program will accept a series that exhibits little irregularity with as few as three years of monthly observations and produce a reasonable centrally weighted value, for most series, which exhibit some irregularity a minimum of 7 years of monthly observations is necessary. Table 48 shows the CPS employment estimates for the period, January. 1967-December 1977 as adjusted on a current basis (using factors generated through December 1972), then seasonally adjusted historically (using factors generated with observations for the period January 1967 through December 1977). The month, June 1974, was selected for analysis a month in which seasonality was computed on a current basis with asymmetrical weights, then after availability of three years of additional observations, with symmetrical weights. 'ne absolute differences between the current and historically adjusted data are shown. For total employed, the ratio of the difference between the current and historically adjusted data to the current data is 0.07 percent. The difference for nonagricultural employed, males 16-19 years, is 1.03 percent. In large part, the difference or "error" for this series is related to an improper specification of the model of behavior of the seasonal component. Had the series been adjusted with the additive rather than multiplicative option, as has been the procedure since January 1978, the difference would have been 0.11 percent. Finkner and Nisselson (1977) described the implications of the use of sampling errors estimated Table 48. Original, Current and Historically Seasonally Adjusted1 Data for June 1974 (000) Seasonally adjusted estimates On As Unadjusted current centrally Differ- estimate basis weighted ence Series (1) (2) (3) (4) Employed, total 87,166 86,165 86.102 - 63 Nonagriculture Employed Male, 16-19 4,242 3,679 3,717 + 38 Female 16-19 3,456 3,285 3,258 - 27 Male, 20 years and over 46,385 46,063 46,080 + 17 Female, 20 years and over 29,188 29,859 29,690 - 163 Agriculture Employed Male, 16-19 552 337 366 + 29 Female, 16-19 113 67 71 + 4 Male, 20 years and over 2,609 2,420 2,431 + 11 Female, 20 years and over 621 469 499 + 30 1 Series adjusted with multiplicative option of X-11 program to assure comparability with 1974 procedures 2 Based on data from the BLS Employment Earnings Series. 68 from the observed series as applied to the seasonally adjusted series. Drawing on the work of Kaitz (1974) and Monsour (1975), they showed that the relationship between the sampling errors of the origi- nal series and the adjusted series depends on the seasonal adjustment procedure used. In comparing the current adjusted data with historical central term adjusted data and historical end term adjusted data, it was shown that the current seasonally adjusted data had standard errors always greater than either version of historical seasonally adjusted data, both for monthly level and month-to-month change. The standard errors for the current adjusted series were also larger than the standard errors of the. original series. C. Specification of Estimation Procedures for Errors As part of the CPS estimation procedure, sampling errors are also estimated. In addition, there is a reinterview program in which each month a certain part of a sample of interviewers' work is reinter- viewed. From the reinterview, estimates of simple response variance and bias are made quarterly. Estimation of Errors 1. The CPS Variance Estimation Procedure The CPS variance estimation procedure makes use of a Taylor approximation. The variance is estimated under the assumption that the sampling variance is approximately equal to the sampling variance of the first degree terms of the Taylor approximation of the estimator. For a detailed description of the development of the variance estimation procedure, see Hanson 1978. The CPS variance program has a capacity of 100 items. Variances can be produced each month for 100 labor force items. The estimate of variance has several components: a. within PSU variance attributable to the selection of a sample of segments within each sample PSU. b. between PSU variance, attributable to the selection of one PSU from a stratum; only nonselfrepresenting PSU's are subject to this variability. c. between stratum variance attributable to the selection of one PSU from each pair of strata in the selection of the C design sample. Here again only NSR PSU's are subject to this variability. Table 49 shows the components of variance for selected employment items for 1975. The major part of the variance comes from the within PSU variance. For total persons employed in nonagriculture, an estimated 90.3 percent of the variance was within PSU variance. Table 50 shows a comparison of the variances for the composite estimates and that of the otber stages of estimation. For all items shown except "with a job, not at work", the composite estimate had the lowest variance. As calculated, the sampling error includes not only the standard error, but the impact of the random component of response and processing error and that part of the interviewer variance present in nonself-representing PSU's. 2. Generalizing Estimated Sampling Errors The sampling errors provided to the Bureau of Labor Statistics for users checking reliability of the data are generalized standard errors. Generalized standard errors are used instead of those com- puted from the variance run for each item because (a) it would be impractical to compute and/or publish sampling errors for every estimate and (b) the generalized standard error gives some stability to the estimates of error. The method of generalization is described below: The following equation is used in generalizing the errors: V.x2 = a + b/x where V.x2 is the relvariance of the estimate x, and a and b are two parameters, fitted by a least squares process to a set of observed estimates and their computed relvariances. To develop the a and b used in obtaining the generalized standard error tables, a set of statistics, concerned with labor force items and covering a wide numerical range is selected. Through an iterative process, the esti- mates and their corresponding relvariances are used to estimate a and b With the derived a and b, a generalized standard error table for estimates of level is developed. Standard errors are presented in Table 51 for estimates of monthly levels of employment and unemployment. 3. Simple Response Variance and Response Bias Simple response variance is the trial-to-trial variability in response that occurs in different trials of exactly the same survey procedure. To obtain an unbiased estimate of the simple response variance, at least two independent measurements of the characteristic for each sample person, using the identical measurement procedure for each trial is necessary. The reinterview survey provides estimates of simple response variance; however, as discussed in the next section, these estimates have major limitations. The simple response variance is included in the sampling variance estimates. The index of inconsistency is the ratio of the simple response variance to this total of the simple response and sampling variances. It is based on the results for a sample of size 1. Thus, if the index is high, it is an indication that the concept is "fuzzy" and increasing the sample size would not help. In the estimation of simple response variance and the index of inconsistency, there is an effort to make the reinterview as much like the original interview as possible. For that reason, the only data that are used are those that have come from the 20 percent of the reinterview sample in which the original and reinterview responses were not reconciled. The net difference between the total cases for the class obtained in the original survey and the reinterview is used to provide an estimate of bias using the 80 percent reconciled results from the reinterview survey as a standard for measuring the original survey results. Table 52 gives the net difference rates and indices of inconsistency by quarter for the years 70 Click HERE for graphic. 71 Click HERE for graphic. Click HERE for graphic. 1974, 1975 and 1st quarter 1976 for three employment characteristics. Potential Problems in the Estimation of Errors As discussed above, the CPS variance program uses the Taylor approximation. In 1971, Frankel included in his research on inference from sample surveys an empirical investigation into the accuracy of the Taylor approximation in estimating the sampling error of complex estimates. As designs he used three clustered-stratified sample designs of 6, 12, and 30 strata based on 45,121 noninstitutional fami- lies and primary individuals collected by the Bureau of the Census in the March 1967 Current Population Survey. Table 53 gives these results for the Taylor approximation. The total mean square error decreases with the increased number of strata. With 270 strata the total MSE of the Taylor approximation was 2.9 percent and for 810 strata it was 1.0 percent for estimates of the mean. At present, variances are not, routinely calculated for seasonally adjusted estimates, but rather the Table 53. REL-MSE of Estimates of Variance Based on Taylor Approximations1 (Means) Taylor Squared Var. of Total Sample Design bias Variance MSE 6 Strata........... .003 .512 .515 12 Strata.......... .009 .254 .263 30 Strata.......... .004 .146 .150 90 Strata.......... .007 .061 .068 270 Strata......... .013 .026 .039 810 Strata......... .006 .004 .010 1Frankel, 1971. variance estimates calculated for the original data are used for the seasonally adjusted data. Kaitz did some work comparing the sampling error for the original and adjusted data in 1974. There is a serial correlation pattern in the standard error of CPS data resulting from the overlap structure in the,sample. Table 54 presents the sample overlap and estimates of serial correlation in the sampling error for employment items. The effect of the serial correlation structure on sampling error of the seasonally adjusted data is given in Table 55. It shows speculated sampling errors for original and seasonally adjusted data for employed persons, level and month-to-month change. The sampling error of the historical data is lower than that of the original unadjusted data. For persons employed in nonagriculture, the sampling error was approximately 14 percent lower for the histori- cal data than the original; however, for current Table 54. Serial Correlation in Sampling Error1 Civilian Labor Force Gap Percent Total Employment Between Sample Nonagriculture Agriculture Months overlap Employment Employment 1 75.0 0.66 0.80 2 50.0 0.40 0.50 3 25.0 0.15 0.20 9 12.5 0.08 0.10 10 25.0 0.16 0.19 11 37.5 0.24 0.29 12 50.0 0.31 0.39 13 37.5 0.24 0.29 14 25.0 0.16 0.19 15 12.5 0.08 0.10 1 Kaitz, 1974. Table 55. Sampling Errors for CPS Seasonally Adjusted Series1 Seasonally Adjusted Data Original--------------------------- Series Data Current Historical Civilian labor force Total employment Nonagriculture Level......... 1.000 1.136 0.862 Monthly change 0.825 0.951 0.674 Agriculture Level......... 1.000 1.136 0.854 Monthly change 0.632 0.739 0.460 1 Kaitz, 1974. seasonally adjusted data, the standard error was higher than that for original data. For persons employed in nonagriculture and in agriculture, the standard error of the current seasonally adjusted data was 13.6 percent higher than that of the original For monthly change, the current seasonally adjusted data were 16.9 and 15.3 percent higher than the original data for persons employed in nonagriculture and agriculture, respectively. Thus the use of standard errors of original data could cause the acceptance of a numerical difference in employment between months as a statistically significant difference when it is not. Some problems with the sampling variances are listed below: 1. The selection of USU's within the sample PSU's is not treated as a systematic selection in the variance estimation procedure. Since this systematic selection of PSU's represents a further stratification of the sample by geography and size, the estimation of variance without regard to this approach could possibly result in an estimate of variance that is somewhat high. 2. As with many surveys, the between PSU variance in the CPS cannot be directly estimated, resulting in extremely variable and unstable estimates of this component. Table 56 shows the between PSU variances for selected employment items for the 12 months in 1975. The percent this variance component is of the total variances varies for persons employed in agriculture from a negative value to 20 percent of the variance; for persons employed in nonagriculture, the range is from 0.4 percent to 14 percent of the total variance. 3. The estimate of variance ignores the impact of the controlled selection of PSU's which tends to reduce the between PSU variance for most items. This then results in an overestimation of the variance. 4. The use of generalized standard error tables (even though the generalization tends to stabilize the error estimates) results in some error in the estimation of variances. 5. The correlated response variance component is reflected in the NSR strata only. The variance is thus underestimated for this component. Though there has not been an interviewer variance study conducted for the CPS, Section III.C gives some results of a study on the interviewer variance which suggest the need for a randomized experiment to measure this component. The estimate of simple response variance as estimated by the reinterview survey has serious limitations. 1. The. reinterview survey is not independent of the original survey since the respondents frequently remember the original survey. Therefore, the estimator is biased since it is assumed that the responses in the two interviews are uncorrelated. 2. The data collection methods used in the original interview and in the reinterview are seldom the same. For example, 1st and 5th month households cannot be inter-viewed by telephone in the original survey, and 2nd month households can only be interviewed by telephone on callback in the original interview. In the reinterview, all households are eligible for interview by telephone, and though we do not have exact figures, it is thought that almost all reinterview is conducted by telephone. 3. There is a high noninterview rate connected with the reinterview procedures. Only those households that can be interviewed in both the original and reinterview and that can be successfully matched are used in estimating the simple response variance. 4. The reinterview results are not subjected to the regular CPS estimation procedures. If the reinterview data were processed, weighted, edited, and the identical estimation procedure used in the original survey, the results would be applicable to the published CPS estimates. At present, they are not. 5. The assumption of simple random sampling used in the estimation of the simple response variance is invalid. A method of estimating the simple response variance that takes into account the structure of the sample is necessary. The current procedure yields underestimates for the same reasons that the sampling variance would be underestimated by assuming a simple random sample. 6. The index of inconsistency which is a ratio of simple response variance to the sampling variance plus the simple response variance should have a range of 0 to 1; with the procedure used, the range is 0 to 2. However, unless the responses are negatively correlated-something not expected in the CPS-the estimated index will remain,within the appropriate 0 to 1 range. The reinterview estimate of response bias is itself biased, but the extent of the bias is unknown., Limitations of the reinterview program have been discussed above and in Section III C. Since it oper- ates within the CPS design, it is subject to the same procedural biases. 74 Click HERE for graphic. 75 D. Quality Control of the Estimation Procedure Quality Control of the Weighting and Estimation Procedure There was a dependent check of the first stage factors used in the weighting procedure in DPD in Jeffersonville and further spot checking of these ,factors in Washington. These factors are used throughout the decade, except for changes that result because of the introduction of rotating PSU's. The monthly factors and output of the other stages of estimation are not subject to any formal quality control procedures. All the factors and results from all stages of estimation are. computed each month and "eyeballed" to see if they seem reasonable. No written procedures have been developed to guide one in this process. Quality Control of the Keyfitz Estimate and Variance Program The 100 items for which variances are' estimated for the monthly CPS were tested to determine the validity of the definitions of the items. To check these item definitions, the combined first and second stage estimates resulting from running the March 1973 data file through the program were compared to the composite estimates published in the Bureau of Labor Statistics Employment and Earnings, Vol. 19, No. 10, dated April 1973' If the estimate could not be located in this publication' then it was searched for in the complete set of CPS tabulations for March 1973. The two estimates were compared and reconciled. Those items which showed less than a 10 percent difference were accepted as correct. Definitions of items for which the difference was greater than 10 percent were reconciled. After reconciliation eight items still displayed differences greater than 10 percent; it was concluded that these estimates, which were all small, were more affected by the composite estimation than others (Smouse, 1976). The variance program was checked with test files before it was presumed to be working correctly. Test files were created and were used as input to the variance program. All the variances of the esti- mates in the test file were computed by hand and compared to the program output. The two outputs were in agreement and the variance program was presumed correct. There is no formal quality control of the monthly variance output. Control is confined to "eyeballing of various estimates and variances to see if they seem reasonable. 6 CHAPTER IV Analysis and Publication A. Description The analysis and statistical review of the monthly labor force estimates are performed at the Bureau of Labor Statistics (BLS), taking into account the measures of survey error provided by the Bureau of the Census. In analyzing survey results, the BLS considers the difference between two estimates to be 1, statistically significant" if it exceeds 1.6 times the standard error of the difference. Data are presented monthly in Employment and Earnings issued by the Bureau of Labor Statistics. A brief section at the beginning of the publication highlights the employment and unemployment developments for the current month over the previous month. For example, for January 1977, the summary read "Total employment as measured by the monthly survey of households-rose by 220,000 to a new high of 88.4 million. After holding about steady from July to October, total employment advances in November and December have amounted to nearly 600,000." A series of tables appear in Employment and Earnings that are based on "household data." These are known as the "A tables." There were such tables about-the January 1977 labor force data. The tables show employment status and number of employed for persons 16 years of age and over from 1947 to the present. The tables provide totals and percentages by race, age, and sex. There are no standard error tables in this set. At the end of the publication there is a section called "Explanatory Notes." In the part that deals with the CPS, there are sections on the following: Collection and coverage Concepts Historic comparability Estimating methods Rounding of estimates Reliability of estimates. In the last section, the approximate standard errors for level and for month-to-month change are given. At the end of the report there is a note that additional information is available from the Handbook of Methods, BLS Bulletin 1910. Though Employment and Earnings is the main publication for the employment figures, BLS publishes several other reports, including Monthly Labor Review. B. Potential Sources of Error in Analysis and Publication 1. At times the report comments on differences for which the appropriate check has not 'been made, particularly "differences" involving changes over time. Thus some statements discuss "differences" which are probably not statistically significant. 2. The "Historic Comparability" section in Employment and Earnings discusses certain major changes in the survey which result in nonsampling errors when making comparisons over time. Until early 1978 this was the only information on nonsampling errors presented. This is not a reflection on BLS, since the Bureau of the Census does not provide them with nonsampling errors. Since the biases probably dominate the mean square error, for characteristics based on large numbers of people, this was a serious omission. However, the sampling statement section on the reliability of the estimates has been revised to include a discussion on nonsampling errors. 3. In Moore (1973), it is pointed out that some troublesome situations develop in describing changes in employment data to the public in a way that accounts for sampling errors. The situations develop because of the terminology used to describe whether figures have changed or not. A figure that has not changed "significantly" is described as "substantially un- changed"; a change on the borderline of statistical significance is described as "edging up' or "edging down"; when the change is beyond the borderline of significance, it is characterized as a rise or fall. A significance test applied to month-to-month change answers only one question: whether this month's figure differs significantly from last month's figure in light of the sampling errors. It does not answer the question of whether the change (or absence of change) differs significantly from last month's change, which is the appropriate question concerning a turning point. 77 Moore also pointed out that during 1972, the total civilian employment grew by 2.2 million persons-a rate of increase rarely exceeded in any other 12-month period. In only 6 of the 12 months was the increase "statistically significant." It could have been possible for each of the month-to-month changes to have been "insignificant" in which case employment would have been described as "substantially un-changed" month after month, while, in fact. the year-to-year change was very large. 4. In the same article. Moore pointed out that sampling errors are only one of the considerations in judging the significance of changes in employment data. Errors in seasonal adjustment may be equally important. These errors are independent of and additional to the sampling error. The same could be said of mann other nonsampling errors. 78 CHAPTER VII Conclusion As pointed out in the preface of this report, there are four main purposes for compiling an error profile for the Current Population Survey. Let us examine these purposes now in relation to the material assembled in this report. A. To illustrate how an error profile is created in an effort to encourage government statisticians to provide error profiles for the major recurrent survey statistics. In putting together this report, we made an effort to follow the major survey steps in sequence. We would recommend this for the writing of an error profile for any major survey. B. To compile in a single document the sources of error and the information that is available about these sources of error and their impact. We tracked down any leads we had on information on potential sources of error and their impact. Some information was probably overlooked. To have such information available is valuable for at least three main reasons: 1. This, information can be used as the basis for a total survey design since information is available on nonsampling errors to be used with information available on sampling errors. 2. The areas that we know least about, the impact of nonsampling errors, are clearly highlighted. This can serve as the basis for a comprehensive methodological research program. 3. The information highlights problems in specific areas and improvements can be made immediately. Some examples from this report are: a. The inconsistency in the weighting of mixed race households can be easily resolved. b. The extreme variability in the weights and the causes of that variability are now being examined Procedures that are the basis for extreme variability can be evaluated. c. Documentation can be made of what is done about "poor" interviewers. Records are currently not available. 4. Informing sophisticated users of the design and potential errors in the statistics of interest. C. To illustrate the need for controlled experiments to measure nonsampling errors because of the lack of knowledge of the impact of these errors. There are areas in the survey process that seem to be in excellent control. For example, nonsampling errors caused by the sampling procedure are well-known and procedures are underway to reduce these errors. Also. the FOSDIC process and the microfilming are under tight controls. In contrast there are several areas which need study. Some of these are as follows: 1. The training program for interviewers can have a large impact on the kinds of errors interviewers make. Experimentation with different kinds of training packages is needed. 2. Studies of the differences between telephone and personal interviewing on employment statistics are needed. 3. Studies of the differences between proxy and self- respondents on employment statistics are needed. 4. Procedures that minimize the impact of rotation group bias should be studied. 5. Alternative weighting procedures should be studied. 6. An interviewer variability experiment should be incorporated into the CPS. D. To stimulate development of a mathematical model that will reflect the ways in which the errors from different sources interact. The Bureau of the Census had done pioneering,work in developing mathematical models that reflect different sources of nonsampling and sampling error, as well as their interaction. The models have been very helpful in encouraging the use of experiments to estimate the parameters of the models. For example, we now have ways to measure interviewer variability. We can also measure coder variability. However, we do not have a model that allows us to look at the error that comes from nonresponse as well as the error that may come from the adjustment for this nonresponse. These errors are not additive. It would be very useful if each step of the survey process could be identified as being a major or minor contributor to the total survey error. 79 Steps are being taken to get quantified measurements of these errors. A Methods Test Panel is designated outside of the regular CPS to measure differences in procedures. Yet a comprehensive research program which would have people engaged in modeling and others engaged in experimental design and measurement of errors would be the most useful way to make progress m the advancement of knowledge about the nonsampling errors. 80 References [1] Baer, L., "Permit Status of TAR ED's in the Redesign," Bureau of the Census memorandum to M. Boisen' January 5, 1973. [2] Bailar, B.A., "The Effect of Repeated Interviewing," Unpublished paper presented at the Conference on Econometrics and Mathe- matical Economics, Ann Arbor, Michigan, 1973. [3] Bailar, B.A., "The Effects of Rotation Group Bias on Estimates from Panel Surveys," Journal of the American Statistical Asso- ciation, 70( 1975), 23-30. [4] Bailar, B.A., Bailey, L., and Corby, C., "A Comparison of Some Adjustment and Weighting Procedures for Survey Data," Unpublished paper presented at 1977 Sampling Symposium, Chapel Hill, North Carolina. [5] Banks, M. and Shapiro, G., "Variances of the Current Population Survey, Including Within-and Between-PSU Components and the Effect of the Different Stages of Estimation," Proceedings of the Social Statistics Section of the American Statistical Association, 1971, pp. 40-49. [6] Boisen, M., "Determination of Type of Segment for Tape Address Register (TAR) ED's Which Are in Non-permit Areas," Bureau of the Census, memorandum to J. Waksberg, July 26, 1971. [7] Boisen, M., "The Replacement Schedule for the Rotating PSU's for CPS," Bureau of the Census Final Specifications Memorandum No. 24 (Second Revision), July 24, 1973. [8] Boisen, M., "Final Report on CPS/FOSDIC Experiment," Bureau of the Census memorandum to D. Levine, June 27, 1975.-[9] Corby, C. and Bailey, L.; "Some Results of a [9] Recent Investigation of CPS Adjustment Procedures," Bureau of the Census memorandum to B. Bailar, January 31, 1978. [10] Dagum, E.B., "Recent Developments in Seasonal Adjustment Methods and Application," Selected Papers from North American Conference on Labor Statistics, U.S. Department of Labor, 1976, 146-161. [11] Deighton, R.E., "Methods Test Report: Some Results of Experimentation with Self-Response Interviewing Procedures, February 1965-June 1966," Bureau of the Census memorandum, February 28, 1967. [12] Deighton, R.E. and Zinter, M.A., "Description of the Methods Test-Research in the Selection of a Best Respondent," Methods Test-Bureau of the Census Internal Working Paper No. 1, April 16, 1965. [13] Dinwiddie, J., "Results and Analysis of the 1975 Rural Listing Test," Bureau of the Census 1975 Rural Listing Test Results Memorandum No. 4, November 14, 1977. [14] Fasteau, H.H., "CPS Redesign-Evaluation of Encoding Subsystem," Bureau of the Census memorandum to M. Boisen, April 5, 1973. [15] Finkner, A.L. and Nisselson, H., "Some Statistical Problems Associated with Continuing Cross-Sectional Survey," Unpublished paper presented at the 1977 Sampling Symposium, Chapel Hill, North Carolina. [16] Frankel, M.R., Inference from Survey Samples. An Empirical Investigation, Ann Arbor, Michigan: The University of Michigan, 1971. [17] Goodman, R. and Kish, L., "Controlled Selection-A Technique in Probability Sampling," Journal of the American Statistical Association, 45 (1950), 350-372. [18] Greenough, M.L. and Epley, K., Analysis of Data on Opacity and Brightness of Paper Samples for the 1970 Census, Unpublished Report No. NBS446.02, August 14, 1969. [19] Hanson, R.H., Statistical Methods in the Current Population Survey, Draft of the Revision of Technical Paper No. 7, 1976. [20] Harris, M., "Interviewer-Research: Paper VI, The Grading of Interviewers: An Examination of Visible and Concealed Interviewer Error as Revealed by the Grading Tests, and Some Suggestions for Future Grading Procedure," Documents Used During the Selection and Training of Social Survey Interviewers and Selected Papers on Interviewers and Interviewing, The Social Survey Division, Central Office of Information, Great Britain, May 1952. [21] Hirschberg, D., Yuskavage, R., and Scheuren, F., "The Impact on Personal and Family Income of Adjusting the Current Population Survey for Undercoverage," Proceedings of the Social Statistics Section of the American Statistical Association, 1977. [22] Jablin, C., "Results of December 1976 Processing Tests and PRERA Verification," 81 Bureau of the Census memorandum to L.Goldberg, January 4, 1977a. [23] Jablin C., Unpublished Monthly Report of the Methods, Procedures, and Quality Control Branch of the Demographic Surveys Division, Bureau of the Census, 1977b. [24] Johnston, D.F. and Wetzel, J.R., Effect of the Census Undercount on Labor Force Estimates, Special Labor Force Report No. 105, Bureau of Labor Statistics, March 1969. [25] Jones, C., "Coverage Improvement for AHS National Sample and SMSA," Bureau of the Census memorandum to E. Gerson, September 14, 1976. [26] Jones, C., "CPS Variances-1975 Averages," Bureau of the Census memorandum to R. Hanson, April 1, 1977a. [27] Jones, C., "CPS Variances-Net Standard Errors for Monthly Estimates of Levels, Percentages and Participation Rates for the CPS Labor Force Data for the 461 Area Design," Bureau of the Census memorandum to E. Gerson, June 15, 1977b. [28] Jones, C and Aquilino, R., "Methods Test Phase III- Second Report on the Accuracy of Retrospective Interviewing and Effects of Nonself-Response on Labor Force Status," Bureau of the Census memorandum to W. Perkins, January 29, 1970. [29] Jones, C. and Blass, R., "Population Undercoverage Estimates from the CPS-Census Match," Bureau of the Census memorandum to H. Nisselson, January 17, 1975. [30] Kaitz, H. B., "On the Measurement and Concept of the Irregular Component in the Seasonal Adjustment of Economic Data", Proceeding of the Business and Economic Statistics Section of the American Statistical Association, 1962, 200-209. [31] Kaitz, H. R., "Analysis of Current Economic Data on Employment and Unemployment," Unpublished paper (draft), November 13, 1974. [32] Keyfitz, N., "Sampling with Probability Proportional to Size: Adjustment for Changes in the Probabilities," Journal of the American Statistical Association, 46 (1951), 105-109. [33] Korns, A., "Coverage Issues Raised by Comparisons Between CPS and Establishment Employment," Proceedings of the Social Statistics Section of the American Statistical Association, 1977. [34] Linebarger, J. S., "CPS/FOSDIC Study Interim Report No. 3," Bureau of the Census memorandum to H. Fasteau, August 28 1973. [35] Linebarger, J. S., "Variation in FOSDIC Scanning of Monthly Quality Control Samples," Bureau of the Census memorandum to R. T. O'Reagan, March 11, 1974. [36] Linebarger, J. S., "New Construction Time Lag Study," Bureau of the Census memorandum to C. Bostrum, August 22, 1975. [37] Linebarger, J. and Davie, W., "Proposed Quality Control Plans for CPS/FOSDIC Operation," Bureau of the Census memorandum to R. T. O'Reagan, September 12, 1973. [38] Love, L., "Preliminary Report on Finding Concerning Initial Training for CPS Interviewers," Bureau of the Census Memorandum from The Advisory Committee (Current Program Training, August 6, 197 [39) MacKenzie, W., Documentation of any Housing Survey Coverage Improvement Program, Bureau,of the Census, 1977. [40] McPherson, J. L. and Volk, M., "FOSDIC Microfilm Problems and their Solutions Bureau of the Census paper prepared for the Eleventh Annual Convention of t National Microfilm Association, Washington, D.C., April 25-27, 1962. [41] Monsour, N. J., "On the Analysis of the Star and Error of Seasonally Adjusted Series Bureau of the Census memorandum to Nisselson, July 17. 1975. [42] Montie, I. and Schwanz. D., "Coverage Improvement in the Annal Housing Survey Proceedings of the Social Statistics Section of the American Statistical Association 1977. [43] Moore, G. H., "On the Statistical Significance of Changes in Employment and Unemployment,". Statistical Reporter, No. 73-9, May 1973, 137-139. [44] Moye, D., "CPS Reinterview Results from Listing Check, Check of Noninterview Classifications, and the Household Composition Check for 1975." Bureau of Census memorandum. May 21. 1976a. [45] Moye, D., "Comparison of the 80 Percent Sample Before Reconciliation and the Percent Unreconciled Sample for 82 1975," Bureau of the Census memorandum, August 23, 1976b. [46] Novoa, R., "1970 Census: Preliminary Evaluation Results Memorandum No. 21," Bureau of the Census memorandum, October, 1971. [47] O'Reagan, R.T., "Analysis of Reflectance Measurements Made on CPS Questionnaires-CPS/FOSDIC Study: Interim Report No. I," Bureau of the Census memorandum to H. Fasteau, July 11, 1973. [48] O'Reagan, R. T., "Summary of the Interim Report on FOSDIC/CPS Study," Bureau of the Census memorandum to H. Fasteau, April 4, 1974. [49] Palmer, S. and Jones, C. "A Look at Alternative Imputation Procedures for CPS Noninterviews," Bureau of the Census unpub- lished paper, November 10, 1966. [50] Perkins, W.M., "1970 CPS Redesign: Proposed Method for Deriving Sample PSU Selection Probabilities Within 1970 NSR Strata," Bureau of the Census memorandum to J. Waksberg, August 5, 1970. [51] Perkins, W.M., "1970 CPS Redesign: Proposed Method for Deriving Sample PSU Selection Probabilities Within 1970 NSR Strata- Justification of Procedure as Unbiased," Bureau of the Census memorandum to J. Waksberg, February 19, 1971. [52] Rustemeyer, A., "Measuring Interviewer Performance in Mock Interviews," Proceedings of the Social Statistics Section of the American Statistical Association, 1977. [53] Rustemeyer, A. and Rothwell, N. D., "Observation on CPS," Bureau of the Census memorandum, December 8, 1969. [54] Rustemeyer, A. and Rothwell, N. D., "Observation of CPS Interviews in January 1970," Bureau of the Census 'memorandum, December 29, 1971. [55] Schreiner, I., "Summary Report of CPS Reinterview, First Quarter 1976," Bureau of the Census memorandum, August 27, 1976. [56] Schreiner, I., "CPS Reinterview Results from the Listing Check, Check of Noninterview Classifications, and the Household Com- position Check for 1976," Bureau of the Census memorandum, April 27, 1977. [57] Shapiro, G. M., "Current Survey Segmenting Error Involving Two Addresses with Identical House Numbers, " Bureau of the Census memorandum, August 15, 1972. [58] Shiskin, J., "Measuring Current Economic Fluctuations," Annals of Economic and Social Measurements, National Bureau of Economic Research, Vol. 2, 1973, 1-16. [59] Smouse, E., "Keyfitz Estimate Program-Verification of Program Output," Bureau of the Census memorandum, June 1, 1976. [60] U.S. Bureau of the Census, The X-11 Variant of the Census Method 11 Seasonal Adjustment Program, Technical Paper No. 15 (1967 revision) by J. Shiskin, A. H. Young, and J. C. Musgrave, U.S Government Printing Office, Washington, D.C., 1967. [61] U.S. Bureau of the Census, The Current Population Survey Reinterview Program, January 1961 through December 1966, Technical Paper No. 19, U.S. Government Printing Office, Washington, D.C., 1968. [62] U.S. Bureau of the Census, Current Population Survey Interviewer's Reference Manual, CPS-250, 1971-1976. [63] U.S. Bureau of the Census, Investigation of Census Bureau Interviewer Characteristics, Performance, and Attitudes: A Summary, G.P. Inderfurth, Working Paper No. 34, U.S. Government Printing Office, Washington, D.C. 1972. [64] U.S. Bureau of the Census, Working Paper No. 36, Response Variance in the Current Population Survey, by B.J. Tepping and K.L. Boland, U.S. Government Printing Office, Washington, D.C., 1972. [65] U.S. Bureau of the Census, Minutes of the BLS-Census Labor Force Committee Meeting of March 27, 1974 (241)," 1974. [66] U.S. Bureau of the Census, The CPS Reinterview Survey, CPS-HVS- 256, 1975. [67] U.S. Bureau of the Census, The Current Population Survey; Design and Methodology, by R.H. Hanson, Technical Paper No. 40, U.S. Government Printing Office, Washington, D.C. 1978. [68] U.S. Bureau of the Census and U.S. Bureau of Labor Statistics, Concepts and Methods Used in Labor Force Statistics Derived from the Current Population Survey, BLS Report No. 463-Series P-25, No. 62, October, 1976. 83 [69] U.S. Bureau of Labor Statistics, Employment and Earnings, Volume 19, No. 10, April 1973. [70] U.S. Bureau of Labor Statistics, Employment and Earnings, Volume 20, No. 8, February 1974. [71] U.S. Bureau of Labor Statistics, Handbook of Methods, Bulletin 910, 1976. [72] Waksberg, J., "Computer Edit of the Screening Results for Sample ED's," Bureau of the Census Final Specifications Memorandum No. 15, June 7, 1971. [73] Williams, L. E., "Methods Test Phase III: First Report on the Accuracy of Retrospective Interviewing and Effects of NonSelf- Response on Labor Force Status." Bureau of the Census memorandum to W. M. Perkins, June 24, 1969. [74] Woltman, H and Bushery, J., "Results of NCS Maximum Personal Visit-Maximum Telephone Interview Experiment," Bureau of the Census draft memorandum. September 22, 1977. [75] Woodruff, R. S. and Causey, B. D., "Computerized Method for Approximating the Variance of a Complicated Estimate," Journal of the American Statistical Association, 71 (June 1976), 315-321. 84

(spp.html)

Page Last Modified: April 20, 2007

FCSM Home
Methodology Reports