Federal
Committee on Statistical
Methodology Office of Management and Budget |
FCSM
Home ^ Methodology Reports ^ |
Statistical Policy Working Paper 9 - Contracting for Surveys
Click HERE for graphic.MEMBERS OF THE FEDERAL COMMITTEE ON STATISTICAL METHODOLOGY (June 1983) Maria Elena Gonzalez (Chair) Office of Information and Regulatory Affairs (OMB) Barbara A. Bailer Bureau of the Census (Commerce) Norman D. Beller National Center for Education Statistics (Education) Yvonne M. Bishop Energy Information Administration (Energy) Edwin J. Coleman Bureau of Economic Analysis (Commerce) John E. Cremeans Bureau of Industrial Economics (Commerce) Zahava D. Doering Defense Manpower Data Center (Defense) Marie D. Eldridge National Center for Education Statistics (Education) Daniel H. Garnick Bureau of Economic Analysis (Commerce) Charles, D. Jones Bureau of the Census (Commerce) Daniel Kasprzyk Bureau of the Census (Commerce) William E. Kibler Statistical Reporting Service (Agriculture) Thomas Plewes Bureau of Labor Statistics Raymond C. Sansing Internal Revenue Service (Treasury) Fritz J. Scheuren Internal Revenue Service (Treasury) Monroe G. Sirken National Center for Health Statistics (Health and Human Services) Wray Smith Energy Information Administration Energy Thomas G. Staples Social Security Administration (Health and Human Services) OFFICE OF INFORMATION AND REGULATORY AFFAIRS Christopher DeMuth, Administrator Thomas D. Hopkins, Deputy Administrator for Regulatory and Statistical Analysis Maria E. Gonzalez, Chairperson Federal Committee on Statistical Methodology PREFACE The Subcommittee on Contracting for Statistical Surveys was formed to review current practices by agencies in contracting for surveys and to make recommendations for their improvement. This working paper, the report of that Subcommittee, addresses the following: technical expertise needed in statistical contracting, preparation of the Request for Proposal, contractor selection, and post-award activities. While the report is intended to be useful to agencies that may utilize contracting for surveys, a broader audience may find the report of interest. Some" of the material should be useful as a supplement to the existing training provided, agency project officers. Seminars and meetings will be organized to discuss the findings of this subcommittee with Federal agency personnel involved with contracting for surveys. The working paper was prepared by the members of the Subcommittee on Contracting for Statistical Surveys, Federal Committee on Statistical Methodology. The Subcommittee was chaired by Thomas G. Staples, Social Security Administration, Department of Health and Human Services. As a subcommittee report, this document does not necessarily represent the views of the Office of Management and Budget. MEMBERS OF THE SUBCOMMITTEE ON CONTRACTING FOR STATISTICAL SURVEYS Thomas G. Staples, Chairperson Social Security Administration Department of Health and Human Services Henry P. Brehm Social Security Administration Department of Health and Human Services David W. Chapman Bureau of the Census Department of Commerce Maria E. Gonazalez Office of Information and Regulatory Affairs Office of Management and Budget Nancy D. Pearce National Center for Health Statistics Department of Health and Human Services Richard J. Pratt Bureau of Labor Statistics Department of Labor Margaret Weidenhamer Statistical Reporting Service Department of Agriculture (ii) ACKNOWLEDGEMENTS This report represents the collective effort of the Subcommittee on Contracting for Statistical Surveys. Although the subcommittee members reviewed and commented on all parts of the report, the following specific individuals were responsible for preparing the chapters: Chapter Authors I Henry P. Brehm II David W. Chapman III Margaret Weidenhamer/Richard J. Pratt IV Henry P. Brehm V Nancy D. Pearce/Thomas G. Staples Appendix I J. Timothy Sprehe II Nancy D. Pearce In addition to the subcommittee members, special recognition goes to the Federal Committee on Statistical Methodology, whose membership identified the need for this report. Portions of several committee meetings were devoted to feedback to the subcommittee as the report was developing; individual members of the committee provided detailed oral and written comments. In addition, special recognition goes (1) to the contractor and Federal agency personnel who reviewed draft copies of the report and (2) to the contractor and Federal agency representatives who participated in the surveys of sponsoring agencies and contractors. Also, several individuals who made significant contributions to the report participated in the early stages of the subcommittee's work. These include Thomas B. Jabine, Leon Gilford, Barbara A. Bailar, Bette S. Mahoney, Henrietta Hyatt, David B. Orr, Irene C. Montie, J. Timothy Sprehe, and Corrinne B. Lennox. Jacob Deutch and David Arnaudo participated in the surveys of agency and contractor representatives. Among the subcommittee members special recognition goes to Nancy D. Pearce for organizing the draft and final versions of the report and to Nancy D. Pearce and Henry P. Brehm, for pulling together a compact version of the report for the 1982 proceedings of the American Statistical Association. Maria E. Gonzalez, on numerous occasions, chaired meetings and performed other responsibilities when the chairperson was absent. Thomas G. Staples (iii) TABLE OF CONTENTS Chapter 1. Introduction. . . . . . . . . . . . . . . . . . . 1 Background and Purpose . . . . . . . . . . . . . . . . . . . 1 The Chapters to Follow . . . . . . . . . . . . . . . . . . . 4 Chapter 2. Technical Skills Required in Contracting for Surveys. . . . . . . . . . . . . . . . . . . . . . 7 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 7 Basic Types of Skills Required . . . . . . . . . . . . . . . 8 Technical Input at Various Stages of Contracting for a Survey. . . . . . . . . . . . . . . . . . . . . . . . 12 Some Technical Aspects Associated with Current Practice. . .14 Chapter 3. Considerations in the Preparation of Requests for Proposal . . . . . . . . . . . . . . . . . . . . .17 Introduction . . . . . . . . . . . . . . . . . . . . . . . .17 Initial Considerations . . . . . . . . . . . . . . . . . . .17 Possible Contents of Requests for Proposal . . . . . . . . .20 Role of the Project Officer. . . . . . . . . . . . . . . . .28 Evaluation Criteria. . . . . . . . . . . . . . . . . . . . .29 Chapter 4. Contractor Selection Process . . . . . . . . . . .31 Introduction . . . . . . . . . . . . . . . . . . . . . . . .31 Components of the Selection Process. . . . . . . . . . . . .31 Technical Evaluation of Content and Capability . . . . . . .33 Discussions with Offerors in the Competitive Range . . . . .34 The Final Award Process. . . . . . . . . . . . . . . . . . .35 Chapter 5. Post-Award Activities. . . . . . . . . . . . . . .37 Introduction . . . . . . . . . . . . . . . . . . . . . . . .37 General Techniques for Monitoring Contractor Performance . .38 Administrative Responsibilities of the Project Officer . . .38 Quality Assurance Responsibilities of the Project Officer. .40 Appendix I. Informal Survey of Agencies and Contractors Concerning Contracting for Surveys . . . . . . . .47 Introduction . . . . . . . . . . . . . . . . . . . . . . . .47 Federal Agency Responses . . . . . . . . . . . . . . . . . .48 Contractor Responses . . . . . . . . . . . . . . . . . . . .50 Appendix II. Office of Management and Budget Requirements: Information Collection Budget and Clearance. . . .55 Introduction . . . . . . . . . . . . . . . . . . . . . . . .55 Information Collection Budget. . . . . . . . . . . . . . . .56 Clearance. . . . . . . . . . . . . . . . . . . . . . . . . .56 After Receipt of Approval. . . . . . . . . . . . . . . . . .58 (v) Chapter 1. INTRODUCTION BACKGROUND AND PURPOSE Surveys are a major source of data for many agencies of the Federal Government. Surveys deal with a multitude of subjects, for example, demographic, economic, psychological, sociological, and medical topics, spanning a multiplicity of diverse governmental concerns for the general welfare. Included are surveys on the effectiveness of government programs, the impact of regulated and unregulated industrial activities, and problems for which population groups are currently or may in the future be at risk or in need of government assistance in some form. Although these topics are highly diverse. the issues in the use and conduct of surveys to obtain data on them are far more amenable to systematic treatment with a unified focus. For the Federal Government to conduct a survey, there are the three following primary options: - do the work in-house; - enter into an interagency agreement; or - contract the work out. These options can be used in combination with each other, and in actual practice this is often the case. An agency may have the data collection, reduction, and tabulation phases of a survey conducted under a contract or an interagency agreement.1 It may choose to design the survey and conduct the analysis in-house. The Annual Survey of Manufactures is an example of a statistical survey carried out by in-house staff. Planning and preparation, data collection and tabulation, and presentation of the data are done by or under the immediate supervision of Bureau of the Census employees. Additional staff needed to satisfy the demands of high volume data collection and handling are obtained on a limited individual employment basis. Because of its capacity to conduct statistical surveys, the Bureau of the Census is one of the agencies often used by Government agencies to conduct surveys. For example, the Current Population Survey conducted by the Bureau of the Census for the Bureau of Labor Statistics is a statistical survey carried out through an interagency agreement. However, agencies that do reimbursable work have limited subject matter expertise beyond their own missions. Additionally, their facilities are in demand and are by no means unlimited. For many survey efforts, then, it becomes necessary that agencies of government utilize the services of outside organizations. A range of such services is available under contract from non-federal government ___________________________ 1Statistical Policy Working Paper 8, "Statistical Interagency Agreements" (October 1982; Regulatory and Statistical Analysis Division, Office of Information and Regulatory Affairs, Office of Management and Budget), prepared by the Subcommittee on Statistical Interagency Agreements, presents data on the prevalence of interagency agreements and summarizes the prevailing practices and policies of Federal agencies with respect to their use. 1 sources. Depending on the needs of the individual agency or survey, these may include carefully delineated and defined services related to data collection and processing or more general involvement in all phases of a survey from planning to final report. In some cases the suggestion of the subject matter area in which a survey is needed may originate outside the government and result in a proposal for the entire effort, including the definition of the area for study. The overall functions of the Office of Management and Budget (OMB) as related to review and approval of agency budgets, program plans, and data collection plans set limits and provide guidance for the use of statistical surveys. For the most part, the issues of concern center on the need for the data for policy and program activities, the reasonableness of the data collection plan and questionnaires, the coordination among agencies with overlapping interests to avoid duplication of effort, and the concern for the burden on potential respondents. From another perspective, the Federal Procurement Regulations and the unit within each agency responsible for the administrative aspects of formal procurement actions, in accordance with these general regulations, provide standards and requirements for a contract as a legal document independent of what is being purchased. OMB Circular A-76 sets restrictions and requirements on developing in-house capacity versus using nonfederal contracting sources to obtain various services. This affects the justification for moving work in-house that has been done previously by contractors and contracting work out for which an in-house capacity exists. The implications of Circular A-76 for contracting for a survey should be reviewed with the procurement office. However, beyond the general standards there is no specific guidance on the use by Federal agencies of contractors for the conduct of surveys. They do not define when and how the procurement mechanism should be used for such a purpose nor do they indicate what the relationships should be among an agency's need for survey data and analysis, its staff resources, and the specification of contract requirements and monitoring of contractor performance. The Subcommittee on Contracting for Statistical Surveys of the Federal Committee on Statistical Methodology was organized to suggest ways to improve the survey contracting process in the U.S. Federal Government in the future. The Subcommittee's work was done under the general mandate of OMB. Because of its unique responsibilities, OMB was able to call upon the experience of a variety of individuals with backgrounds in diverse aspects of the conduct of surveys and the use of contracts. This report represents the collective effort of the Subcommittee. The opinions expressed here reflect the collective judgment of the Subcommittee and do not necessarily reflect those of OMB. It is not possible within any given presentation of this nature to cover all the issues related to the use and conduct of surveys. The following questions are some that need to be answered: What is the problem on which the study should focus? Should a survey be conducted as opposed to the use of some alternative research strategy or data collection approach? If a survey is to be used, should it be done totally in-house, totally under an interagency agreement, totally under a contract, or using some combination of these three approaches? The Subcommittee focused on what should be done after the decision is made to contract out the survey, and not on the factors involved in selecting this option as against an interagency agreement or in-house conduct of the survey. 2 As related to survey design and conduct, issues are: What should be the universe for study? What is the most appropriate sample design? Should the data be collected by mail, by telephone, or by personal interview? What specific questions should be asked and how should the data be analyzed? These and many other issues must be addressed at some point in the course of deciding to use, planning and preparing for, and actually conducting a survey. While the need to address such issues is discussed, there is no effort to provide input into the decision making process on such matters. There are many references and resources available on survey design and content considerations so that it would be inappropriate to try to duplicate these here. This report is intended for circulation to Federal agencies and offices which may utilize contracting for surveys. The report tries to draw attention to the issues that need to be considered, how to use or obtain the necessary expertise in preparing for and using the contract mechanism, how to prepare specifications and carry out contractor selection, and how to maximize contractor performance. It should be pointed out that this report does not deal with surveys conducted or supported under grants or interagency agreements. The report focuses on contracts awarded through an open competitive negotiated procurement mechanism based on evaluation of the technical and business proposals submitted in response to generally available Requests for Proposal (RFPs). An RFP is a formally issued notice of a government agency's need to have work done by outside organizations where contractor selection cannot be based on price alone. The RFP is the publicly released notice which contains the internally approved program Request for Contract. The primary reason for focusing on this mechanism is that it offers the agency the maximum opportunity to set the requirements for the survey and the standards for contractor performance and to select the organization that offers the best combination of anticipated product quality and cost factors, without regard to issues unrelated to these concerns. Other contract approaches, while serving other purposes, do not necessarily promote selection of the best choice of contractor for conducting a given survey at a given point in time. Among the alternative contract approaches are sole source selection because of a unique capacity to perform or based on an unsolicited proposal, and set-aside programs for competition only among small businesses in general or minority-owned small businesses. The small business set-aside and the minority-owned business programs permit competition for prime contract work among firms which are qualified for these programs but totally exclude the larger organizations with greater resources and all nonprofit research units, including academically-based units. This is part of the Federal Government's efforts to promote the economic viability of small, and particularly minority-owned, businesses. However, on any given procurement, these restrictions limit the range of competition. Although excluded from the small business and minority-owned business programs, nonprofit organizations are eligible for sole source contracting. The mechanism totally restricts competition, either because only one organization is determined to have the capacity to perform or because the organization submitted an unsolicited proposal that represents a new and unique approach and opportunity to obtain a valuable product. In this latter situation, the organization submitting the proposal controls the product offered, that is, the particular survey to be performed, and is the only one to which the contract can be awarded. In line with the intention to provide information designed to improve the ability to use the contracting mechanism for procurement of survey services it was important to learn more about what the agencies were doing at that time. Since no useful data 3 existed on this process, Subcommittee members interviewed officials of several selected Federal agencies involved in contracting for surveys and some of the contractors with which the agencies dealt. Appendix I summarizes the data collected. The agencies selected for the survey of their contracting operations were asked to have representatives of both the program and the administrative aspects of contracting operations available. The da were collected by personal interview. The questions were directed toward contracting for surveys only. The questions traced the use of this mechanism from the factors in the decision to use different types of contracts (cost or fixed-price) and how often they were used, through developing RFPS, soliciting bids and evaluating proposals, to monitoring contracts. Similar information was collected from a limited number of contractors. This information has been analyzed and its implications assessed for developing suggestions designed to improve the use of the contract mechanism for conducting surveys. The various reviews of current practice that have been made have been interwoven into this presentation. THE CHAPTERS TO FOLLOW In structuring the material to be presented, an effort was made to follow a logical order of stages in the development and management of contracts for surveys. The sequence of events in this process was a guiding principle within the general format, but not an absolute basis for organizing the chapters. The different chapters deal with the development of requirements for RFPs and with issues in the selection and monitoring of a contractor after an open competitive RFP has been issued. It must be understood that no part of this process is independent but is closely linked to and an integral part of the entire process of defining a requirement to be done under contract and seeing the contract work through to completion. In this report, "technical skills" refers to survey and programmatic content skills as distinct from management and administrative aspects of the procurement process. Chapter 2 deals extensively with general issues related to the technical skills involved in survey planning and conduct and in the use of contractors for the performance of various tasks in the process. The remaining chapters follow an order of activities in the use of contracts. Chapter 3 contains a review technical survey design issues and other considerations in preparing RFPS. Chapters 4 and 5 deal with issues in the selection of contractors and the management and evaluation of contracts after award. Chapter 2 is a detailed discussion of the skills required in contracting for surveys and the types of individuals who may serve as technical advisors. This chapter also includes a discussion of the activities and concerns in the planning and preparation for a survey to be done under a contract, the competitive selection of a contractor, and the monitoring of performance as these relate to the skills required at various stages in this process. A variety of technical survey research skills, as well as content area competencies, is needed to deal with issues such as determining the overall research design to be followed, the design of the data collection instrument or instruments, the universe to be studied, the design of the sample, the conduct of the data collection effort itself, data editing, coding, and quality control. At each stage of a survey performed under a contract, there are various technical skills required to maintain a smooth operation. Differences of in-house technical capabilities of agencies and the possible need to supplement those skills are discussed. Both the use and misuse of technical advisors by agencies are considered. 4 Chapter 3, on considerations in the preparation of RFPS, contains a review of the tasks that may need to be performed in preparing a statement of work for competition among prospective offerors rather than the skills that need to be available to the agency in undertaking this effort. The chapter includes a discussion of issues to be considered in developing an RFP. The interaction between technical requirements and degree of specification and the contract reimbursement method (that is, fixed-price or cost-type contract) is also reviewed. Different types of contracts imply different assumptions about the ability to predict and specify activities, results, and costs and, therefore, impose different limitations on the ability to trade off between cost and performance. In Chapter 4, on the contractor selection process, the selection process for open market competitive RFPs is discussed. The various stages of the contractor selection and contract negotiation process and the roles played by the contracting officer, project officer and technical evaluation panel are described. This is the framework within which the technical merits of the work proposal are presented and evaluated, and in which a contract is awarded. The process of debriefing unsuccessful offerors is also discussed. The last of the substantive chapters deals with contract management and evaluation after award. This is the period of actual contractor performance in line with the terms and requirements of the contract. The project officer and the contracting officer are jointly involved in monitoring and controlling the contract effort. The discussion in the chapter centers on the roles of these individuals and contact that should be maintained between the project officer and the contractor in monitoring these activities. Depending on the specifics of the contract and the background and competence of the project officer, this person's input into the technical aspects of the survey will vary. There may be specific activities which require approval; data collection instruments usually require formal clearance. The project officer must certify invoices for payment and must, therefore, be aware of contractor progress and current and planned activities under the contract as these relate to the remaining time and funds available. In addition to these four substantive chapters, there are two appendixes. These are a review of the informal survey of agencies and contractors conducted by the Subcommittee, and a discussion of the OMB clearance process for surveys. 5 Chapter 2. TECHNICAL SKILLS REQUIRED IN CONTRACTING FOR SURVEYS NTRODUCTION A survey is generally a rather complex activity. Furthermore, the results to be derived from a survey are usually difficult to specify, for example, types of data and analyses desired. Consequently, the task of contracting for a survey is not straightforward. Specifically, a considerable amount of effort is required to provide complete and clearly written survey requirements in a Request for Proposal (RFP), select the best offeror to receive the contract, and adequately monitor and review the progress of the survey. Technical expertise is essential to the success of contracting for and monitoring the execution of a survey. In this chapter, technical skills that should be available when contracting for surveys are presented. Following are areas in which technical skills are required: - Developing project objectives and specifications, Subject matter knowledge, - Project cost and scheduling, - Questionnaire design and testing, - Sample design and selection, - Data collection, including follow-up procedures, - Data processing, including coding, editing and file creation, - Data analysis, - Quality control, and - Survey documentation and report preparation. The subject matter knowledge required for a particular survey will, of course, vary depending on the nature of the survey. As mentioned previously, surveys deal with many subjects, for example, demographic, economic, psychological, sociological, and medical topics. The other types of technical skills listed above are more generally applicable to all surveys. It is very unusual to find all the skills required for a particular survey in one individual, although it is often true that some people will possess more than one of these skills. Consequently, it is usually necessary that there be a Government project team consisting of the project officer and one or more technical advisors to supply the required technical skills. The number and types of advisors needed will vary according to the size and complexity of the survey being planned and the experience of the project officer. Smaller and less complex surveys will usually require fewer, and less specialized advisors than large, complex surveys. 7 It is important that the need for a project team and the types of members needed be recognized early in the development of a project. At least some of the advisors should participate with the project officer in the following major phases of the survey contracting process: (1) project planning and preparation of the RFP, (2) evaluating proposals and awarding the contract, (3) and monitoring the work done under the contract. In many cases the agency contracting for a survey will not have on its staff all the technical advisors needed. In such a case, the agency should use technical advisors as part of the project team. These advisors could be persons employed by other Federal agencies or nonfederal consultants. Nonfederal consultants who assist in development of the RFP are not eligible to respond to the RFP. If advisors are needed, they should be brought in at the beginning of the contracting process so that they can participate in planning the project, in preparing the RFP, and in evaluating the proposals. if advisors are brought into the project early, they can provide the guidance and assistance needed in designing and executing a survey. It is generally a mistake to rely solely on the contractor to provide the technical expertise required during the execution of the survey plan and the processing and analysis of the data. If this is done, the technical quality of the survey may be diminished because the contractor's technical expertise may not be fully adequate, or the contractor may decide that it is not in his or her best interest to take steps to ensure the highest technical quality. Even with the highest quality contractor, the Government should still provide technical review and guidance as part of the monitoring of the survey process; otherwise, the value of the project may be reduced. The technical aspects of some phases of a survey are often not fully appreciated. For example, in the design of the data collection activities of a survey, a reasonable time schedule cannot be formulated without regard to technical input on follow-up activities, such as the number and type of follow-up attempts required. The next section of this chapter contains a discussion of the required technical skills listed earlier. The following section covers the type of technical input required at the three major stages of the contracting process. The final section includes a discussion of some of the technical aspects of current practice in contracting for surveys. BASIC TYPES OF SKILLS REQUIRED Developing Project objectives and Specifications The formulation and definition of project goals and objectives is the most fundamental aspect of a project. Often goals and objectives are not defined adequately. In such cases there will be confusion regarding the type and amount of survey information to be collected, and hence the strong possibility that the project objectives will not be met. Sometimes a project may have multiple objectives that may be somewhat conflicting. For example, if both subgroup and total population estimates are important, a decision must be made on how to allocate the sample to these subgroups. The sample allocation that would provide equal precision, or some, other specified precision, for subgroup estimates will generally differ from the sample allocation that would maximize the precision for total population estimates. Priorities must be set for each established goal and objective at the beginning of the project in order to resolve any conflicts that may subsequently evolve. 8 In addition to stating the objectives clearly, an effort should be made to define the objectives to be as narrow or specific as possible. If the objectives are too broad or general, the survey might become too large in scope to manage adequately for the time and funding available. If the goals and objectives are defined adequately and if the study population has been defined, then appropriate technical specifications can be developed for instrument design, sample design, and data analysis. consequently, these technical aspects of the project should be kept in mind as the goals and objectives are developed. Subject Matter As mentioned previously, sample surveys cover many different subjects. In order to formulate and develop a project adequately, there should be a subject matter expert on the project team at the start of the project. Such subject matter expertise is required in order to insure that all data requirements. are delineated. The subject matter expert may not know the appropriate manner in which to phrase a question in order to elicit the desired response, but will know the fundamental issues that must be explored in order to satisfy the data requirements. For example, to develop a survey to investigate the accuracy of the amounts paid to railroad retirement beneficiaries, there should be someone on the project team with knowledge of the railroad retirement system. This subject matter expertise should allow for the formulation of the most appropriate survey questions and should enable the project to collect adequate information from respondents. Project Scheduling and Costing Scheduling and costing various phases of a survey are generally difficult. Typically, a gross cost for the project is set very early, particularly for one-time surveys. Agency personnel are generally limited by the initial gross cost allocated to the project. The amount of time required for planning and writing the RFP is often underestimated. This can delay the scheduled start of the survey which, in turn, can jeopardize the quality of the work. Sufficient time for execution of the survey by the contractor must be allowed. Since there are several aspects of a survey, such as printing, data collection, and data processing, that may require special clearances, applicable requirements should be determined as early as possible in development of a survey to be conducted under contract. The procurement office can provide a list of required clearances so that their applicability to a particular survey can be determined and provision for compliance can be made in the project schedule. In addition, there are several survey tasks which involve aspects for which the completion time and cost are typically hard to predict. For example, a survey may involve a large frame development phase in which a number of frame sources have to be investigated. This could require working with a number of computer files involving problems of incompatibility between various computer systems, different record layouts and file definitions, and matching elements in different files. These types of activities are typically difficult to assess in terms of time and cost. Other survey activities that are typically difficult to assess in terms of cost and time requirements are questionnaire design and testing, data collection, and data processing. Due to the potential for cost and scheduling problems as indicated above, it is important during the project development phase to include a person on the 9 project team who has experience in developing project schedules and cost estimates for surveys. Questionnaire Development and Testing Proper development of a survey questionnaire and other data collection instruments is a deceptively difficult phase of a project. The knowledge, time, and effort required to complete this phase adequately is often underrated. In addition to preparing the instruments, time must be allowed to adequately test them. Testing is important since it is generally impossible to anticipate all the problems that will arise when an instrument is used. It is important to include someone on the project team who has had considerable experience in questionnaire development and testing so that the common pitfalls of instrument design can be avoided and so that the instrument may be designed to facilitate data processing. If a questionnaire is used to collect the survey data and the questionnaire is not developed by someone with appropriate experience, it is likely that some questions will be unclear to the respondents, will be misinterpreted by the respondents, will confuse or upset some respondents, or will tend to influence or "lead" the respondents. The position of the question on the questionnaire may influence the response rate to the question. For example, sensitive questions, like income, are usually answered at a higher rate if they are asked after the respondent has become more comfortable with the interviewer. Also, the nature of the questions surrounding a question of interest may influence response proportions by several percentage points. Furthermore, the questionnaire may be difficult to administer and may not provide adequate survey information. In addition to operational difficulties, these problems can cause both response and nonresponse biases in survey estimates. Sample Design and Selection Even for a survey that seems straightforward in terms of the information to be gathered and the definition of the target population, there are usually some difficult questions that arise in designing the sample selection method, particularly if the survey has multiple objectives. Questions about sample size determination, frame development, stratification procedures, sample allocation to strata, or whether or not to use differential probabilities of selection arise in all surveys. These questions become particularly difficult to answer when there are conflicting objectives. For example, if the project team has not decided whether subgroup estimates are more important than overall estimates or vice versa, satisfactory decisions on stratification and allocation cannot be made. ,Many of the questions that arise are not easily answered by referring to standard textbooks. Consequently, the project team should include a person who not only has. the theoretical knowledge of sampling methods, but who has also had substantial experience in the practical application of sampling principles to the design of the type of sample survey involved in the contract. This type of experience would include, for example, knowledge of available frame sources and of data available to use for strata formation or for assigning differential probabilities of selection. Data Collection The data collection phase of a project contains several aspects. First, the mode of interviewing - personal visit, self-administered questionnaire (usually mail), or telephone - should be selected based on the type of questions to be asked, the population to be studied, and the time and resources available. Next, interviewers must be recruited and trained for the in-person or telephone administration of a 10 questionnaire. Even if a staff of experienced interviewers is available, some training for each new questionnaire is necessary. A budget and schedule for interviewing, including callback procedures, must be developed. For personal interviews, the schedule often includes a mailing or telephone call to the selected sample units prior to the personal interview. The number and timing of callback visits allowed in the schedule will have an important impact on the survey response rate, which, in turn, is a key factor in controlling nonresponse bias. Acceptable response rates need to be determined and follow-up procedures must be devised and implemented. Another aspect of data collection is questionnaire check-in. In addition to recording the receipt of the questionnaires, this involves the development of editing procedures to check for faulty or missing data. Procedures also have to be developed to follow-up on questionnaires that fail this edit. These procedures are important in order to improve data quality and reduce the response and nonresponse biases. Data Processing The data processing phase usually involves entering the survey data into a computer file so that it can be accessed and analyzed. The first step is to provide any coding and editing of questionnaire responses that are needed. The coding is needed to convert all the responses into numerical codes for entry into a computer file. For example, if occupation was a survey item, a numerical code would be assigned to each occupation to use in recording the responses on the file. Proficiency in this activity requires considerable practical experience in coding and editing of survey data and knowledge of computer systems. Editing is necessary to detect responses that are unreasonable or that are inconsistent with other responses. The editing may either be performed manually or by use of the computer. Developing editing rules is difficult and requires input from subject matter specialists. Responses that fail edit checks can either be followed up, replaced (imputed for) or discarded. All imputed data should be flagged on the file. The survey data are placed into a computer file by having the coded questionnaire responses keyed to a specified record format. It is important that the computer record layout be documented completely and accurately so that the data will be correctly accessed. Data Analysis An analytic plan and strategy should be developed in the earliest stages of the survey development process. Often included in the plan are specific table shells to be used in survey tabulations. This plan is the foundation for sample and questionnaire design and is essential for the survey to provide the results intended. Often the survey analysis phase includes the estimation of population and subpopulation characteristics (for example, means, totals, and proportions). In addition, there are many types of more complex statistical analyses that may be applied to survey data, depending on the objectives of the survey. Some examples of analyses performed are the following: hypothesis testing on population means or proportions, regression and correlation, analysis of variance, and log-linear analysis. Before statistical analyses are applied to survey data, imputed (or pseudo) responses are often inserted in the records of respondents that have missing or faulty responses. Setting up an imputation procedure for missing data is a complex process that 11 requires experience. Also, differential probabilities of selection should be taken into account in applying statistical analyses. This is often accomplished by assigning weights to survey respondents to reflect varying probabilities of selection. These weights are usually adjusted to account for eligible sample units that do not respond. Once the survey estimates and statistical analyses have been made, the results must be examined. Study conclusions, and perhaps recommendations, are then made. The data analysis phase is a very important part of the project and requires the assistance of appropriate subject matter experts and of a statistician who has had experience in analyzing data from surveys. Quality Control During virtually any survey operation there are many points when data collection or processing errors can be made that diminish the accuracy of survey data. These instances include sample selection, interviewing,and/or completing the survey instrument, check-in and field editing, coding and office editing, and keying the survey data. It is strongly recommended that quality control procedures be set up at each step to detect and correct as many of these survey operations errors as is feasible. At a minimum, the quality control program should cover interviewing, field editing, coding, and keying. These procedures should be set up by a person who has had experience with survey quality control procedures. Survey Documentation and Report Preparation It is important throughout the project to prepare enough project reports to adequately document the survey procedures, results, and conclusions. Poor documentation is a very common problem with survey projects. One reason for inadequate documentation is that project funds often run low and time becomes short near the end of a project. A conscientious effort should be made to allow adequate time in the project schedule for ongoing documentation of the survey procedures used. These procedures include definition of the target population and frame construction; interviewing procedures, including callback and follow-up procedures; sample design and selection; editing and coding procedures; weighting and nonresponse adjustment methods; estimation and variance estimation equations and statistical analyses used; and quality control methods applied. The final project report should include survey results, analyses, conclusions, recommendations based on the results, and suggestions for additional research. Project specialists, such as subject matter specialists and statisticians, should be involved in preparation of the final report. TECHNICAL INPUT AT VARIOUS STAGES OF CONTRACTING FOR A SURVEY As mentioned earlier, substantial technical input is needed at each of the following three major phases of contracting for a survey: project planning and preparation of the RFP; evaluating proposals and awarding the contract; and monitoring the contractor's performance. 12 Unfortunately, the importance and/or extent of technical input needed at these stages is often underestimated, thus diminishing the quality of the survey. The required inputs are discussed below for each of these major phases. Project Planning and Preparation of the RFP Right from the start of the process of contracting for a survey, it is important to have technical input and program and subject matter expertise. Subject matter experts and statisticians can help define specific project goals and objectives that will satisfy the purpose of the study and yet will be achievable, within the time and budget available for the project, by applying appropriate survey and statistical methods. A statistician should participate in the development of the RFP so that it will be complete and accurate in terms of the statistical requirements of the survey. In cooperation with the project staff, the statistician should prepare or assist with the preparation of the following portions of the RFP;1 - Sample size specifications, if any, for the target population and/or for population subgroups, - Precision requirements for one or more key survey estimates, including any requirements for population subgroups, - Response rate required and/or number of callbacks specified, - Requirements for nonresponse imputation procedures, - Quality control procedures, - Variance estimation requirements, and - Specification of particular population or subpopulation estimates required. If a statistician does not participate in the preparation of those portions of the RFP that address the above topics, it is virtually certain that some of these specifications will be omitted from the RFP, and some of those included will not be properly stated. In such a case, offerors would be confused and may prepare proposals that miss the real objectives or which are somewhat vague, incomplete, or misdirected in terms of important technical aspects of the proposed survey. This, in turn, would make it more difficult to compare the proposals and to accurately evaluate them in terms of technical content. Judging and Selecting Among Offerors Technical input is essential in the evaluation of the proposals. In addition to subject matter specialists, at least two persons with varying experience in the different aspects of a survey should be included on the proposal review panel. At least one of these persons should be a statistician. A major focus of the technical review of proposals is the offerors' responsiveness to the survey specifications, such as frame construction, sample design and selection, quality control plans, and estimation and analysis plans. Just as important as evaluating these technical responses is the assessment of the offerors' abilities to fulfill the statements and claims made in their proposals. ___________________________ 1For a general type of RFP the technical input discussed above is still needed. However, some of these items would not usually be specified. 13 For example, if they claim that they would achieve an 85 percent response rate, is there sufficient evidence to support this? The basis for this type of evaluation would primarily be the previous experience and perforMance of each offeror and the technical skill and experience of the proposed project staff. Of course, the survey plans in the proposal would also be used to help assess the offerors' potential to achieve the proposed statistical standards. Monitoring the Contractor's Performance After a contract is awarded and the project begins, technical input is still very important to the success of the project. The contractor will presumably be describing details of their survey plans at meetings and in written documents. Technical personnel representing the agency should attend all project meetings which involve technical issues and should review all project documents to ensure that specified technical standards are met. Even if the contractor's project staff appears to be technically sound, it is necessary for the agency to continue technical input. This should ensure that the standards of the contract are met and may avoid technical problems that might have appeared near the end of the project if technical input had been interrupted. Any problems on technical issues between the agency and the contractor should be discussed and resolved. If resolution of these problems requires changes in the specifications, the changes should be proposed to the contracting officer for decision and formal action. To facilitate resolution of problems, there should be an open line of communication between the contractor's statisticians and other technical staff personnel and a liaison for he agency's project staff. Good communication between contractor and agency staffs can prevent substantial misunderstandings on technical issues that may arise. Ideally, there should be enough discussion of technical issues between the agency staff and contractor staff so that when the contractor submits a technical document, such as a sampling or analysis plan, agency personnel will be aware of the major technical aspects of the document. If communication has not been good and major problems with technical issues arise, there may be substantial time delays in the project work. In addition to participating in meetings and reviewing documents, in some surveys it is useful for agency technical personnel to make site visits. The purpose of a site visit might be to observe and review the implementation of sampling procedures, interview activities, coding and editing, or quality control activities. SOME TECHNICAL ASPECTS ASSOCIATED WITH CURRENT PRACTICE Use and Misuse of Technical Advisors As expressed previously, the use of technical advisors in contracting for a survey is vital to the success of the project. Although project managers generally recognize this need, technical advisors are not always used properly. In the previous section it was pointed out that it is important to have technical input and program-specific expertise from the beginning of the process. Often the technical advisors are added to the project team too late to assist with the preparation of the RFP. In such a case, the specifications of the survey may be inadequately defined. For example, the precision requirements for the survey may be unclear, causing confusion and apprehension among offerors. As another example, the minimum acceptable response rate may have been omitted, which would probably cause problems in comparing proposals. 14 Technical advisors are generally called upon to review and judge the technical proposals submitted. However, in addition to rating proposals on technical aspects with which they have knowledge and or experience, members of the review panel are also asked to rate proposals with respect to technical aspects with which they are less "familiar, or even unfamiliar. For example, a statistician who specializes in sample design may be asked to rate the proposals with respect to questionnaire design and quality control procedures, in addition to sampling aspects. The reviewer's experience may be rather limited in areas other than sampling. As a result, ratings of these aspects of the proposals might be inappropriate. To try to minimize this problem, individual reviewers, during the discussion and review of ratings, may revise their original score on an item to reflect the opinion of another reviewer with particular expertise in the area. To allow for this type of rating adjustment, it is advisable for an agency to attempt to balance the members of the review panel with respect to the important technical aspects of the proposals that need to be rated. Use of Outside Help to Supplement in-house Technical skills For many surveys being contracted, the in-house technical skills needed to design the survey, to judge offerors, and to monitor survey progress, are not fully adequate. In such cases technical advisors from other Federal agencies or nonfederal consultants should be added to the agency's project team. For example, advisors specializing in sample" design, questionnaire design, or field operations may be needed. it is, of course, important that the advisors required are added to the project team early in the development of the survey. As was indicated earlier in this chapter, the agency should have its own technical advisors to provide the technical expertise in a particular area, rather than relying solely on the contractor's expertise. Even if outside consultation is needed, it is important to have the appropriate technical advisors on the agency's project staff to protect the agency's interests. There are a number of Federal agencies that have technical staff who should be able to provide assistance directly or to suggest others with appropriate skills. These agencies include the following: Bureau of Economic Analysis, Department of Commerce; Bureau of Labor Statistics, Department of Labor; Bureau of the Census, Department of Commerce; Bureau of Justice Statistics, Department of Justice; National Center for Education Statistics, Department of Education; National Center for Health Statistics, Department of Health and Human Services; Office of Research and Statistics, Social Security Administration, Department of Health and Human Services; Statistical Reporting Service, Department of Agriculture; Statistics Division, Internal Revenue Service, Department of the Treasury. 15 Chapter 3. CONSIDERATIONS IN THE PREPARATION OF RFPS INTRODUCTION This chapter covers some considerations in the preparation of a Request for Proposal (RFP) for a survey. These comments and suggestions are not all-inclusive; additional considerations will be warranted for some RFPS. For example, certain types of surveys - such as repetitive or longitudinal surveys - pose special problems that are not addressed here. All of the possibilities cannot be covered because of the wide range of purposes and survey designs encountered in surveys sponsored by the Federal Government. Emphasis is placed on elements which are included in many of the RFPs prepared in Federal Government agencies and on those portions of the RFP to which the project officer or other technical personnel are most likely to contribute.1 On the other hand, the comments and suggestions which are presented are not likely to be applicable to every RFP, either because of the nature of the proposed surveys or because of variations in regulations, policies or procedures among government agencies. INITIAL CONSIDERATIONS Because of the variations noted above, it is essential to contact the procurement off ice that services the sponsoring agency at the conceptual stage when a survey contract is contemplated. Early discussions with the contracting office will enable determination of the contracting options, if any, which are available, the input expected from the program staff, the steps which should be followed from development of the RFP and a list of potential offerors through award of the contract, and the amount of time each step is likely to take. Payment Provisions One option that might be available concerns the payment provisions of the proposed contract. A contract may stipulate a fixed-price which is agreed upon in advance by the sponsor and the contractor, or it may provide for reimbursement of allowable costs (up to a preset maximum) incurred by the contractor in performing the survey. The decision about which approach to use is usually made by the procurement office based on technical contracting considerations, not by the program staff. Some agencies use only one of these financial arrangements for all of their contracts; some agencies use one or the other, depending" on the nature of the survey to be undertaken. The RFP should indicate whether the proposed contract will stipulate a fixed price or will provide for reimbursement of costs incurred by the contractor. There are advantages and disadvantages to each of these payment mechanisms. For example, fixed-price contracts assure, in advance, that sufficient funds are available for completion of the work and keep the sponsor's administrative audit expenses to a ___________________________ 1A complete RFP contains a number of standard provisions or sections originating in the contracting office to which the program staff have little or no input. 17 minimum. However, some qualified potential offerors will not submit a proposal for a fixed-price contract and competition is therefore reduced. it is particularly important that fixed-price contracts contain comprehensive, detailed statements of the requirements for the work to be performed by the contractor to avoid misunderstandings about what the agreed-upon price actually covers. These requirements should be based on a combination or merging of the specifications in the RFP and the contents of the successful offeror's final proposal. If there are many unknown or undecided factors in the project plan that affect the predictability of costs, a cost-type contract may be preferable, provided adequate funding will be available. A "dollar cap" or maximum cost can be included in a cost-type contract. If a cap is specified, flexibility should be built into the survey design so that if avoiding cost over-runs, which jeopardize completion of the survey, is not possible, a limited but still useful product can be obtained. Phased Surveys If there are any serious doubts about the feasibility of key aspects of the survey plans because of unknown factors such as the ability or willingness of designated respondents to provide the information sought, it may be prudent to limit the scope Of the project to which the sponsor will be committed.2 One approach which might be appropriate in this situation is to issue an RFP solely for an evaluation of the proposed methodology, with the stated intention of issuing another RFP for a full-scale survey if the initial effort indicates the methodology is feasible. Obviously, this approach would consume more time because of the two RFPs involved. Another possible approach is to specify work segments which may be done during the course of the contract if the concerns are resolved in the earlier stages of the contract. If a phased survey is deemed advisable, early discussions with the Procurement office will indicate what contracting alternatives are available for consideration and their advantages and disadvantages. Survey Design Specification RFPs may state all, some, or none of the details of the survey design. The level of survey design specification found in RFPS, in practice, ranges from "This is exactly what we want, including the questionnaire to be useful" to "We have a problem and need information to help solve it; what do you suggest?" The choice of the level of specification is dictated by a number of considerations such as the nature of the survey to be conducted, the applicable regulations and policies, and the technical skills available among the sponsor's staff or advisors. Detailed survey design specifications may be appropriate if the sponsoring agency has highly qualified and experienced statistical and survey staff available to develop the RFP who are aware of potential contractors' capabilities and resources, and who know exactly how the agency wants the survey to be conducted. ___________________________ 2Although the general provisions incorporated in Federal survey contracts usually provide for termination of the contract for the convenience of the Government, exercising provision can be difficult, complex, and time consuming. 18 Such an RFP may be more difficult and time consuming to develop than one that calls for offerors to suggest survey designs. Maximal specification of the survey design will indicate the quality standards of the sponsor to potential offerors, and will also permit them to repeat the requirements in their proposals without demonstrating capability or understanding of the problem. It may be easier and quicker to evaluate the responses to a detailed RFP, but more difficult to determine the technical capabilities of the offerors. Selection is therefore more likely to be made primarily on price. Any unnecessary constraints imposed on the survey design would preclude desirable flexibility and innovation by offerors which could improve the quality of the survey or decrease its cost. Furthermore, some of the more competent survey research organizations dislike this approach and are reluctant to respond to this type of RFP. Less detailed RFPs which, invite offerors to make suggestions or develop survey designs, allow them to apply their knowledge and demonstrate their ingenuity or creativity. However, minimum design criteria or quality standards should be stated, when known, even if the offerors are asked to develop the survey design. Evaluating the proposals received in response to a less detailed RFP is likely to be more difficult and time consuming. When an RFP does not specify the details of the survey design, the successful offeror's explicit statement of the survey design specifications is commonly incorporated in the contract awarded. An alternative that incorporates many of the advantages of both the detailed and less detailed survey design specification approaches has been used effectively by some agencies. In this procedure, potential offerors are invited to submit proposals based on detailed specifications but are also encouraged to suggest modifications to the detailed plans which they believe will be advantageous to the sponsoring agency. Bidders' Conferences and Due Dates Some RFPs - usually only those for large or complex surveys - include provision for a Bidders' Conference on a specified date to which prospective offerors are invited. The conference should be held after prospective offerors have had the opportunity to study the RFP, but well before the due date for proposals. They may be asked to submit their questions beforehand so that answers can be prepared in advance for the conference. At the meeting, technical and procurement personnel from the sponsoring agency answer questions raised by potential offerors about the RFP or arrange to provide answers later for any issues which cannot be resolved during the conference. They may also review contractual administrative requirements. Discussions at a Bidders' Conference will benefit the sponsoring agency if they identify RFP specifications which are misunderstood or which appear to potential contractors to be inconsistent, inadvisable or unacceptable. Any additional information or material developed as a result of the Bidders' Conference should be mailed to all requesters of the RFP. It is very important in any RFP to allow a reasonable amount of time for preparation and submission of proposals - at least 30 calendar days after the RFPs are mailed and more than 30 days if the proposed survey is complex or offerors are asked to develop substantial portions of the survey design. The availability of an RFP is announced through the Commerce Business Daily. it is also advisable that the names and addresses on mailing lists of those to receive the RFP are current. 19 POSSIBLE CONTENTS OF RFP's When developing an RFP, it may be helpful to refer to RFPs for similar surveys conducted by the sponsor or by another Federal agency as sources of ideas for content and wording. However, these RFPs should be used together with comments or suggestions from persons familiar with those surveys about problems or misunderstandings which arose during the contractor selection process or the conduct of the study - particularly difficulties which might have been forestalled if the RFP had been written differently. The goals of the procurement process should be kept in mind when an RFP is written. Ideally, an RFP does the following: - Requests the development of or specifies a technically sound survey design which will meet the sponsor's information needs and is realistic, considering the time and funds available for the project and the capabilities and resources of potential contractors. - Conveys all the information needed by potential contractors to prepare a responsive proposal, including all the terms, conditions, and provisions that the sponsor intends to incorporate in the contract. - Asks for all the information that procurement and technical personnel will need to conduct an accurate and equitable evaluation of the proposals received. - Attracts responsive proposals from enough of the qualified potential contractors to provide adequate competition and to permit a suitable selection. Some agencies include in the RFP a table of contents for responses to be followed by all offerors and stipulate that staffing requirements should be specified in a common measure (for example, hours). This simplifies reading and comparison of proposals. Specific suggestions for developing an RFP that meets these criteria are presented in the remainder of this chapter. Statement of Survey Objectives Every RFP should contain a clear statement of the specific objectives of the survey to be conducted under the ensuing contract. Discussing the proposed research in the larger context of the mission of the agency may also be advisable, particularly if the project is part of an ongoing integrated program. The statement of the survey objectives should also indicate what groups (such as government agencies, congressional committees, state or local governments, trade associations, and business firms) will be the key users of the findings. If data from the survey or an interpretative report of the findings must be provided in time to meet a deadline, the date of and reason for the deadline should also be stated. Any germane reference material that would be helpful to potential offerors in planning the broad outline of the proposed survey or in estimating time, skill, or cost requirements should be cited and also included in the RFP package. If the material cannot be provided with the RFP, information on where and when it can be accessed or obtained should be supplied. Appropriate reference material might include such information as estimated numbers of eligible respondents, geographic distribution of eligible respondents, and the methodology and results of any related research that has been conducted by or is known to the sponsoring agency. However, materials which 20 would be of interest only to the successful offeror should not be included with the RFP. Although RFPs sometimes state anticipated level of effort in terms of person-years or similar proxies for dollar estimates, the Subcommittee's interviews with contractors indicated that these proxies are less helpful to them than dollar estimates would be because of wide variations from agency to agency in the dollar amount one person-year signifies. If agency procurement policies permit, it may be desirable to inform potential offerors of the approximate amount of funds available for the contract. Providing offerors with this information would help avert some of the difficulties that might otherwise be encountered in the contractor selection stage, such as the problems that arise when the cost estimates for all of the technically sound proposals received by the sponsoring agency substantially exceed the funds available for the contract. If some but not all of the technically sound proposals fall outside the anticipated price range, this reduction in competition may result in a real but less obvious disadvantage to the sponsoring agency. Request for Information About the Offeror The RFP should request the relevant information about the offeror. It is not sufficient for an RFP to simply define the requirements for the technical proposals. Before a contract is awarded, the reliability of the contractor should be established. When appropriate, the following specific requests should be responded to by the offerors before a contract is awarded. Description of experiences. The following list contains areas where it could be important to obtain specific descriptions of the offerors' experiences: - frame refinement processes, - probability sample designs, especially for large-scale surveys, - data collection methodology (including callback procedures), - questionnaire development and testing, - estimation procedures, including nonresponse adjustments, - variance computation procedures, especially for complex samples, - training and supervision of interviewers, coders, and others, - analysis of data from large-scale surveys, - data processing, - quality control, and - publications. Examples of previous relevant work. The examples of offerors' work should include questionnaires, sampling documentation, instruction manuals, and reports containing the results of the projects. The agency should request information on all previous work pertaining to the subject matter areas for a specified period of time. The time period should be such that enough material is received but that the offeror does not inundate the agency with paper. In most cases, five years should be a reasonable period for both the offerors and the agency. Included with this information should be the name and current telephone number of the sponsor's representative (or government project officer). At least a sample of these references should be verified by the agency before the award of any contract. 21 Staff experience. This should include the experience of the current staff. This could differ substantially from the company's experience. Their background in statistical methodology, previous contracts, and subject matter areas, especially the ones related to the RFP, should be stated. Key project staff. The names of key project staff members should be required along with the percent of time each one will work on the project. Their resumes should be included with the proposal. It may be desirable to have a key personnel clause stating which staff members cannot be replaced on the project without government approval. Letters of intent should be provided by key persons who are not members of the offeror's firm. Field staff. The interviewers' field operations background in the primary sampling units (PSUs) in the frame from which the sample will be drawn should be requested. Also their experience in handling nonrespondents should be stated. The background of supervisory and clerical field staff should also be requested. Conflicts of interest. The RFP should contain a section requesting information that will allow the agency to determine if any possible conflicts of interest exist. This section could include questions about the offeror regarding other clients, activities of staff members, subsidiaries, or parent companies. Facilities. In the RFP, the agency should request information about the potential offerors! facilities in order to make certain they are adequate to complete the contract. This is especially true of computer facilities. The offeror should have the computer capabilities to handle all data processing requirements. If necessary, this should include a requirement of compatibility with the agency's data processing facilities. Subcontracting. If an offeror plans to use a subcontractor in any aspect of performance under the contract, detailed information should be provided on their experience, resources, and facilities. If subcontracting is anticipated but there is no identified subcontractor, standards for subcontractor selection should be specified. Survey Design Checklist The purpose of this section is to present a descriptive checklist which contains topics that should be addressed or considered when writing the survey design portion of an RFP. As was stated previously, the amount of specification is dependent on many factors. Ideally the amount of detail that each item in the checklist requires will depend on the type of study to be conducted. For this reason, the following list should be used only as a guide and the appropriateness of each item should be judged separately for each RFP. However, if a design topic is not specified in detail, it should be mentioned as an item to be discussed by the offeror. If an item is not mentioned in the RFP, the agency might not be able to use it as a criterion for evaluating the proposals. In addition, the importance of consulting a qualified sampling statistician before proceeding with this section cannot be overemphasized. Design and selection Definition of the target population and key subgroups. The objectives of the survey should define the population that is intended to be covered. However, it may be necessary to eliminate certain segments of the population and thereby redefine the scope of the survey because of practical problems. Examples would be excluding Alaska and Hawaii because of the high cost of collecting information by personal visit, excluding nonrepriceable items in the production of a price 22 index, and excluding nontelephone households when telephone interviewing is the primary method of data collection. The survey results will apply only to the population sampled. However, it might be possible to obtain estimates for the excluded populations by using alternative collection methods or by using auxiliary data which are correlated to the information gathered in the survey. The RFP could require that proposals include a method of estimating for the excluded population. Any conclusions for these groups would have to be stated separately and their limitations would have to be noted. A description of any subpopulations for which estimates are desired should be included. Examples could be geographic and demographic detail. Sampling frame. Ideally the sampling frame should fully cover the target population. Every effort should be made to.ensure that the source is as current and as complete as possible. Consideration should be given to requiring procedures for updating the available source, including merging information from different records and removing duplicates. If more than one frame is available, a decision must be made on which one to use or whether to use multiple frames in the survey. If multiple frames are used, a method to identify and handle overlap in the frame must be developed. Sample selection. The RFP should specify that a probability sample be selected for the survey and that the sample design be adequately defined. This includes a description of the proposed sampling plan for each stage of sampling. The details of this description should include the following topics for each stage of selection: - Definition of the sampling unit. A sampling unit is a cluster of one or more elementary (or observation) units. Each elementary unit should belong to exactly one sampling unit. In some RFPs the sampling units for some of the stages of selection (for example, the primary sampling units or PSUS) are specified. Any sampling units that are not specified in the RFP should be clearly defined in each proposal. Examples of sampling units are counties, enumeration districts, households, persons, companies, and retail outlets. - Number (or expected number) of sampling units to be chosen. In many cases the RFP gives the target final stage sample size (for example, 5,000 completed interviews) and sometimes the first stage target sample size (for example, 100 PSUS). A proposal should indicate all selection stage sample sizes not specified in the RFP. - Specific method of selecting units from the frame. The method of selecting sampling units at a given stage (for example, simple random sampling, systematic sampling, or stratified sampling) should be clearly specified so that the probabilities of selection of the sampling units can be derived. In addition, any variation from strict probability sampling that an offeror feels is required should be clearly described and justified in the proposal. Examples of cases for which variations from probability sampling may be appropriate are unavoidable coverage deficiencies, or the selection of a major brand to represent a product line in a price survey. Offerors should base their choices of sampling units, sample sizes, and selection methods for the various stages of a sample design on optimization. considerations (that is, maximum precision per unit of cost). The RFP should require that proposals include a justification for all the sample design choices made. 23 It is essential that at least one sampling statistician with experience in the particular area of interest assist in the preparation of the sampling portions of the RFP. It should be noted that unless probability sampling is required in the RFP, a proposal cannot be considered technically unacceptable just because it allows for some form of nonprobability sampling. This could cause problems during the contractor selection procedure. In addition, the Office of Management and Budget (OMB) requires justification if probability sampling is not used. Sampling error. Two different approaches are commonly used in RFPs for sample surveys to specify the level of effort desired by sponsors to provide estimates that are sufficiently reliable for their purposes. One method is to specify the sample size; the other is to specify the precision (that is, desired minimum level of sampling error) for estimates of one or more key variables. It is not desirable or practical to specify both sample size and precision in the RFP, since the desired precision may be impossible to achieve with the specified sample size. Whichever is specified, the other should be an important factor in evaluating the proposal. The justification for using one method over the other is dependent on the study to be done and the resources available. The RFP often specifies the target sample size, usually in terms of the number of completed questionnaires or interviews. The derivation of a target sample size may be based on cost considerations or on the approximate precision levels desired. If a sample size is specified in the RFP, it is important to clearly define the final stage unit and whether or not the sample size is in terms of original selections, inscope selection, or completed interviews. Any required sample sizes for population subgroups should be clearly stated. Because of the different components of the sampling error, it may be necessary to require a minimum number of sampling units at each stage of selection. Otherwise the resulting sample, although meeting all the specified criteria, might not yield acceptable results. Finally, if the sample size requirements are in terms of completed interviews, the RFP should include a precise definition of what data must be collected before an interview is complete. If the sampling error is to be specified then a decision must be made whether it will apply to one key variable or more than one key variable. If it applies to more than one, will there be a controlling variable, that is, the design which meets requirements for that variable will exceed requirements for all other variables. In addition, it must be made clear whether the precision requirements are in terms of the standard error, the coefficient of variation (relative standard error), or 95 percent confidence levels (+ 2 standard errors). Any misinterpretation of these requirements can lead to gross differences in the proposed sample sizes. Also it must be made clear whether the required minimum sampling error will apply to specific subpopulations "or just the overall population under consideration. A qualified sampling statistician should be consulted to assist with the decisions made in these areas. Data collection, questionnaire development and testing This section of the RFP, more than any other, will require input from the.people who handle the agency's OMB clearance process. In addition to knowing the agency's procedures regarding data collection they are familiar with OMB's regulations. (See appendix 11 for a discussion of OMB's clearance and, Information Collection Budget 24 requirements.) Their assistance can help in making the clearance process go as smoothly as possible. When preparing an RFP, the following issues concerning data collection, questionnaire development, and testing, should be considered. While they may not apply in all cases, when they do they will have an effect on the contract price. Therefore, in an attempt to avoid any misinterpretation by potential offerors, these issues should be addressed when applicable. Depending on the situation, they can be either explicitly defined or stated in general terms and then used in the evaluation of the proposals. Method of data collection. The three most common types are personal (face-to-face) interview, telephone interview, and self-administered (primarily mail) questionnaires. Each has specific advantages and disadvantages which must be considered. They may also be used together in a survey. An example would be a self-administered questionnaire and a personal interview follow-up. It is important to make certain that the desired information can be collected successfully or more accurately and that the target response rates can be achieved by the proposed method or combination of methods. Mail questionnaires seldom produce adequate response rates without some form of telephone or field follow-up. Data specifications or information to be obtained. They should be listed in sufficient detail to indicate what the questionnaire content should be and what the amount of effort required in developing and testing the questionnaire will be. They should be complete and realistic. When the survey is covering a new topic, the sponsor should identify the variables for which data are needed, but should allow the contractor, once selected, to play a major role in developing the specific questions needed to get data on these variables. Pretest. The need for a pretest is paramount in any survey. Without it the chances of achieving good results are minimal. The three following primary issues arise in a pretest which must be considered in any survey: sample size, probability sampling, and the survey time schedule. Sample sizes should be sufficiently large to learn about significant problems that may be encountered. For this reason, specifying a pretest of nine respondents so that the pretest does not have to be cleared through OMB should be avoided. Furthermore, clearance at an early stage of development may identify possible overlap of the planned survey with work of other agencies or may bring to light experience in other surveys that will help to avoid repetition of past mistakes. It is better to determine the sample sizes needed without reference to clearance requirements and to ensure that the survey development schedule allows enough time for necessary clearances. While probability sampling at all levels of selection is desirable for nearly all surveys, it may not always be indicated for pretests. Purposive sample designs are frequently used in pretests to ensure inclusion of important subpopulations. Probability selection methods at the final stages (that is, households within block) are still desirable to avoid selecting the "easy to get" units. The survey time schedule should allow for adequate time and professional staff to both conduct and analyze the findings from the pretest and to use the findings effectively in subsequent stages of development. 25 The following is a minimum list of topics which should be addressed in the planning of a pretest: - Should exploratory interviews be conducted? The purposes might include identifying items that are difficult or sensitive to answer, determining if the data specifications are complete, or ascertaining what records, if any, are needed and if they are available. - Will the pretest be conducted in waves? That is, will the pretest sample be sent out in parts, each of which contains questionnaires modified based on the findings in the previous part? - Will the effect of the sequence of the questions be considered? - How long is the average time for pretest interviews likely to be? - Will the pretest determine the required experience level of the interviewers or is this one of the requirements in the RFP? - What records, if any, will the respondent need to refer to and will they be reasonably accessible? - What results are necessary before advancing to the next stage of the study? Respondents' obligation to reply. Any information about the respondents' obligation to reply that would help the offerors should be included in the RFP. Most surveys are voluntary. However, if the survey is mandatory, if a response is required in order to obtain or retain benefits, or if informed consent is required, this could greatly affect the proposals and should be included. Length of interviews. If the contractors are to develop the questionnaire, a ceiling on the interview time should be set. A respondent may agree to cooperate only if this time is kept to a minimum. Care should be taken to obtain the required information in this limited time. To do this, a determination of the minimum number of questions needed to accomplish the survey's purpose should be made. Confidentiality. Any questions regarding confidentiality or the Privacy Act system of records should be addressed in the RFP. If there is a promise of confidentiality in the survey, the contractor may have to take special provisions to allow for it. These could include locking up data and only allowing access by authorized personnel. Special provisions regarding the security of the questionnaires might also be needed. Payments to respondents. If there is to be financial compensation to respondents for completing a questionnaire, the offerors should not only be made aware of this fact, but also the reason for it. Following are several reasons (each of which could have different effects on the bids) are: studies have shown that the success of the project depends on the agency providing compensation, or the data collection requires a significant amount of work on the part of the respondents and they should be compensated for their time. In addition, OMB and many Government agencies require a description or justification of any remuneration to persons supplying information. If the success of the survey depends on these payments, approval should be obtained from both the Department and OMB before the contract is signed. Advance notice to respondents. Many surveys require advance notice to the respondents. This is so they can consult their records and documents, thereby 26 making the interview as useful as possible. If this time is needed, the offerors should be made aware of it so they can plan their resources accordingly. Dates for data collection. There may be reasons that specific beginning and ending dates for the data collection must be set. These reasons could be that the particular event being measured will only occur during those times or that the collection is not allowed while the decennial census is being conducted. Make certain the reasons are justified. Moving up the time schedule for a survey is usually not a sufficient justification. Training. It must be determined who will be responsible for training interviewers and preparing the necessary materials. In addition, what specific types of training requirements exist and will the sponsor participate in the training., Response rate. The required minimum response rate should be specified along with the method which will be used to calculate it. Included with this information should be the number of completed questionnaires required and the criteria for defining a completed questionnaire. The procedures for field follow-ups should be stated. These should include any time-of-day and day-of-week requirements along with the number of follow-ups and a requirement that the contractor complete all specified follow-ups, even if the required minimum response rate is met. In addition, there should be a provision that the contractor must exceed the minimum number of callbacks if they are required to meet the minimum response rate. Of course, any additional field follow-ups needed to meet the response rate will have a cost associated with them. This tradeoff should be considered while decisions on the survey requirements are being made. No attempt is made in this report to determine what the minimum response rate should be for every survey; each survey will have factors that will have an effect on this number. The best advice is to check current policy and consult a survey statistician before writing this section of the RFP. Quality control. The quality controls that will be required in the data collection process and the validation requirements must be addressed in the RFP. Who will be conducting these tests must also be specified. It is usually best to indicate that quality control is a necessary part of the survey and let the offerors indicate how they propose to control quality at each stage. Interviewer qualifications. If the interviewers must possess certain qualifications because the survey has special circumstances that require them, they must be specified in the RFP if they are to be used in the evaluation process. These could include expertise in the specific field of study, knowledge of a foreign language, previous data collection experience, or knowledge of survey sampling. Additional questionnaires. If there are to be different versions of the questionnaire , these requirements must be specified along with the reasons for needing them. They could be necessary due to the questionnaire must be tailored to the various regions of the country because of content or other considerations. Examples are asking about citrus fruit production only in the appropriate areas or having a foreign language version of the questionnaire in areas where English is not the predominant language. Tabulations and reports The final products and quality of the completed work should be monitored very closely. To facilitate the monitoring process, certain deliverables should be 27 specifically requested in the RFP. A schedule should be set up and rigidly followed. The following list contains several of the more important deliverables. Progress reports. Progress reports should be required according to an agreed upon plan. The p!an should specify content as well as time of delivery. Draft reports for agency review. The RFP should indicate that a draft of each required task report will have to be submitted for approval before the final report is written and the task is considered complete. Tabulations required. When it is feasible, the format and content of all required tables should be included in the proposals. The RFP should also state whether the contractor will be expected to perform analyses of the data and in what form they must be delivered. Data descriptions. All data supplied should be accompanied by sufficient documentation. For example, the formats of any data tapes must be required and the formula used to calculate any weights must be supplied with the weights. Survey documentation. The RFP should require a complete detailing of the methodology used. This report should include the contractor's impressions of the effectiveness of the survey procedures, suggestions for improved methods in future surveys, and all appropriate technical materials. Included would be the following: copies of all forms; documentation of all (especially sampling and data collection) procedures in all phases; all adjustments for nonresponse and missing data; survey estimates and estimates of sampling errors; full accounting of all data collection results, including response rates and interviewer validation results; coding error rates, along with all quality control procedures used; methodological findings; and a description of the procedures used in handling confidential data. Analytic reports. If the agency wants the contractor to produce one or more reports analyzing the findings of the survey, such a report should be requested in the RFP. Frequently this analysis and the resulting report are done in- house. If this is to be the case, it should be stated in the RFP. Analytic reports should contain a complete analysis of the survey data along with any appropriate conclusions and recommendations. ROLE OF THE PROJECT OFFICER Monitoring functions of the project officer that directly affect the performance of the contractor should be clearly discussed in the RFP. This allows the offerors to know in advance how the project officer intends to monitor the contract. Following are several important points which should be defined. The RFP should list all areas where reviews and approvals will be required. These could include sampling plans, questionnaires, training materials, and data processing requirements. If site visits are planned or meetings are to be scheduled, they should be mentioned in the RFP. The RFP should clearly specify the role of the Project Officer regarding attendance at any training that is planned. Detailing the role of the project officer in the RFP allows the offeror to plan for additional resource requirements and permits a smoother relationship in the performance of the contract. 28 EVALUATION CRITERIA Every RFP must include the criteria to be used in evaluating the technical merits of the offeror's proposal. These will be listed in descending order of points to be allocated to the item and may show the actual number of points. The technical evaluation criteria need to cover the critical elements to be used in determining the acceptability of the offeror and the proposal submitted for the contract survey. Differentiation between acceptable and unacceptable proposals and offerors is made on the basis of these criteria. If a vital element of approach or capacity is not reflected in the evaluation criteria, it may not be possible to distinguish among offerors or their ratings based on the issue. All too often program staff members concern themselves with technical aspects of the survey (that is, sampling, interviewer training, field work, editing and coding) and ignore the need to translate these specifications into the criteria that will be used to evaluate the proposals received. The criteria will be used in the selection process. The survey specifications are the basis on which the offeror provides the proposal that will be the subject for evaluation. Proposals can only be evaluated based on the published criteria. 29 Chapter 4. CONTRACTOR SELECTION PROCESS INTRODUCTION The work to be done under the contract will have been described in the Request for Proposal (RFP) as well as the material and presentation to be submitted by the offeror as a basis for evaluating the response. The criteria and scoring to be used in this evaluation are also included in the RFP, thus giving each offeror the same knowledge about evaluation of the technical proposal. It is in this sense that contractor selection starts when the decision is first made to issue a competitive RFP. By that time, there should have been consideration of the possibility of doing the work in-house or through another Federal agency, of contracting with an 8(a) firm, or of limiting competition to small business firms under the set aside program. The decision by the contracting officer to issue an open competitive RFP instead of taking one of these other options involves decisions and justifications within the agency as to the conduct of the work or some basic characteristics that potential contractors must have. These may relate to performance or capacity requirements or experience or reputation requirements of potential contractors. These characteristics should be part of the specifications included in the RFP. Since the agency has made the decision to go competitive based on specific requirements of the proposed contract as these relate to capacities of potential offerors, it is incumbent upon the writers of the RFP to make this information available to potential offeror organizations. It will be useful to them in making their decisions as to whether to respond to the RFP. The mechanism for doing this is the RFP itself. An adjunct to this that is sometimes used is a Bidders' Conference as discussed in Chapter 3. The thinking that resulted in a competitive RFP should provide the framework of offeror capacities and expected activities performance that will be used to evaluate responses. The remainder of this chapter is based on the premise that the RFP discussed what was being looked for in the contractor selection process. COMPONENTS OF THE SELECTION PROCESS The response to an RFP is submitted in two separate parts, the Technical Proposal and the Business and Cost Proposal. These are evaluated separately. Evaluation of the Business and Cost Proposal is the responsibility of the contracting officer and is done in terms of the appropriateness and documentation of costs relative to resources to be devoted to the effort. The technical proposal is evaluated in terms of the material on project conduct and management, the staff and other resources to be devoted to the project, and the qualifications of the staff. If the RFP was specific in the description of the work to be done, the capacity and experience of the organization and staff will probably be weighted more heavily in the evaluation criteria. If the RFP was more general in the description of the work to be done, the approach to conducting the project proposed by the offeror will probably be weighted more heavily. In any case, the RFP should 31 have specified what the offeror was to submit as a basis for technical evaluation and the criteria to be used. The contracting officer has the responsibility to ensure that the technical evaluation is conducted in a way that satisfies the Federal Procurement Regulations and agency requirements. The program office bears responsibility for the conduct of this review. The guidelines in the regulations governing technical review require establishing a selection process that will insure fair and impartial treatment of all offerors, the selection of sources whose performance is expected to meet the contract requirements at a reasonable cost or price, and avoidance of arbitrary or capricious behavior, inequitable treatment, or undue influence. It is essential that persons involved in the evaluation process have no conflict of interest with this function and that no information concerning the evaluations be released to another individual except one who is participating in the process. The project officer for the contract is responsible for proposing the names of individuals to be on the technical evaluation panel to his/her superiors within the program office. The project officer is expected to be a member of the panel and may serve as its chairperson. The panel is usually composed only of government personnel. Outside evaluators are used only under unusual circumstances, such as when required expertise for evaluation is not available within the government. The project officer should consult with the contracting officer, who bears official responsibility for the overall contracting process, on aspects of the technical review process and current standards and requirements of the Federal Procurement Regulations and agency regulations on technical evaluation. In general, project officers are given a significant amount of flexibility and latitude in establishing panels and review plans and in conducting technical evaluations. However, a list of persons to serve on the technical evaluation panel and a plan for the technical evaluation must be submitted to the contracting officer before the technical proposals are released to the project officer for panel evaluation. To the extent possible, members of the project team should be included on the panel. The contracting officer should attend the initial panel meeting to clarify the basic rules and considerations under which the technical review will take place. Beyond that, the prime requirements are that the process be equitable, that the basis for the evaluation of each proposal be documented, and that the evaluation be conducted in terms of the criteria published in the RFP for contractor selection and the evaluation plan. The purpose is to make the contractor selection process as fair and open to competition as possible. The process is designed so that all potential offerors have the same base of information on which to assess their qualifications for competition for award of the contract, to know the basis on which a selection will be made and to have the selection based on preset criteria. The goal of this approach is to eliminate the actuality and, to the extent possible, the appearance of bias and preselection. In assembling the technical review panel, the project officer should try to obtain a balance of areas of survey expertise and subject matter interest as related to the purpose of the contract and the needs of the agency. For example, this might include experts on survey methodology and sample design and experts representing interrelated content and program areas. Expertise and experience available in other agencies should be used as a resource. Representation of other agencies is also advantageous when there is overlapping program interest. A panel that is biased in composition toward a given expertise may distort the contractor selection in that direction independent of the balance built into the technical evaluation criteria and" plan. A minimum of three persons should be on the panel to allow for more effective 32 discussion and resolution of disagreements. The project officer should arrange for a meeting place where panel members can discuss the merits of each of the proposals in private. It is important that there be continuity over the full process of proposal review and contractor selection. Individuals selected for the panel should be available to review later modifications and additions as well as the initial responses to the RFP. TECHNICAL EVALUATION OF CONTENT AND CAPABILITY The technical evaluation process is designed to carry out the objective consideration of the technical merits of proposals submitted in response to an RFP in terms of the published evaluation criteria contained in the RFP. By the end of the technical review process each proposal will be identified as being either acceptable or unacceptable. A proposal can be determined at the outset to be technically unacceptable without detailed review if it fails to meet a specific expressed requirement of the RFP. As an example, if the RFP requires probability sampling for the planned statistical survey and an offeror proposes only an alternate sampling approach, the proposal can be considered an inappropriate response to the RFP and technically unacceptable. A technical evaluation plan should be prepared by the program office. It should include information for the panel members based on the specific criteria, as included in the RFP, to be used in rating proposals. The evaluation plan should provide more specific guidance for the panel on details to be looked for within an offeror's proposal in scoring it under various elements of the evaluation criteria. The plan will specify the number of points to be allocated to each part of the evaluation criteria. In general, these criteria should deal with the offeror's approach to the contract tasks and the organizational ability to perform those tasks. The evaluation criteria for ability or capacity to perform, which should have been built into the RFP, should include some forms of the following considerations: demonstrated organizational expertise in the subject area, staff availability and their expertise in the subject area, past experience in surveys under contract, ability to meet special requirements, and, availability of needed resources. The references for past surveys under contract should be checked. Before discussing the proposal in a group, each member of the evaluation panel should individually read each proposal and prepare a tentative indication of its strengths and weaknesses and a preliminary score on each evaluation criterion using the rating sheets in the technical evaluation plan. Adequate time should be allowed for this key aspect of the process. After this has been done, the panel members should meet to discuss the relative merits of each proposal and try to arrive at a consensus opinion on the strengths and weaknesses of each proposal and the possibilities for correcting the weaknesses. The evaluators should then individually rate each proposal. The individual scores of the panel members have to be combined into a single ranking of all of the proposals. While a numerical average of the panel members ratings for each proposal is generally used in developing a ranking, this is not a requirement and depends on the rating plan used. The panel then should identify each proposal as acceptable or unacceptable. However, this can not be done by the use of predetermined cutoff scores. The definition of a proposal as acceptable or unacceptable is based on technical consideration of the offeror's proposal in terms of the requirements of the RFP "as assessed, using the evaluation criteria published in the RFP. A determination of 33 unacceptability must be supported by concrete factual statements consistent with the evaluation criteria and indicate that the proposal's deficiencies are so major it could not be brought up to an acceptable level without the equivalent of a new proposal being submitted. The technical evaluation panel members will prepare and sign a report to the contracting officer showing the ranking of the proposals and identifying each as acceptable or unacceptable. The individual rating sheets should be included. A narrative assessment of each proposals strengths and weaknesses must be a part of the report. The report should also include any considerations related to the selection of sources for negotiation and award and any specific issues and questions for discussions. DISCUSSIONS WITH OFFERORS IN THE COMPETITIVE RANGE After the technical evaluation and analysis of the business proposals, the contracting officer will determine which proposals are in the competitive range as a basis for conducting written or oral discussions. Discussions are communications essential for determining a proposals acceptability or to provide the offeror with the opportunity to revise or modify its proposal based on deficiencies or weaknesses noted in the evaluation process. More than one round of discussions may be held by the contracting officer with offerors depending on the situation. The determination for inclusion in the competitive range will be based on price or cost and technical and other important factors as stated in the RFP. The competitive range will include all proposals that have a reasonable chance of being selected for award. In making this determination, the contracting officer may request the technical evaluation panel to review the cost or price data, or request comments from the project officer on labor and other resources needed to conduct the project. Discussions will be held with all offerors in the competitive range. The competitive range may be narrowed after initial discussions by dropping an offeror if a revised proposal is determined to no longer have a reasonable chance of being awarded. All discussions with offerors are under the control of the contracting officer. Offerors are given the opportunity to correct deficiencies, resolve uncertainties and errors, and to submit revised material based on these discussions. However, no information is to be given to offerors in these discussions on other offerors' proposals or the evaluations, or on competitive or estimated prices other than to indicate that the Government considers its price too high. No specific help should be given an offeror in rewriting the proposal. At the end of the discussions with all offerors in the competitive range, each offeror will be notified that the discussions have ended and that they have an opportunity to submit a "best and final" offer by a common cutoff date, including a revision of an earlier offer or a confirmation that a prior offer is a "best and final" offer. These "best and final" technical and business proposals are subject to evaluation on all significant factors by the contracting officer and project officer and, if necessary, a" full technical rescoring and reranking by the technical evaluation panel. At the end of this process, the contracting officer will select for award the proposal that offers the Government the greatest advantage, price, and other factors considered. While the final selection decision is the responsibility of the contracting officer, this' decision is usually made after consultation with the project officer. Since it will be 34 the project officer's responsibility to monitor technical performance, he or she should be part of the assessment of the relative importance of technical quality and price differences. The analysis of business management and price factors is the contracting officer's domain. Although price or cost is one factor in the final decision, whether certain items are allowable and the assessment of the basis for estimating cost is an administrative issue. Usually, whether a contract is to be a fixed-price or a cost-type is specified in the RFP. However, the final decision to use a fixed-price or a cost-type contract is a contracting officer decision and the ability to estimate certain costs is a factor in this decision. THE FINAL AWARD PROCESS .After the successful offeror has been selected, the contracting officer may hold limited negotiations with this offeror. However, these negotiations cannot relate to any factor which could have an effect on the Selection process. It is the Contracting Officer's responsibility to ensure that negotiations do not change the RFP,s requirements nor make any other changes that would impact on the selection process. If there is a material change in requirements because of negotiations, the competition must be reopened to all offerors who were in the competitive range. A significant increase in the offeror's cost proposal may also require reopening of the competition. After any negotiations are held, the final contract document is prepared. In preparing the contract document, including all the agreed upon terms and conditions, the option should be considered of specifically incorporating the offeror's final proposal to avoid any question as to what was offered and accepted. This can be of particular value when the RFP specified only an area of survey concern and not the specific approach to be used. After the contract has been awarded, the unsuccessful offerors will be notified. Unsuccessful offerors can, and often do, request debriefing. The project officer and possibly other members of the technical evaluation panel may be. called upon in a requested debriefing of an unsuccessful offeror. It is at this point that the documentation of the basis for technical evaluation can be of critical importance in explaining why the offeror's proposal was not selected for award and in establishing that the competitive process was appropriately conducted. Unsuccessful offerors may formally appeal a contract award. However, a well documented and unbiased basis for selection will reduce such appeals of the selection process. 35 Chapter 5. POST-AWARD ACTIVITIES INTRODUCTION Following award of a contract, both the contractor and the Government have responsibilities that must be fulfilled. Post- award activities are all activities that occur or should occur once the negotiated contract has been signed by both parties. it is the joint responsibility of the contractor and of the Government to see that work under the contract is accomplished in a timely and proper manner. Government policy and procedures designed to ensure that the contractor and the Government staff understand their mutual responsibilities after the contract is awarded are set forth in the Federal Procurement Regulations. The major elements of post-award contract administration include monitoring of performance (including review and approval of products, services, or reports), contract modifications, approval of subcontracts, property administration, and contract closeout. Aspects of these elements may be performed by various Government personnel, but the two principal persons are the project officer and the contracting officer. It should be made clear at the outset of the contract exactly what the respective roles of these two individuals are with respect to contract administration and monitoring. Typically, the contracting officer, the person who actually signed the contract on behalf of the Government, delegates to the project officer responsibility for monitoring technical aspects of work under the contract. In most instances, a contract identifies tasks, subtasks, schedules, review processes, criteria identifying skills, techniques and methodologies to be used, and a schedule of deliverables. These requirements are usually included in the Request for Proposal (RFP) and are addressed specifically in the contractor's technical proposal, which is often incorporated into the final negotiated contract. The project officer must monitor closely all aspects of work to ensure that it is carried out as specified. In addition to technical and procedural requirements for conduct of the work associated with the survey, the contract should also include performance requirements and a description of measures to be used in determining if certain requirements, such as the specified response rate, have been met. The contract is the basis for monitoring and assessing contractor performance. A project officer should have a complete copy of the final negotiated contract for a survey, should read it completely to ensure that it contains all that it should contain, and should reread it periodically to insure that all of its terms and specifications are being followed as specified. Most post-award responsibilities of the project officer relate directly or indirectly to technical aspects of the work. The role of the project officer is discussed in the following material, which emphasizes activities related to insuring that the product resulting from the contract is of the highest quality possible. 37 GENERAL TECHNIQUES FOR MONITORING CONTRACTOR PERFORMANCE Meetings with the Contractor To the extent possible, there should be regular meetings between the project officer and the contractor. For a large, complex contract such meetings may have been included in the statement of work and travel funds provided for contractor participation. If such meetings are not possible or are only possible on an infrequent basis, conference telephone calls may serve as a reasonable, though less satisfactory, substitute. These meetings provide a forum for structured, though somewhat informal, review of activities and can offer an early alert to technical problems or to slippage of the project schedule. Maintenance of Files Ideally, the same project officer will serve from the inception of the project through its completion. Experience shows, however, that this is often not the case and that project officers may change several times during the course of a project. Thus, it is essential that the project officer maintain complete, organized files pertaining to the project. Included in the files should be the following: all formal contract documents, including the negotiated, signed contract and any modifications to it; copies of all correspondence relating to the project between the contractor and the project officer; notes or minutes of any meetings or substantive telephone conversations with key contractor staff; copies of all progress reports and other materials received from the contractor; copies of or notes on all communications with the contracting officer; and documentation of any other significant materials, including various approvals such as required departmental clearances and Office of Management and Budget approval. Project files are also important as a source of documentation if there is a disagreement between the contractor and the project officer about whether information was communicated, and if so how and when. ADMINISTRATIVE RESPONSIBILITIES OF THE PROJECT OFFICER Review of Contractor Progress Reports and Submissions The project officer should, as soon as possible upon receipt, read and provide comment, preferably in writing, to the contractor on all interim progress reports and other documents received. No matter how often there are conversations between project officer and contractor, they do not replace the formal progress reports that should be required by that contract. These reports are the appropriate forum for the contractor to officially inform the project officer of project accomplishments and problems. Reports should contain sufficient information to determine that work is being accomplished on or ahead of schedule and that it is being completed in a satisfactory manner. It will facilitate the project officer's review, and perhaps even the contractor's preparation, of progress reports if a mutually agreed upon outline is followed for each periodic progress report. Such an outline may even have been specified in the contract. In addition to a description and summary of activities since the last report, topics to be addressed might also include a statement of any problems encountered and their resolution, personnel changes, a summary of expenditures to date, and planned activities for the coming reporting period. When data collection is involved, the contract should also specify provision to the project officer, for review and approval, of all data collection plans; instruments and 38 associated materials, including sampling procedures; letters to respondents; interviewer or respondent instructions for completion of the questionnaire; training materials for data collection; coding manuals; and data processing specifications. If the project officer does not have the requisite expertise to review and approve these materials; one or more technical advisors on the project team should have such expertise and should be asked to conduct the review, giving advice on appropriate action. Should the contractor fail to submit required reports or other deliverables in a timely or technically acceptable manner as specified in the contract, the project officer should notify the contracting officer so that determination of appropriate action can be made. Unless the delay is minor or affects only a small, inconsequential portion of the contract, it will probably be necessary for the contracting Officer to extend officially the contract schedule or terminate the contract completely. Review of Financial Reports The contracting officer will, particularly in the case of cost- reimbursement contracts, often ask the project officer to review the periodic vouchers submitted by the contractor for payment of costs incurred. For a large or complex project, it may be helpful to have the information for the financial reports provided in specified categories. This will facilitate the assessment of whether the project is likely to be completed within the budgeted resources and will be useful if the survey is ever repeated. In addition to a report of the current status of expenditures, it may also be desirable to require projections of expenditures for the coming project period and perhaps for the remainder of the project. Review of expenditure information in conjunction with technical progress reports will enable the project officer to assess likelihood of cost overruns and to take early steps to avoid or minimize them. The project officer must determine that expenditures are in keeping with technical progress before recommending to the contracting officer that payment be made. If any problems are identified, the project officer should inform the contracting officer and effort can be initiated to take remedial action. Modifications to the Contract When the contractor or project officer identifies the apparent need for a change in the contract specifications, they may discuss it informally between themselves, but no deviations from the specifications of the written contract may occur until the contract has been formally modified in writing by the contracting officer. If there is a need to change aspects of the work requirements, deliverables, or schedule for completion of activities, formal, written modification to the contract will be necessary. Approval of Subcontracts It is not unusual for a contractor to engage a subcontractor to complete one or more tasks under a contract. It is important to the Government that the capabilities of the subcontractor be such that a high quality job will result. if it is known that a subcontract will be necessary, it and the proposed subcontractor should be identified before the contract is signed. However, the need for a subcontract is sometimes not identified until after the contract has been awarded. Therefore, the negotiated contract should contain a clause requiring approval by the contracting officer of any subcontract. 39 Before entering into a subcontract, the prime contractor should provide information about the work to be done under the subcontract, the anticipated costs and dates for completion of work, along with identification of the proposed subcontractor and information about why and how the subcontractor was selected. Upon receipt of notification of plans to subcontract, the contracting officer will provide the information to the project officer for concurrence that the proposed subcontractor has the requisite capabilities to perform in a technically acceptable manner. The project officer should monitor the prime contractor's management of work under the subcontract. However, if the work of the subcontractor or the management of the subcontractor by the prime contractor is not satisfactory, the project officer should not take direct action. Instead, the contracting officer should be notified and asked to take appropriate action through the prime contractor. Only under rare circumstances would the project officer engage directly in contact with the subcontractor. Administration of Government Property The contractor will usually have all equipment and other resources necessary for conduct of the work under a contract. However, a contract may specify that the Government will provide certain materials necessary to complete the work. If data collection is involved, this might be a sampling frame based on administrative or other records that are in the possession of the sponsoring agency. It might be access to Government computer facilities, or it might be acquisition of specialized equipment specifically for the work of the contract. The project officer is responsible for seeing that the appropriate arrangements are made for contractor access to, acquisition of, or use of such equipment or materials. The contracting officer is also likely to seek the advice of the project officer on disposition of any equipment purchased under the contract. Closing Out the Contract Upon completion of all work under the contract and provision to the Government of all required reports, materials and information, and review and approval of all specified deliverables by the project officer, the final payment will be made to the contractor. Although contract closeout is the responsibility of the contracting officer, the project officer may be called upon to assist. QUALITY ASSURANCE RESPONSIBILITIES OF THE PROJECT OFFICER It is implicit throughout the following discussion that the project officer will probably not have the full range of technical skills needed to monitor all aspects of the work. As appropriate, other members of the project team should be called upon to participate in post-award monitoring activities. These persons should have the necessary technical knowledge, and familiarity with and understanding of the objectives and specifications of the contract to provide advice to the project officer. Every survey should include a program of built-in quality assurance activities. It is the responsibility of the project officer to ensure that appropriate steps are taken to quantify and document the quality of survey data. Provision should have been made in the contract specifications for inclusion of various quality assurance components, and the contractor's technical proposal should have addressed them. This section of the chapter presents an overview of quality assurances that should, as appropriate, be built into any survey. The discussion is intended to apprise a project 40 officer of the variety of ways in which data quality can be maximized. Not all of these measures will be appropriate or necessary for every survey. In general, large studies require all or most of these quality assurance procedures; smaller, less complex studies may require only some of them. The project officer should be thoroughly familiar with the required and proposed steps to be taken by the contractor to measure data quality and achieve a satisfactory level of quality at all steps from instrument development through data collection, data reduction, and final data tape preparation. If no quality assurance program is specified in the contract the project officer should discuss plans for one with the contractor's project director and agreement should be reached and documented on a step-by-step basis so that there is no room for misunderstanding; a formal contract modification may be made to make the quality assurance program a contract requirement. A key element in assurance of data quality is the active involvement of the project officer at all steps in the data collection and data reduction processes. Active participation by the project officer and other members of the project team will often involve costs for travel and other activities. There should be provision for these costs in the agency's operating budget over the period of the project. The role of the project officer is sometimes specified in the contract itself; the project officer should make clear to the contractor's project director as early as possible the intended nature and extent of this involvement. Following is an enumeration and discussion of quality assurance activities in which the project officer should take an active role. Sample Selection, Questionnaire Design and Data Collection Data collection activities conducted under contract usually require approval by the Office of Management and Budget. It is the responsibility of the project officer to learn about these requirements, determine how they pertain to a particular project, oversee the development of necessary descriptive materials, and shepherd the submission through the review and approval process. Appendix 11 gives an overview of these requirements but does not replace early consultation with the agency reports clearance officer. In addition to reviewing and approving sample selection specifications, interviewer and coder manuals, and training materials, the project officer should attend and, as appropriate, participate in interviewer and coder training. project officer involvement in activities relating to sampling, questionnaire design, interviewer training, and field procedures during the pretest and pilot surveys is particularly important. Sample design and selection are critical survey underpinnings. Questions for a survey must be carefully worded to ensure that what is conveyed to a respondent is what is intended, and responses must be recorded (either verbatim or in predetermined categories) so that the planned analysis can be adequately accomplished. Similarly, the interviewer training and actual data collection activities constitute survey activities in which errors or oversights that occur can negate the success of the entire project. Sample selection. The process of sample selection is critical and should be carefully monitor d by both the project officer and the contractor. Depending on the type of survey and whether the sponsor provides a sampling frame, it may be necessary for the contractor to obtain, create, or refine the frame from which sample units for the survey are to be selected. It is important to assess, prior to selection of the sample, 41 the completeness of coverage of the sampling frame, the currency of the frame and its contents, and the associated problems arising from limitations of coverage and currency. These steps are essential even if the survey is a census of the universe of eligible individuals or other units. Once the frame has been determined to be satisfactory, it is necessary to ensure that the specified procedures are followed in selection of the sample. This should involve a monitoring and review of the selection that takes place at each stage of sampling. If the sample procedures are being carried out incorrectly at one or more stages of selection, biases in survey estimates will probably be introduced. If independent population and subpopulation counts (or good estimates of them) are available from a census or large sample, these should be used to help evaluate the quality of the sample. For example, suppose that a national probability sample of persons age 18 years and older. is selected. Estimates of population counts by age, race, and sex cross- classifications from the sample should be compared to corresponding census counts; any substantial variations could indicate that the sample is not being selected correctly. Questionnaire development. On occasion an agency provides the contractor with a completely developed questionnaire or set of data collection instruments. But even when this occurs, there is usually the need for a pretest of the questionnaire because seldom has it been used exactly as intended in the present effort. More typically, the contractor is responsible for development of the data collection instruments. The advice and experience of subject matter experts are usually the basis for instrument development, and to the extent possible, previously developed questions whose efficacy has been demonstrated should be used. Questionnaires should receive several levels of pretesting, the least formal of which is testing on office staff or friends. This type of testing offers an opportunity to identify difficulty in understanding the intent of the question and question flow. Subsequently, there should be more formal tests of the questionnaires. The project officer should not only review the questionnaires and any accompanying instructions for their administration or completion, but should also test the instructions and questionnaires by administering or completing them. Only in this way can first-hand experience be gained with the instruments and the problems associated with their administration. Interviewer and coder training. The project officer is responsible to monitor training for data collectors in order to be certain that instruction, given is accurate and thorough. If possible, the project officer should attend the training sessions; at a minimum, the project officer can explain the objectives and importance of the study to provide project staff with an understanding of why the data are needed and how they will be used. This is also an opportunity for direct acquaintance with the data collectors and the level of project understanding with which they embark upon their work. Interviewer observation. Throughout the data collection process, as circumstances permit or dictate, the project officer should participate in an interviewer field observation program. Survey firms usually have a program for observation of each interviewer early in their field assignment or for monitoring telephone interviewers throughout the survey. This observation or monitoring is typically done by a field supervisor and is intended to determine how the interviewer presents himself or herself and the survey to respondents, whether procedures are followed as specified (including asking the questions exactly as they are worded on the questionnaire), and what problems are encountered in the field (including efficient use of time in the field and correct field editing of completed data collection instruments). 42 The project officer should try to observe or monitor several interviewers, possibly identifying at the training session those to accompany to the field or otherwise monitor. Interviewers are seldom perfect in their work, but by observing several the project officer will be able to assess strengths and weaknesses in the data collection process. Observers, however, must remember that their role is a passive one and that no active role should be taken during the interview to either converse with or correct the interviewer during the interview itself. Reinterview program. Many surveys have as a component of their quality assurance program a reinterview program in which a subsample of the respondents and nonrespondents are recontacted to validate selected information reported by the interviewers. Reinterview programs are usually developed to check on the work of individual interviewers in order to identify interviewers who are reporting imaginary interviews. For some surveys, reinterviews are used to try to measure response bias and variation. Some survey organizations routinely send respondents a postcard to verify that the interviewer contacted them and the length of the interview. This augments but does not replace a personal recontact with a sample of the original respondents for each interviewer in which a brief reinterview is conducted to verify that the original interview was, in fact, conducted. Effort is made to conduct the reinterview with the initial respondent; typically included are a few basic questions about household or respondent characteristics needed to verify the completeness and accuracy of information about their eligibility for the survey. Response rate. It is important that the project officer monitor closely the response rate being achieved by the contractor. Several aspects of the response rate should be calculated by the contractor on an ongoing basis and should be communicated to the project officer on a regular basis, perhaps weekly. Overall response rates, interviewer-specific response rates, and response rates for key subgroups or characteristics of the study population should be monitored. Problems of failure to locate sample units and to complete interviews or obtain completed questionnaires for sample units must be identified as quickly as possible in the data collection effort so that corrective action can be taken. It is important to achieve acceptable response rates for specific subgroups as well as for the entire population. Poor response rates can lead to substantial nonresponse biases that cannot be estimated. Furthermore, if there are large differences in the nonresponse rates among population subgroups, there may be differential nonresponse biases in subgroup estimates. Consequently, there may be some misleading subgroup comparisons due to differential biases. If there is more than one stage to data collection, such as a screener questionnaire to identify those eligible for a more detailed questionnaire, the response rate for the screener is particularly critical and must be given special attention. The nonresponse rate at the screening stage is part of the survey nonresponse rate. Furthermore, a high screener nonresponse rate could create a substantial bias in the estimate of the size of the target subpopulation. Data Reduction Tolerable error levels for coding and data entry must be established and may have been specified in the contract, but there must be a monitoring system to assure that they are met. These levels are often specified in the contract. As for interviewers, training is very important for coders. The project officer should consider observing at least some of the coder training to ensure that it is comprehensive and clear. 43 Coding. Questionnaires are usually processed in batches through the coding and data entry phases of the survey with a single individual having responsibility for them at each stage. For example, early in a project all of a coder's work is usually checked or verified on a 100-percent basis. Then, if the person is operating within the limit of tolerance for error for the project, this recheck is cut back to a fraction of the completed work. Should it occur that the acceptable error level is exceeded for a batch, the entire batch may then be checked, the coder may be given special retraining if appropriate, and the coder's work is again verified on a 100-percent basis until tolerable accuracy is achieved or the person is removed from the project. A similar approach is used for data entry, but data entry is often verified on a 100-percent basis for the duration of a project. Surveys often include open-ended items that allow the respondent to describe or explain something in his or her own words. Such information is then coded into predetermined categories by specially trained persons. It is common practice to have two persons code the responses to some or all of the open-ended questions on a survey. This is known as independent coding when the second coder or verifier works without knowing how the first person coded the information. When the two coders disagree, there may be adjudication. In highly complex coding, there may be three- way independent verification in which two verifiers independently code a sample of a production coder's work with a majority rule applying when two or more coders agree on a code. Independent coding may be continued throughout a project. For large scale surveys it is not unusual for the contractor to turn over to the sponsoring agency, for totally independent coding, a small percentage of completed records. These should be selected in such a way that they can be considered representative of the completed work done by the contractor's staff. Independent coding is done by agency staff or another contractor using the detailed coding manual (including documentation on coding decisions made on difficult and unusual cases). Any discrepancies are then reviewed to determine their nature, reasons, and impact on the data base and analyses. Data cleaning. Vigilance to ensure high quality survey results must Continue through the data processing phase of the project. To ensure that computer programs are actually doing what they are specified to do, it is prudent to put through the system test data for which the desired values are known through hand or other independent calculation. This set of test data should represent the full range of operations expected of the complete program. Then, if the expected results are not obtained, it is clear that there is at least one problem with the purportedly debugged program. Even though the contractor may think the programs are functioning, it is wise for the project officer to make certain that it is indeed the case that only valid codes are read and that data manipulation is occurring properly. It is important to be certain that only the admissible codes are contained for each item on a survey record. However, it is possible for errors to be present on the final data tape. In addition to checking that only reasonable, allowable codes are present for each item, there are usually between item checks made to test for consistency within a subject's data record. For example, a medical procedure code for hysterectomy can be a valid code in studies of hospitalizations, but such a code should not be valid if the person involved is a male. Such a combination of codes on a record requires checking back to determine the source and nature of the error. 44 Data Documentation The project officer should be certain that complete documentation of all sample selection, data collection, reduction, and processing activities is created, reviewed, and kept up to date so that it can be provided with the data tape or tabulations that are received from the contractor. This documentation should include information about sample design specifications, sample frame construction, sample selection, calculation of respondent weights, field procedures for initial respondent contact, follow-up of nonrespondents, and out-of-scope cases. Any and all decisions about coding, resolution of apparent inconsistencies in data items for a given respondent, imputation for missing data, and other nonresponse adjustments, as well as full documentation of all codes used in the processing of data and preparation of the final data tape, must be documented and made available to anyone who will be involved in analysis, reanalysis or review of data from the survey. The data tape received from the contractor should include "flags" to indicate any imputed values so that analysts will know which data were actually reported by respondents and which were statistically derived. Without this information, along with documentation of how sampling weights and estimation procedures are derived and used, the value and utility of the survey data will be greatly diminished. Data Analysis If the contractor is responsible for data analysis, care should be taken by the project officer that there is a good understanding of the specific tabulations to be provided by the contractor. The RFP for a large, complex survey often includes either actual or representative table shells for tabulations to be provided by the contractor. Manipulation of large and complex data bases is a difficult and costly task and provision should be made well in advance for it. In addition to the tabulations to be produced, attention should be given to the analytic procedures and tests to be used in data analysis. If certain assumptions (for example, a random sample) must or can be made about the selection of the sample in order to use existing software packages, it is important to be sure that the requisite assumptions are reasonably well met by the sample. It is recognized, however, that for some purposes a systematic random sample may be equivalent to a simple random sample. Appropriate statistical tests to be applied in determining whether differences between various estimates are significant should be agreed upon in advance if the contractor's staff is to perform analysis of the data. If the contractor performs data analysis, information on testing procedures used should be included in the final documentation for the project provided by the contractor. 45 Appendix 1. INFORMAL SURVEY OF AGENCIES AND CONTRACTORS CONCERNING CONTRACTING FOR SURVEYS INTRODUCTION When the Subcommittee on Contracting for Statistical Surveys set out to do its work, the members believed it would be important to solicit facts and judgments from persons who were directly involved both from the point of view of Federal agencies and from the point of view of contracting firms. The Subcommittee realized that, within its frame of reference, it could not field a formal statistical survey of Federal agencies and contractors concerning the broad spectrum of problems encountered in contracting for statistical surveys. On the other hand, the Subcommittee could and did adopt some procedures which would assure then that its report reflected more than just the views of its members and their immediate contacts. It should be emphasized that the results are not the careful findings one would encounter in a scientific sample survey. They are the considered judgments of a selection of individuals who are personally involved in the contracting process both from the Federal agency and private contractor point of view. The Subcommittee devised two interview schedules - one for representatives of contracting firms and one for representatives of Federal agencies which use contracts for statistical surveys. Respondents were usually groups of individuals who were interviewed simultaneously; in a typical agency interview, for example, a subject matter specialist, contracts expert and administrative official would be present. The agencies were selected first and then a list of their contractors was obtained. The sample of contractors was selected from that list. This procedure offered some assurance that the two groups of interviewees were both concerned with similar Federal contracts, although from different points of view. All the contractors were engaged in the competitive procurement process with the exception of one contractor selected from those largely involved in noncompetitive procurements. For the competitive contractors, the first step in the sample selection process was to review the contract awards of the selected agencies in the statistical contracting area. All contracts that seemed to be within the area of review were identified. Next, the sample was drawn and the contracts were reviewed more carefully to ensure that they were consistent with Subcommittee judgments regarding statistical contracts. For the non-competitive contractor group, a list of such contractors was obtained. The list was reviewed by the Subcommittee. Most were determined not to engage in statistical contracting. One contractor was selected from the remaining few based on Subcommittee members' knowledge of the contractors and the type of work they typically performed. Interviews were conducted by Subcommittee members. The interview schedule was comprised mostly of open-ended questions which would elicit principally qualitative information. Results were compiled at the Office of Management and Budget (OMB), 47 and Subcommittee members used the results as background information in preparing this report. FEDERAL AGENCY RESPONSES Eleven interviews were ultimately conducted with Federal agencies, two in the nature of pretests. The results described below do not always pertain to the same number of agencies and contractors because the pretest schedule differed in some respects from the schedule finally adopted and because the interviews were unstructured. Seven of these agencies permit contractors to use subcontractors, although some restrictions may be placed on the extent of subcontractor involvement. The number of statistical survey contracts awarded by these agencies during fiscal year 1977 and fiscal year 1978 ranged from a low of 2 in one agency to a high of "25 or more in another." Four agencies used cost-type contracts; one used fixed-price contracts; and three used both. Contract amounts varied between $40,000 for 1 year to $20 million over a 3-year period. Developing RFPs and Soliciting Bids Typically, the first step for agencies is to develop a Request for Proposal (RFP) which provides prospective contractors with information concerning the specifications of the proposed survey. Usually, no formal guidelines or checklists are used in preparing the RFP. On the other hand, the respondents in the agencies agreed as to features which should appear in all RFPS. More than half of the interviewees stated that the following items should appear in RFPS: proposal evaluation weights, population to be studied, level of effort, key content variables, data collection method, sampling frame, required response rate, analysis of data, and sampling design. Items such as data processing procedures, sampling reliability, level of response, questionnaire development, trade- offs between cost and quality, and quality control procedures were believed to be less important. Sometimes agencies consult OMB concerning the contents of a survey RFP before issuance; sometimes they ask other agencies for review and comment. The availability of the RFP is usually announced in the Commerce Business Daily as a formal notification to potential contractors. Advertisements may also be put in trade papers or announcements placed in the Federal Register. Mailings are also made to lists derived from past agency experience or from the expertise of persons in the industry. A few agencies utilize sources-sought announcements as a way to stimulate offeror interest. Of ten agencies make use of Bidders' Conferences to explain the RFP and answer technical questions. Sometimes these conferences result in changes, or clarifications in the RFP. The number of proposals received after solicitation of bids varies widely among the agencies and by the specific subjects of the RFPS. Some receive only three or four proposals for each RFP, others may receive as many as 30 proposals. The size and cost of the contemplated survey are often major factors in determining the number of proposals offered: generally, the bigger the size and cost, the greater the number of proposals. 48 Evaluating Proposals Agencies are governed by Federal procurement laws and regulations in evaluating proposals. The following results are not a description of the legal procedures but rather an encapsulation of the processes agencies generally go through. Agencies evaluate first the technical aspects of proposals and then the business aspects. Selection committees are formed to evaluate and choose the best proposal. The technical review committee usually consists of three to eight people and includes persons with backgrounds in statistical surveys and contracting. Actual costs are usually not known by the committee. Some agencies allow personnel from outside the agency to participate in these committees, particularly when special expertise is needed. The business evaluation is usually conducted by contract and procurement personnel. This process consists of a review for appropriateness and consistency of financial resources and project organization. Following the initial evaluation of proposals, offerors may be asked to revise their proposals based on results of the initial evaluation. A final evaluation is then performed. In the final evaluation, rating sheets are completed for each technically acceptable proposal. The contracting officer determines the official competitive range. The finally selected proposal must always be acceptable on technical criteria but need not be the least expensive. From issuance of the RFP to awarding of the contract requires at least 60 to 90 days, often longer. The final selection process can take at least 30 days. All agencies suggested that the length of time for the entire process is too long. They suggested eliminating certain procedures to expedite selection of the contractor. Offerors not selected may ask for a debriefing, at which time unsuccessful offerors can learn about the deficiencies in their proposals. Few of the agency respondents had experienced any protests. Monitoring and Other Considerations A project officer, who is usually a person with academic training and job experience in surveys, manages the survey for the life of the contract. Some agencies report problems with turnover in project officers. Agencies generally cited some problems in the execution of survey contracts. Some contractors have not performed well; others have needed more time to complete the job; some difficulties have been encountered with response levels and deciding when to stop seeking survey responses. A few agencies require progress reports and some send staff for site visits. For the most part, however, feedback concerning contractor performance is obtained informally. Most of the agencies stated that they perceived no ethical problems in contracting. Of those reporting ethical problems, one had a staff member who had previously worked for a company specializing in the survey subject. Another expressed the fear that technical personnel may sometimes tend to reveal too much information arising from the confidential evaluation process. 49 CONTRACTOR RESPONSES Eleven interviews were conducted with contractors, two of which were pretests. The interviewees were eight commercial statistical firms and three nonprofit institutions. Most were engaged in a broad range of statistical work. Of the eleven contractors that were interviewed, five specified the professional staff available at their organizations. Most of their staffs included statisticians, computer systems analysts, programmers, sociologists, and support staff. Psychologists, economists, and market researchers were also available full time on some staffs. One of the contractors stated that key personnel are maintained in-house but extra staff are hired for contracts. Sources of Information on RFPs Almost all contractors consulted the Commerce Business for RFPS but all have other sources of information as well such as personal and professional contacts. Most considered sources other than CBD to be their most productive, particularly the personal and professional ones. Several contractors expressed reservations about relying solely on announcements in CBD, saying that they rarely win contracts they first heard about in CBD. Level of Specificity in RFPs Contractors had much to say about the level of specificity in RFPS. They noted that "the level varies widely among agencies. Some believed that contracting agencies make unrealistic specifications; others said ambiguity or overspecification in RFPs reflect a lack of sophistication on the part of the RFP writer. A few stated they were very comfortable with either very general or very specific RFPS but the ones in between proved most troublesome. Different levels of specificity were believed appropriate for different agencies. Agencies with a good technical staff could write a detailed RFP (for example, including specifications of sampling reliability, specific questions to be asked, and required response rate); those without such staff could write only very general RFPS. Levels of specificity were also related to the kinds of studies. Some thought evaluation studies needed very general requirements; studies where a specific set of data is sought should have stricter requirements. A frequent complaint was that the RFPs did not specify the level of effort desired by the agency. As to specific items to be included, there was a general desire to have the RFP specify the population, the key variables, and to a lesser extent the response rate. Other factors should be left to the offeror, particularly sampling design, data collection method, data processing procedures, quality control, and analysis of data. Improvements to RFPs Many contractors suggested that RFPs should be written more clearly. Several wanted a clear statement of goals and the amount of money available. Others suggested that the level of effort desired should be specified. A few suggested that outside experts be hired to write RFPS. A few mentioned that time constraints were a problem; the time limitation on preparing proposals often hurts small contractors especially. Other suggestions made for improving RFPs included making RFPs more consistent among agencies and giving the agency and the contractor more flexibility to make 50 changes during the course of the contract. A few contractors stated they were reluctant to bid on fixed-price contracts. A number of suggestions and comments were made regarding whether the sponsoring agency or the offerors should provide various technical aspects of a survey. The majority of the contractors preferred that the RFP state the target population and the key variables to be studied. Several contractors also suggested that the level of effort (anticipated budget) be included in the RFP. Most contractors preferred that the offerors specify the particular questions to be asked, sampling frame, sampling reliability, data collection method, data processing and quality control procedures, and analysis of the data. Problems with Inadequate RFPs Most contractors stated they have had problems with inadequate RFPS. One major problem was unrealistic expectations, learning that the agency actually wants much more than was implied in the RFP. Problems were thought to arise from hasty writing or authorship of the RFP by someone who did not understand the technical issues. Instances were pointed out where precision requirements were incorrectly specified. This can lead to substantial problems in comparing proposals. Other problems have been lack of clarity and lack of understanding of confidentiality issues. In one instance, a contractor was forced to hire a lawyer to educate agencies on issues of confidentiality. Bases for Decisions to Submit Proposals The two most important criteria determining whether contractors submitted proposals were expertise in the subject field and perceived chance of winning the contract. Other factors were expected profitability, size of contract, availability of staff, and the amount of competition, including whether the firm had a head start or inside track. The nonprofit institutions gave "interest in the topic" as a major reason to bid on a contract. More than half of the contractors did not respond to a question concerning the types of professional staff they had available; others who did respond said they had a full staff or could easily get the services of any staff they needed. Reactions to Bidders, Conferences Almost all of the contractors stated that they always attended Bidders' Conferences. They did so mainly to assess the competition and to make sure the competition did not get any advantages over them. A few felt the conferences were occasionally useful in clearing up ambiguities, but one noted that offerors are often reluctant to ask the real questions lest they help competitors. Others stated that Bidders' Conferences were most useful for large technical studies. Preparing Proposals No strategy was consistently applied by contractors in preparing proposals. Replies on this topic tended to be vague, emphasizing case-by-case basis or outdoing the competition. Almost half said they tried to submit a technically superior proposal. Cost estimates for preparing proposals ranged from a low of $1,000 to high estimates of $10,000 and even one of $30,000. Several mentioned that their dollar figures did not take into account time spent by staff. A few gave costs in terms of percentage of the final contract, such as 5 percent of expected dollar amount. Costs of preparing proposals could be reduced by improving RFPs by making them more explicit about capabilities and qualifications. Some contractors suggested 51 limiting the length of proposals or eliminating unnecessary and redundant demands from RFPS. One suggestion was for a two-stage bidding process. The first stage would be open to all offers, who could submit a general description of their proposal and a list of qualifications. The agency would then select the most promising of these for a second stage and invite detailed proposals. This would save money and reduce the number of detailed proposals to be reviewed. Other suggestions were for more qualified project officers and for greater use of sole-source contracts when there is clearly a front running firm. Agency Selection Procedures Most contractors believed that agency selection procedures did not generally result in selection of the best proposal. The primary reason given was that, the winning proposal is not selected primarily on technical grounds; too much emphasis is put on cost. Contractors attributed this to inexperienced or incompetent agency personnel. Contractors felt that lowest offerors were often not well qualified but that taking the lowest offeror was a safe course for agencies to follow. The selection of the winning proposal was not believed by the contractors to revolve around technical issues because agencies judge proposals on the reputation of the firm and its staff. They felt insufficient time is allowed for adequate technical judgment and that agency staff have poor memories, repeatedly selecting contractors who have done poorly in the past. There was some difference of opinion among the contractors between those who believe a proposal should be judged on technical grounds solely and those who believe more emphasis should be placed on the firm's ability to perform. A few of the contractors felt the best proposal was not generally selected because of the alleged practice of "wiring" contracts, where the agency knows beforehand which firm will get the contract. Improvement in Selection Procedures Many suggestions focused on getting more technically qualified people to evaluate the proposals within the agencies. Some stated that the selection procedures will be improved when the agencies have competent people to write and evaluate the proposals. Suggestions were made for more peer review or for specification of the methodology of selection. Debriefings Most contractors indicated that they had asked for a debriefing on a contract not awarded to them. Most said the debriefing was useful, especially if they could compare their proposal to the winning bid. One contractor said the debriefings were good for indicating what areas of the proposal were weak, what the agency was really looking for and how the procurement process works. Another contractor noted that agencies are occasionally defensive at debriefings, especially when minority firms are involved. Ethical Considerations Several contractors stated that, while the selection system is generally clean and fair, some contract rigging or "wiring" exists. A number mentioned problems of confidentiality. A few were concerned with whether their proposals were kept confidential during the evaluation process; they believed there were occasions when they had not been awarded the contract, but that their ideas had found their way into the winning proposal. 52 Another confidentiality problem concerns survey respondents' rights to privacy. Contractors expressed concern about being able to maintain confidentiality of data collected, some contractors noted that agencies lack experience in knowing how to use such data and require education by contractors. Another issue was sole-source contracts. Some contractors believed government regulations should be changed so that an agency can award follow-on sole-source contracts to firms with which it is well satisfied. Other ethical issues raised were the assumption that small firms cannot finance a large job and a bias against g(a) firms. An unethical practice on the part of the agencies was asking the offeror to propose alternatives to the RFP specifications then rewriting the RFP to use these alternatives. Other Considerations Among the other comments made by contractors were complaints concerning the lengthy OMB clearance process as well as other long delays. Some of them felt the length of time it takes to get questionnaires cleared was an ethical problem. Others complained of too much irrelevant information being required on proposals. some contractors felt the government should consolidate the procurement regulations of the various agencies. Others complained that year-end contracts were not really competitive because time constraints precluded careful evaluation, that references on proposals were not checked, and that many contractors defined person-years differently which led to problems in comparing costs. 53 Appendix II. OFFICE OF MANAGEMENT AND BUDGET REQUIREMENTs: Information Collection Budget and Clearance INTRODUCTION Almost all Federal agencies are required under the Paperwork Reduction Act of 1980 (P.L. 96-511)1 to obtain approval from the Office of Management and Budget (OMB) each time they propose to collect or sponsor, under a contract or other agreement, the collection of identical information from more than nine respondents. In addition to main data collection, activities related to development or testing of data collection plans are also subject to OMB review and approval if identical information is sought from more than nine respondents in either a formal or informal manner. The document to be submitted foe OMB approval and the time of submission will depend in part upon the agency practice. The development and shepherding through the applicable channels of the request for approval is one of the most important and demanding responsibilities of the project officer on a contract. Due to variations in agency requirements and policies pertaining to OMB clearance, it is especially important to determine them at the earliest possible stage in development of a project. Documents that require approval under the Paperwork Reduction Act include the following: - report forms, application forms, questionnaires, or interview guides for in person or telephone surveys. - orders, regulations or other directives that include requirements for respondents to provide information or maintain records to be used or made available for use in the collection of information. - requests for answers to identical questions which are addressed to respondents by telephone, telegraph, form letters, information circulars and other devices. - any supplementary documents involved in these reporting or recordkeeping requirements, such as instructions or covering letters or introductory statements. A number of standards have been developed and promulgated relating to surveys, including standards for statistical surveys, publication of statistics, and definitions for certain items; these must be adhered to as they apply to a particular survey. Early in the development of a survey, the project officer should discuss the nature of and plans for the survey with the agency reports clearance officer. This will provide an opportunity to become familiar with the most recent directives pertaining to OMB clearance and to acquire an understanding of the process as it will apply to the particular survey. A list of agency clearance officers is available from the Office of ___________________________ 1This legislation supersedes and extends the Federal Reports Act of 1942. 55 Information and Regulatory Affairs at OMB if there is any difficulty in identifying an appropriate agency official with whom to consult. INFORMATION COLLECTION BUDGET A project officer should also be certain that provision for a particular project is made in the agency's Information Collection Budget (ICB). The ICB is an annual, comprehensive submission from each agency to the Director of the Office of Management and Budget describing each existing and new information collection project proposed to be implemented or continued in the succeeding fiscal year.2 In the case of a project that has not been included in the agency ICB, it will be necessary to determine the implications for project timing and scope. CLEARANCE Types of Clearance When the Request for Proposal (RFP) prescribes the data collection plan (that is, the information to be collected and the method of collection), OMB permits submission of the plan for OMB approval prior to issuance of the RFP or signing the contract. Agency contracting policies vary with respect to insistence on receipt of OMB clearance prior to actual signing of a contract. But for large data collection projects, it is wise to obtain OMB approval before the contract is signed. Contracts that include data collection activities should contain a special provision in the RFP and in the final negotiated contract that no data may be collected without receipt of OMB approval. If data collection is required by or implicit in the statement of work for the contract, the project might be submitted for preliminary (or concept) approval from OMB prior to signing of the contract. This can obviate some problems later, after work on the project has begun, and it should facilitate approval of subsequent OMB clearances for the Project. A survey that has a pretest designed to help make decisions about final questionnaire content or survey methodology will usually require separate approvals for the pretest and the final data collection plan. Thus, up to three separate clearances may be required for a survey and it is not unusual for a survey to require two separate clearances. Timetable for Clearance Clearance is a lengthy process and ample time should be allowed for it. Several agency and contractor representatives interviewed by the Subcommittee expressed ___________________________ 2Upon completion of its review of each department's ICB request, OMB transmits a "Passback" which gives a total information collection allowance of respondent burden-hours for the department for the fiscal year. This passback may disallow specific projects. It may also contain specific suggestions for elimination or reduction of respondent burden. In providing this passback detail, OMB is not committing itself to providing Paperwork Reduction Act approval for any of the items contained in the ICB request; Paperwork Reduction Act decisions will be based upon the merits of each individual request for approval. 56 their frustrations about the time required by the OMB approval process and mentioned additional expenses resulting from delays in the OMB clearance process. The agency clearance officer can provide information about how long it can be expected to take for a particular survey. In general, at least 2 weeks should be allowed for each office that must handle and review the clearance request within the agency. Under the Paperwork Reduction Act, the Director of OMB is given 60 days after receipt of a proposed information request to notify the agency involved of the decision to approve or disapprove the request and is to make such decisions publicly available. However, the Director may determine that a request submitted for review cannot be reviewed within 60 days, in which case after notice to the agency involved the Director may extend the review period for an additional 30 days. If the Director does not notify the agency of an extension, denial or approval within 60 days (or if the review period has been extended for an additional 30 days and notice of action is not given within the time of the extension) a control number is to be assigned without further delay, approval may be inferred, and the agency may collect the information for a period of not more than 1 year. A prudent project officer will allow for the possibility of lengthy delays in obtaining OMB approval for a project. Delays do occur from time to time and when a contractor is involved there can be significant problems that arise at such a time if the project officer has not had a contingency plan. Such a contingency plan should be built into the contract. One way to handle this problem is to write a phased contract, in which certain activities are clearly identified to be done prior to receipt of OMB clearance and others are to be done only after receipt of OMB clearance. Another method, which can be used with fixed-price contracts, is to include provisions that: (1) no additional sums are authorized because of delays in clearance, and (2) the duration of the contract is day- for-day extended for delays encountered in clearance. The program contracts office can provide information on how each agency handles this type of provision. Clearance Submissions There are formal specifications for the information to be provided in support of a request for approval of data collection plans. The format for the supporting statement, as specified by OMB, must be followed in writing a clearance request. Clarity and brevity should be relied upon in preparation of the clearance supporting statement, with lengthy detailed description and discussion reserved for inclusion in attachments. It is the responsibility of the project officer to prepare the OMB clearance request, submit it for approval, and answer any questions that are raised by reviewers. Once a contract has been let, the project officer may wish to have the contractor assist in preparation of the description of survey procedures to be used; in any case, the project officer bears full responsibility for reviewing any material prepared by the contractor for completeness, correctness, and adequacy. Often it is helpful to have the reports clearance officer review the clearance package in draft prior to formal submission. At an early-stage in the project, there should be contact with relevant statistical or other programmatic offices with subject matter related programs and experience. This,will usually ensure proper coordination with other projects in the subject area and will usually facilitate clearance. Such contacts. should be documented in the clearance submission. 57 In developing a timetable for the project, a realistic period should be allowed for each stage in preparation for and conduct of data collection activities (including follow-up of nonrespondents and recontact of respondents for any reason). Some slippage of schedules inevitably occurs and should be allowed for in determination of a requested expiration date. If it becomes clear that data collection cannot be completed by the expiration date for OMB approval, a request for an extension should be submitted to OMB at least 60 days before the existing approval is due to expire. Once the project officer has submitted the clearance request through channels to the appropriate clearance officer, he or she should make every effort to remain informed about its progress through the review process. This will enable the project officer to give the contractor current and correct information about the status of the review and will hopefully keep the contractor from trying independently to make contacts in the clearance chain - something the contractor should never do. Project officers, too, should make all inquiries about projects in the review process through proper channels. AFTER RECEIPT OF APPROVAL Once a survey has received OMB approval, the approval number and the expiration date for the approval must appear in specified locations on the approved documents. Finally, a number of copies of the final printed form(s), transmittal letter, instructions, and any other document sent to each respondent must be provided for the official clearance files. Before a material revision or change is made in an approved information collection, a formal request for revision must be submitted to OMB for approval Changes in an information collection requiring approval include the following: any increase in the kind or amount of information sought; any increase in coverage; any increase in the timing or frequency of reporting; any change in the sample design or collection method; or a change in the purpose for which the information is collected or required to be maintained. 58 Reports Available in the Statistical Policy Working Paper Series 1. Report on Statistics for Allocation of Funds GPO Stock Number 003-005-00178-6, price $2.40 2. Report on Statistical Disclosure and Disclosure-Avoidance Techniques GPO Stock Number 003-005-00177-8, price $2.50 3. An Error Profile: Employment as Measured by the Current Population Survey GPO Stock Number 003-005-00182-4, price $2.75 4. Glossary of Nonsampling Error Terms: An Illustration of a Semantic Problem in Statistics (A limited number of copies are available from OMB.) 5. Report on Exact and Statistical Matching Techniques GPO Stock Number 003-005-00186-7, price $3.50 6. Report on Statistical Uses of Administrative Records GPO Stock Number 003-005-00185-9, price $5.00 7. An Interagency Review of Time-Series Revision Policies (A limited number of copies are available from OMB.) 8. Statistical Interagency Agreements (A limited number of copies are available from OMB.) 9. Contracting for Surveys (A limited number of copies are available from OMB.) Copies of, these working papers, as indicated, may be ordered from the Superintendent of Documents, U.S. Government Printing Office, Washington, D.C. 20402
Page Last Modified: April 20, 2007 | FCSM
Home Methodology Reports |