Federal Committee on Statistical Methodology
Office of Management and Budget
FCSM Home ^
Methodology Reports ^

 

  Statistical Policy Working Paper 9 - Contracting for Surveys


 

 
Click HERE for graphic.
 

 

                MEMBERS OF THE FEDERAL COMMITTEE ON

                      STATISTICAL METHODOLOGY



 

                            (June 1983)

 

 

Maria Elena Gonzalez (Chair)

Office of Information and Regulatory Affairs (OMB)

 

Barbara A. Bailer

Bureau of the Census (Commerce)

 

Norman D. Beller

National Center for Education Statistics (Education)

 

Yvonne M. Bishop

Energy Information Administration (Energy)

 

Edwin J. Coleman

Bureau of Economic Analysis (Commerce)

 

John E. Cremeans

Bureau of Industrial Economics (Commerce)

 

Zahava D. Doering

Defense Manpower Data Center (Defense)

 

Marie D. Eldridge

National Center for Education Statistics (Education)

 

Daniel H. Garnick

Bureau of Economic Analysis (Commerce)

 

Charles, D. Jones

Bureau of the Census (Commerce)

 

Daniel Kasprzyk

Bureau of the Census (Commerce)

 

William E. Kibler

Statistical Reporting Service (Agriculture)

 

Thomas Plewes

Bureau of Labor Statistics

 

Raymond C. Sansing

Internal Revenue Service (Treasury)

 

Fritz J. Scheuren

Internal Revenue Service (Treasury)

 

Monroe G. Sirken

National Center for Health Statistics (Health and Human Services)

 

Wray Smith

Energy Information Administration Energy

 

Thomas G. Staples

Social Security Administration (Health and Human Services)

 

 

 

 

 

           OFFICE OF INFORMATION AND REGULATORY AFFAIRS

 

                 Christopher DeMuth, Administrator

 

            Thomas D. Hopkins, Deputy Administrator for

                Regulatory and Statistical Analysis

 

                  Maria E. Gonzalez, Chairperson

           Federal Committee on Statistical Methodology

 

                              PREFACE

 

The Subcommittee on Contracting for Statistical Surveys was formed

to review current practices by agencies in contracting for surveys

and to make recommendations for their improvement.  This working

paper, the report of that Subcommittee, addresses the following:

technical expertise needed in statistical contracting, preparation

of the Request for Proposal, contractor selection, and post-award

activities.

 

While the report is intended to be useful to agencies that may

utilize contracting for surveys, a broader audience may find the

report of interest.  Some" of the material should be useful as a

supplement to the existing training provided, agency project

officers.  Seminars and meetings will be organized to discuss the

findings of this subcommittee with Federal agency personnel

involved with contracting for surveys.

 

The working paper was prepared by the members of the Subcommittee

on Contracting for Statistical Surveys, Federal Committee on

Statistical Methodology.  The Subcommittee was chaired by Thomas G.

Staples, Social Security Administration, Department of Health and

Human Services.  As a subcommittee report, this document does not

necessarily represent the views of the Office of Management and

Budget.

 

 

 

 

 

                  MEMBERS OF THE SUBCOMMITTEE ON

                CONTRACTING FOR STATISTICAL SURVEYS

 

 

Thomas G. Staples, Chairperson

Social Security Administration

Department of Health and Human Services

 

Henry P. Brehm

Social Security Administration

Department of Health and Human Services

 

David W. Chapman

Bureau of the Census

Department of Commerce

 

Maria E. Gonazalez

Office of Information and Regulatory Affairs

Office of Management and Budget

 

Nancy D. Pearce

National Center for Health Statistics

Department of Health and Human Services

 

Richard J. Pratt

Bureau of Labor Statistics

Department of Labor

 

Margaret Weidenhamer

Statistical Reporting Service

Department of Agriculture

 

                               (ii)



 

 

 

 

 

                         ACKNOWLEDGEMENTS

 

 

This report represents the collective effort of the Subcommittee on

Contracting for Statistical Surveys.  Although the subcommittee

members reviewed and commented on all parts of the report, the

following specific individuals were responsible for preparing the

chapters:

 

     Chapter   Authors

       I       Henry P. Brehm

      II       David W. Chapman

     III       Margaret Weidenhamer/Richard J. Pratt

      IV       Henry P. Brehm

       V       Nancy D. Pearce/Thomas G. Staples

 

     Appendix

 

      I   J. Timothy Sprehe

     II   Nancy D. Pearce

 

 

In addition to the subcommittee members, special recognition goes

to the Federal Committee on Statistical Methodology, whose

membership identified the need for this report.  Portions of

several committee meetings were devoted to feedback to the

subcommittee as the report was developing; individual members of

the committee provided detailed oral and written comments.  In

addition, special recognition goes (1) to the contractor and

Federal agency personnel who reviewed draft copies of the report

and (2) to the contractor and Federal agency representatives who

participated in the surveys of sponsoring agencies and contractors. 

Also, several individuals who made significant contributions to the

report participated in the early stages of the subcommittee's work. 

These include Thomas B. Jabine, Leon Gilford, Barbara A. Bailar,

Bette S. Mahoney, Henrietta Hyatt, David B. Orr, Irene C. Montie,

J. Timothy Sprehe, and Corrinne B. Lennox.  Jacob Deutch and David

Arnaudo participated in the surveys of agency and contractor

representatives.

 

Among the subcommittee members special recognition goes to Nancy D.

Pearce for organizing the draft and final versions of the report

and to Nancy D. Pearce and Henry P. Brehm, for pulling together a

compact version of the report for the 1982 proceedings of the

American Statistical Association.

 

Maria E. Gonzalez, on numerous occasions, chaired meetings and

performed other responsibilities when the chairperson was absent.

 

                         Thomas G. Staples

 

 

                               (iii)

 

 

 

 

 

 

 

 

                         TABLE OF CONTENTS

 

Chapter 1.      Introduction. . . . . . . . . . . . . . . . . . . 1

 

     Background and Purpose . . . . . . . . . . . . . . . . . . . 1

     The Chapters to Follow . . . . . . . . . . . . . . . . . . . 4

 

Chapter 2.     Technical Skills Required in Contracting for

               Surveys. . . . . . . . . . . . . . . . . . . . . . 7

 

     Introduction . . . . . . . . . . . . . . . . . . . . . . . . 7

     Basic Types of Skills Required . . . . . . . . . . . . . . . 8

     Technical Input at Various Stages of Contracting for 

         a Survey. . . . . . . . . . . . . . . . . . . . . . . . 12

     Some Technical Aspects Associated with Current Practice. . .14

 

Chapter 3.     Considerations in the Preparation of Requests for

               Proposal . . . . . . . . . . . . . . . . . . . . .17

 

     Introduction . . . . . . . . . . . . . . . . . . . . . . . .17

     Initial Considerations . . . . . . . . . . . . . . . . . . .17

     Possible Contents of Requests for Proposal . . . . . . . . .20

     Role of the Project Officer. . . . . . . . . . . . . . . . .28

     Evaluation Criteria. . . . . . . . . . . . . . . . . . . . .29

 

Chapter 4.     Contractor Selection Process . . . . . . . . . . .31

 

     Introduction . . . . . . . . . . . . . . . . . . . . . . . .31

     Components of the Selection Process. . . . . . . . . . . . .31

     Technical Evaluation of Content and Capability . . . . . . .33

     Discussions with Offerors in the Competitive Range . . . . .34

     The Final Award Process. . . . . . . . . . . . . . . . . . .35

 

Chapter 5.     Post-Award Activities. . . . . . . . . . . . . . .37

     Introduction . . . . . . . . . . . . . . . . . . . . . . . .37

     General Techniques for Monitoring Contractor Performance . .38

     Administrative Responsibilities of the Project Officer . . .38

     Quality Assurance Responsibilities of the Project Officer. .40

 

Appendix I.    Informal Survey of Agencies and Contractors

               Concerning Contracting for Surveys . . . . . . . .47

     Introduction . . . . . . . . . . . . . . . . . . . . . . . .47

     Federal Agency Responses . . . . . . . . . . . . . . . . . .48

     Contractor Responses . . . . . . . . . . . . . . . . . . . .50

 

Appendix II.   Office of Management and Budget Requirements:

               Information Collection Budget and Clearance. . . .55

     Introduction . . . . . . . . . . . . . . . . . . . . . . . .55

     Information Collection Budget. . . . . . . . . . . . . . . .56

     Clearance. . . . . . . . . . . . . . . . . . . . . . . . . .56

     After Receipt of Approval. . . . . . . . . . . . . . . . . .58

 

                                (v)

 

 

 

 

 

 

 

 

                      Chapter 1. INTRODUCTION

 

 

BACKGROUND AND PURPOSE

 

Surveys are a major source of data for many agencies of the Federal

Government.  Surveys deal with a multitude of subjects, for

example, demographic, economic, psychological, sociological, and

medical topics, spanning a multiplicity of diverse governmental

concerns for the general welfare.  Included are surveys on the

effectiveness of government programs, the impact of regulated and

unregulated industrial activities, and problems for which

population groups are currently or may in the future be at risk or

in need of government assistance in some form.  Although these

topics are highly diverse. the issues in the use and conduct of

surveys to obtain data on them are far more amenable to systematic

treatment with a unified focus.

 

For the Federal Government to conduct a survey, there are the three

following primary options:

 

-    do the work in-house;

 

-    enter into an interagency agreement; or

 

-    contract the work out.

 

These options can be used in combination with each other, and in

actual practice this is often the case.  An agency may have the

data collection, reduction, and tabulation phases of a survey

conducted under a contract or an interagency agreement.1  It may

choose to design the survey and conduct the analysis in-house.

 

The Annual Survey of Manufactures is an example of a statistical

survey carried out by in-house staff.  Planning and preparation,

data collection and tabulation, and presentation of the data are

done by or under the immediate supervision of Bureau of the Census

employees.  Additional staff needed to satisfy the demands of high

volume data collection and handling are obtained on a limited

individual employment basis.

 

Because of its capacity to conduct statistical surveys, the Bureau

of the Census is one of the agencies often used by Government

agencies to conduct surveys.  For example, the Current Population

Survey conducted by the Bureau of the Census for the Bureau of

Labor Statistics is a statistical survey carried out through an

interagency agreement.  However, agencies that do reimbursable work

have limited subject matter expertise beyond their own missions. 

Additionally, their facilities are in demand and are by no means

unlimited.  For many survey efforts, then, it becomes necessary

that agencies of government utilize the services of outside

organizations.  A range of such services is available under

contract from non-federal government

___________________________

 

1Statistical Policy Working Paper 8, "Statistical Interagency

Agreements" (October 1982; Regulatory and Statistical Analysis

Division, Office of Information and Regulatory Affairs, Office of

Management and Budget), prepared by the Subcommittee on Statistical

Interagency Agreements, presents data on the prevalence of

interagency agreements and summarizes the prevailing practices and

policies of Federal agencies with respect to their use.

 

                                 1

 

 

 

 

 

sources.  Depending on the needs of the individual agency or

survey, these may include carefully delineated and defined services

related to data collection and processing or more general

involvement in all phases of a survey from planning to final

report.  In some cases the suggestion of the subject matter area in

which a survey is needed may originate outside the government and

result in a proposal for the entire effort, including the

definition of the area for study.

 

The overall functions of the Office of Management and Budget (OMB)

as related to review and approval of agency budgets, program plans,

and data collection plans set limits and provide guidance for the

use of statistical surveys.  For the most part, the issues of

concern center on the need for the data for policy and program

activities, the reasonableness of the data collection plan and

questionnaires, the coordination among agencies with overlapping

interests to avoid duplication of effort, and the concern for the

burden on potential respondents.

 

From another perspective, the Federal Procurement Regulations and

the unit within each agency responsible for the administrative

aspects of formal procurement actions, in accordance with these

general regulations, provide standards and requirements for a

contract as a legal document independent of what is being

purchased.  OMB Circular A-76 sets restrictions and requirements on

developing in-house capacity versus using nonfederal contracting

sources to obtain various services.  This affects the justification

for moving work in-house that has been done previously by

contractors and contracting work out for which an in-house capacity

exists.  The implications of Circular A-76 for contracting for a

survey should be reviewed with the procurement office.

 

However, beyond the general standards there is no specific guidance

on the use by Federal agencies of contractors for the conduct of

surveys.  They do not define when and how the procurement mechanism

should be used for such a purpose nor do they indicate what the

relationships should be among an agency's need for survey data and

analysis, its staff resources, and the specification of contract

requirements and monitoring of contractor performance.

 

The Subcommittee on Contracting for Statistical Surveys of the

Federal Committee on Statistical Methodology was organized to

suggest ways to improve the survey contracting process in the U.S.

Federal Government in the future.  The Subcommittee's work was done

under the general mandate of OMB.  Because of its unique

responsibilities, OMB was able to call upon the experience of a

variety of individuals with backgrounds in diverse aspects of the

conduct of surveys and the use of contracts.  This report

represents the collective effort of the Subcommittee.  The opinions

expressed here reflect the collective judgment of the Subcommittee

and do not necessarily reflect those of OMB.

 

It is not possible within any given presentation of this nature to

cover all the issues related to the use and conduct of surveys. 

The following questions are some that need to be answered: What is

the problem on which the study should focus? Should a survey be

conducted as opposed to the use of some alternative research

strategy or data collection approach? If a survey is to be used,

should it be done totally in-house, totally under an interagency

agreement, totally under a contract, or using some combination of

these three approaches? The Subcommittee focused on what should be

done after the decision is made to contract out the survey, and not

on the factors involved in selecting this option as against an

interagency agreement or in-house conduct of the survey.

 

                                 2

 

 

 

 

 

As related to survey design and conduct, issues are: What should be

the universe for study? What is the most appropriate sample design?

Should the data be collected by mail, by telephone, or by personal

interview? What specific questions should be asked and how should

the data be analyzed? These and many other issues must be addressed

at some point in the course of deciding to use, planning and

preparing for, and actually conducting a survey.  While the need to

address such issues is discussed, there is no effort to provide

input into the decision making process on such matters.  There are

many references and resources available on survey design and

content considerations so that it would be inappropriate to try to

duplicate these here.

 

This report is intended for circulation to Federal agencies and

offices which may utilize contracting for surveys.  The report

tries to draw attention to the issues that need to be considered,

how to use or obtain the necessary expertise in preparing for and

using the contract mechanism, how to prepare specifications and

carry out contractor selection, and how to maximize contractor

performance.

 

It should be pointed out that this report does not deal with

surveys conducted or supported under grants or interagency

agreements.  The report focuses on contracts awarded through an

open competitive negotiated procurement mechanism based on

evaluation of the technical and business proposals submitted in

response to generally available Requests for Proposal (RFPs).  An

RFP is a formally issued notice of a government agency's need to

have work done by outside organizations where contractor selection

cannot be based on price alone.  The RFP is the publicly released

notice which contains the internally approved program Request for

Contract.  The primary reason for focusing on this mechanism is

that it offers the agency the maximum opportunity to set the

requirements for the survey and the standards for contractor

performance and to select the organization that offers the best

combination of anticipated product quality and cost factors,

without regard to issues unrelated to these concerns.  Other

contract approaches, while serving other purposes, do not

necessarily promote selection of the best choice of contractor for

conducting a given survey at a given point in time.

 

Among the alternative contract approaches are sole source selection

because of a unique capacity to perform or based on an unsolicited

proposal, and set-aside programs for competition only among small

businesses in general or minority-owned small businesses.  The

small business set-aside and the minority-owned business programs

permit competition for prime contract work among firms which are

qualified for these programs but totally exclude the larger

organizations with greater resources and all nonprofit research

units, including academically-based units.  This is part of the

Federal Government's efforts to promote the economic viability of

small, and particularly minority-owned, businesses.  However, on

any given procurement, these restrictions limit the range of

competition.

 

Although excluded from the small business and minority-owned

business programs, nonprofit organizations are eligible for sole

source contracting.  The mechanism totally restricts competition,

either because only one organization is determined to have the

capacity to perform or because the organization submitted an

unsolicited proposal that represents a new and unique approach and

opportunity to obtain a valuable product.  In this latter

situation, the organization submitting the proposal controls the

product offered, that is, the particular survey to be performed,

and is the only one to which the contract can be awarded.

 

In line with the intention to provide information designed to

improve the ability to use the contracting mechanism for

procurement of survey services it was important to learn more about

what the agencies were doing at that time.  Since no useful data

 

                                 3

 

 

 

 

 

existed on this process, Subcommittee members interviewed officials

of several selected Federal agencies involved in contracting for

surveys and some of the contractors with which the agencies dealt. 

Appendix I summarizes the data collected.

 

The agencies selected for the survey of their contracting

operations were asked to have representatives of both the program

and the administrative aspects of contracting operations available. 

The da were collected by personal interview.  The questions were

directed toward contracting for surveys only.  The questions traced

the use of this mechanism from the factors in the decision to use

different types of contracts (cost or fixed-price) and how often

they were used, through developing RFPS, soliciting bids and

evaluating proposals, to monitoring contracts.  Similar information

was collected from a limited number of contractors.  This

information has been analyzed and its implications assessed for

developing suggestions designed to improve the use of the contract

mechanism for conducting surveys.  The various reviews of current

practice that have been made have been interwoven into this

presentation.

 

 

THE CHAPTERS TO FOLLOW

 

In structuring the material to be presented, an effort was made to

follow a logical order of stages in the development and management

of contracts for surveys.  The sequence of events in this process

was a guiding principle within the general format, but not an

absolute basis for organizing the chapters.  The different chapters

deal with the development of requirements for RFPs and with issues

in the selection and monitoring of a contractor after an open

competitive RFP has been issued.  It must be understood that no

part of this process is independent but is closely linked to and an

integral part of the entire process of defining a requirement to be

done under contract and seeing the contract work through to

completion.  In this report, "technical skills" refers to survey

and programmatic content skills as distinct from management and

administrative aspects of the procurement process.

 

Chapter 2 deals extensively with general issues related to the

technical skills involved in survey planning and conduct and in the

use of contractors for the performance of various tasks in the

process.  The remaining chapters follow an order of activities in

the use of contracts.  Chapter 3 contains a review technical survey

design issues and other considerations in preparing RFPS.  Chapters

4 and 5 deal with issues in the selection of contractors and the

management and evaluation of contracts after award.

 

Chapter 2 is a detailed discussion of the skills required in

contracting for surveys and the types of individuals who may serve

as technical advisors.  This chapter also includes a discussion of

the activities and concerns in the planning and preparation for a

survey to be done under a contract, the competitive selection of a

contractor, and the monitoring of performance as these relate to

the skills required at various stages in this process.  A variety

of technical survey research skills, as well as content area

competencies, is needed to deal with issues such as determining the

overall research design to be followed, the design of the data

collection instrument or instruments, the universe to be studied,

the design of the sample, the conduct of the data collection effort

itself, data editing, coding, and quality control.

 

At each stage of a survey performed under a contract, there are

various technical skills required to maintain a smooth operation. 

Differences of in-house technical capabilities of agencies and the

possible need to supplement those skills are discussed.  Both the

use and misuse of technical advisors by agencies are considered.

 

                                 4

 

 

 

 

 

Chapter 3, on considerations in the preparation of RFPS, contains a

review of the tasks that may need to be performed in preparing a

statement of work for competition among prospective offerors rather

than the skills that need to be available to the agency in

undertaking this effort.  The chapter includes a discussion of

issues to be considered in developing an RFP.  The interaction

between technical requirements and degree of specification and the

contract reimbursement method (that is, fixed-price or cost-type

contract) is also reviewed.  Different types of contracts imply

different assumptions about the ability to predict and specify

activities, results, and costs and, therefore, impose different

limitations on the ability to trade off between cost and

performance.

 

In Chapter 4, on the contractor selection process, the selection

process for open market competitive RFPs is discussed.  The various

stages of the contractor selection and contract negotiation process

and the roles played by the contracting officer, project officer

and technical evaluation panel are described.  This is the

framework within which the technical merits of the work proposal

are presented and evaluated, and in which a contract is awarded. 

The process of debriefing unsuccessful offerors is also discussed.

 

The last of the substantive chapters deals with contract management

and evaluation after award.  This is the period of actual

contractor performance in line with the terms and requirements of

the contract.  The project officer and the contracting officer are

jointly involved in monitoring and controlling the contract effort. 

The discussion in the chapter centers on the roles of these

individuals and contact that should be maintained between the

project officer and the contractor in monitoring these activities.

 

Depending on the specifics of the contract and the background and

competence of the project officer, this person's input into the

technical aspects of the survey will vary.  There may be specific

activities which require approval; data collection instruments

usually require formal clearance.  The project officer must certify

invoices for payment and must, therefore, be aware of contractor

progress and current and planned activities under the contract as

these relate to the remaining time and funds available.

 

In addition to these four substantive chapters, there are two

appendixes.  These are a review of the informal survey of agencies

and contractors conducted by the Subcommittee, and a discussion of

the OMB clearance process for surveys.

 

                                 5

 

 

 

 

 

 

 

 

  Chapter 2. TECHNICAL SKILLS REQUIRED IN CONTRACTING FOR SURVEYS

 



NTRODUCTION

 

A survey is generally a rather complex activity.  Furthermore, the

results to be derived from a survey are usually difficult to

specify, for example, types of data and analyses desired. 

Consequently, the task of contracting for a survey is not

straightforward.  Specifically, a considerable amount of effort is

required to provide complete and clearly written survey

requirements in a Request for Proposal (RFP), select the best

offeror to receive the contract, and adequately monitor and review

the progress of the survey.

 

Technical expertise is essential to the success of contracting for

and monitoring the execution of a survey.  In this chapter,

technical skills that should be available when contracting for

surveys are presented.

 

Following are areas in which technical skills are required:

 

-    Developing project objectives and specifications, Subject

     matter knowledge,

-    Project cost and scheduling,

-    Questionnaire design and testing,

-    Sample design and selection,

-    Data collection, including follow-up procedures,

-    Data processing, including coding, editing and file creation,

-    Data analysis,

-    Quality control, and

-    Survey documentation and report preparation.

 

The subject matter knowledge required for a particular survey will,

of course, vary depending on the nature of the survey.  As

mentioned previously, surveys deal with many subjects, for example,

demographic, economic, psychological, sociological, and medical

topics.  The other types of technical skills listed above are more

generally applicable to all surveys.

 

It is very unusual to find all the skills required for a particular

survey in one individual, although it is often true that some

people will possess more than one of these skills.  Consequently,

it is usually necessary that there be a Government project team

consisting of the project officer and one or more technical

advisors to supply the required technical skills.  The number and

types of advisors needed will vary according to the size and

complexity of the survey being planned and the experience of the

project officer.  Smaller and less complex surveys will usually

require fewer, and less specialized advisors than large, complex

surveys.

 

                                 7

 

 

 

 

 

It is important that the need for a project team and the types of

members needed be recognized early in the development of a project. 

At least some of the advisors should participate with the project

officer in the following major phases of the survey contracting

process: (1) project planning and preparation of the RFP, (2)

evaluating proposals and awarding the contract, (3) and monitoring

the work done under the contract.

 

In many cases the agency contracting for a survey will not have on

its staff all the technical advisors needed.  In such a case, the

agency should use technical advisors as part of the project team. 

These advisors could be persons employed by other Federal agencies

or nonfederal consultants.  Nonfederal consultants who assist in

development of the RFP are not eligible to respond to the RFP.  If

advisors are needed, they should be brought in at the beginning of

the contracting process so that they can participate in planning

the project, in preparing the RFP, and in evaluating the proposals.

if advisors are brought into the project early, they can provide

the guidance and assistance needed in designing and executing a

survey.

 

It is generally a mistake to rely solely on the contractor to

provide the technical expertise required during the execution of

the survey plan and the processing and analysis of the data.  If

this is done, the technical quality of the survey may be diminished

because the contractor's technical expertise may not be fully

adequate, or the contractor may decide that it is not in his or her

best interest to take steps to ensure the highest technical

quality.  Even with the highest quality contractor, the Government

should still provide technical review and guidance as part of the

monitoring of the survey process; otherwise, the value of the

project may be reduced.

 

The technical aspects of some phases of a survey are often not

fully appreciated.  For example, in the design of the data

collection activities of a survey, a reasonable time schedule

cannot be formulated without regard to technical input on follow-up

activities, such as the number and type of follow-up attempts

required.

 

The next section of this chapter contains a discussion of the

required technical skills listed earlier.  The following section

covers the type of technical input required at the three major

stages of the contracting process.  The final section includes a

discussion of some of the technical aspects of current practice in

contracting for surveys.

 

 

BASIC TYPES OF SKILLS REQUIRED

 

Developing Project objectives and Specifications

 

 

The formulation and definition of project goals and objectives is

the most fundamental aspect of a project.  Often goals and

objectives are not defined adequately.  In such cases there will be

confusion regarding the type and amount of survey information to be

collected, and hence the strong possibility that the project

objectives will not be met.  Sometimes a project may have multiple

objectives that may be somewhat conflicting.  For example, if both

subgroup and total population estimates are important, a decision

must be made on how to allocate the sample to these subgroups.  The

sample allocation that would provide equal precision, or some,

other specified precision, for subgroup estimates will generally

differ from the sample allocation that would maximize the precision

for total population estimates.  Priorities must be set for each

established goal and objective at the beginning of the project in

order to resolve any conflicts that may subsequently evolve.

 

                                 8

 

 

 

 

 

In addition to stating the objectives clearly, an effort should be

made to define the objectives to be as narrow or specific as

possible.  If the objectives are too broad or general, the survey

might become too large in scope to manage adequately for the time

and funding available.

 

If the goals and objectives are defined adequately and if the study

population has been defined, then appropriate technical

specifications can be developed for instrument design, sample

design, and data analysis. consequently, these technical aspects of

the project should be kept in mind as the goals and objectives are

developed.

 

 

Subject Matter

 

As mentioned previously, sample surveys cover many different

subjects.  In order to formulate and develop a project adequately,

there should be a subject matter expert on the project team at the

start of the project.  Such subject matter expertise is required in

order to insure that all data requirements. are delineated.  The

subject matter expert may not know the appropriate manner in which

to phrase a question in order to elicit the desired response, but

will know the fundamental issues that must be explored in order to

satisfy the data requirements.  For example, to develop a survey to

investigate the accuracy of the amounts paid to railroad retirement

beneficiaries, there should be someone on the project team with

knowledge of the railroad retirement system.  This subject matter

expertise should allow for the formulation of the most appropriate

survey questions and should enable the project to collect adequate

information from respondents.

 

 

Project Scheduling and Costing

 

Scheduling and costing various phases of a survey are generally

difficult.  Typically, a gross cost for the project is set very

early, particularly for one-time surveys.  Agency personnel are

generally limited by the initial gross cost allocated to the

project.

 

The amount of time required for planning and writing the RFP is

often underestimated.  This can delay the scheduled start of the

survey which, in turn, can jeopardize the quality of the work. 

Sufficient time for execution of the survey by the contractor must

be allowed.

 

Since there are several aspects of a survey, such as printing, data

collection, and data processing, that may require special

clearances, applicable requirements should be determined as early

as possible in development of a survey to be conducted under

contract.  The procurement office can provide a list of required

clearances so that their applicability to a particular survey can

be determined and provision for compliance can be made in the

project schedule.

 

In addition, there are several survey tasks which involve aspects

for which the completion time and cost are typically hard to

predict.  For example, a survey may involve a large frame

development phase in which a number of frame sources have to be

investigated.  This could require working with a number of computer

files involving problems of incompatibility between various

computer systems, different record layouts and file definitions,

and matching elements in different files.  These types of

activities are typically difficult to assess in terms of time and

cost.

 

Other survey activities that are typically difficult to assess in

terms of cost and time requirements are questionnaire design and

testing, data collection, and data processing.  Due to the

potential for cost and scheduling problems as indicated above, it

is important during the project development phase to include a

person on the

 

                                 9

 

 

 

 

 

project team who has experience in developing project schedules and

cost estimates for surveys.

 

 

Questionnaire Development and Testing

 

Proper development of a survey questionnaire and other data

collection instruments is a deceptively difficult phase of a

project.  The knowledge, time, and effort required to complete this

phase adequately is often underrated.  In addition to preparing the

instruments, time must be allowed to adequately test them.  Testing

is important since it is generally impossible to anticipate all the

problems that will arise when an instrument is used.  It is

important to include someone on the project team who has had

considerable experience in questionnaire development and testing so

that the common pitfalls of instrument design can be avoided and so

that the instrument may be designed to facilitate data processing.

 

If a questionnaire is used to collect the survey data and the

questionnaire is not developed by someone with appropriate

experience, it is likely that some questions will be unclear to the

respondents, will be misinterpreted by the respondents, will

confuse or upset some respondents, or will tend to influence or

"lead" the respondents.  The position of the question on the

questionnaire may influence the response rate to the question.  For

example, sensitive questions, like income, are usually answered at

a higher rate if they are asked after the respondent has become

more comfortable with the interviewer.  Also, the nature of the

questions surrounding a question of interest may influence response

proportions by several percentage points.  Furthermore, the

questionnaire may be difficult to administer and may not provide

adequate survey information.  In addition to operational

difficulties, these problems can cause both response and

nonresponse biases in survey estimates.

 

 

Sample Design and Selection

 

Even for a survey that seems straightforward in terms of the

information to be gathered and the definition of the target

population, there are usually some difficult questions that arise

in designing the sample selection method, particularly if the

survey has multiple objectives.  Questions about sample size

determination, frame development, stratification procedures, sample

allocation to strata, or whether or not to use differential

probabilities of selection arise in all surveys.  These questions

become particularly difficult to answer when there are conflicting

objectives.  For example, if the project team has not decided

whether subgroup estimates are more important than overall

estimates or vice versa, satisfactory decisions on stratification

and allocation cannot be made.

 

,Many of the questions that arise are not easily answered by

referring to standard textbooks.  Consequently, the project team

should include a person who not only has. the theoretical knowledge

of sampling methods, but who has also had substantial experience in

the practical application of sampling principles to the design of

the type of sample survey involved in the contract.  This type of

experience would include, for example, knowledge of available frame

sources and of data available to use for strata formation or for

assigning differential probabilities of selection.

 

 

Data Collection

 

The data collection phase of a project contains several aspects. 

First, the mode of interviewing - personal visit, self-administered

questionnaire (usually mail), or telephone - should be selected

based on the type of questions to be asked, the population to be

studied, and the time and resources available.  Next, interviewers

must be recruited and trained for the in-person or telephone

administration of a

 

                                10

 

 

 

 

 

questionnaire.  Even if a staff of experienced interviewers is

available, some training for each new questionnaire is necessary.

 

A budget and schedule for interviewing, including callback

procedures, must be developed.  For personal interviews, the

schedule often includes a mailing or telephone call to the selected

sample units prior to the personal interview.  The number and

timing of callback visits allowed in the schedule will have an

important impact on the survey response rate, which, in turn, is a

key factor in controlling nonresponse bias.  Acceptable response

rates need to be determined and follow-up procedures must be

devised and implemented.

 

Another aspect of data collection is questionnaire check-in.  In

addition to recording the receipt of the questionnaires, this

involves the development of editing procedures to check for faulty

or missing data.  Procedures also have to be developed to follow-up

on questionnaires that fail this edit.  These procedures are

important in order to improve data quality and reduce the response

and nonresponse biases.

 

 

Data Processing

 

The data processing phase usually involves entering the survey data

into a computer file so that it can be accessed and analyzed.  The

first step is to provide any coding and editing of questionnaire

responses that are needed.  The coding is needed to convert all the

responses into numerical codes for entry into a computer file.  For

example, if occupation was a survey item, a numerical code would be

assigned to each occupation to use in recording the responses on

the file.  Proficiency in this activity requires considerable

practical experience in coding and editing of survey data and

knowledge of computer systems.

 

Editing is necessary to detect responses that are unreasonable or

that are inconsistent with other responses.  The editing may either

be performed manually or by use of the computer.  Developing

editing rules is difficult and requires input from subject matter

specialists.  Responses that fail edit checks can either be

followed up, replaced (imputed for) or discarded.  All imputed data

should be flagged on the file.

 

The survey data are placed into a computer file by having the coded

questionnaire responses keyed to a specified record format.  It is

important that the computer record layout be documented completely

and accurately so that the data will be correctly accessed.

 

 

Data Analysis

 

An analytic plan and strategy should be developed in the earliest

stages of the survey development process.  Often included in the

plan are specific table shells to be used in survey tabulations. 

This plan is the foundation for sample and questionnaire design and

is essential for the survey to provide the results intended.

 

Often the survey analysis phase includes the estimation of

population and subpopulation characteristics (for example, means,

totals, and proportions).  In addition, there are many types of

more complex statistical analyses that may be applied to survey

data, depending on the objectives of the survey.  Some examples of

analyses performed are the following: hypothesis testing on

population means or proportions, regression and correlation,

analysis of variance, and log-linear analysis.

 

Before statistical analyses are applied to survey data, imputed (or

pseudo) responses are often inserted in the records of respondents

that have missing or faulty responses.  Setting up an imputation

procedure for missing data is a complex process that

 

                                11

 

 

 

 

 

requires experience.  Also, differential probabilities of selection

should be taken into account in applying statistical analyses. 

This is often accomplished by assigning weights to survey

respondents to reflect varying probabilities of selection.  These

weights are usually adjusted to account for eligible sample units

that do not respond.

 

Once the survey estimates and statistical analyses have been made,

the results must be examined.  Study conclusions, and perhaps

recommendations, are then made.  The data analysis phase is a very

important part of the project and requires the assistance of

appropriate subject matter experts and of a statistician who has

had experience in analyzing data from surveys.

 

 

Quality Control

 

During virtually any survey operation there are many points when

data collection or processing errors can be made that diminish the

accuracy of survey data.  These instances include sample selection,

interviewing,and/or completing the survey instrument, check-in and

field editing, coding and office editing, and keying the survey

data.

 

It is strongly recommended that quality control procedures be set

up at each step to detect and correct as many of these survey

operations errors as is feasible.  At a minimum, the quality

control program should cover interviewing, field editing, coding,

and keying.  These procedures should be set up by a person who has

had experience with survey quality control procedures.

 

 

Survey Documentation and Report Preparation

 

It is important throughout the project to prepare enough project

reports to adequately document the survey procedures, results, and

conclusions.  Poor documentation is a very common problem with

survey projects.  One reason for inadequate documentation is that

project funds often run low and time becomes short near the end of

a project.  A conscientious effort should be made to allow adequate

time in the project schedule for ongoing documentation of the

survey procedures used.

 

These procedures include definition of the target population and

frame construction; interviewing procedures, including callback and

follow-up procedures; sample design and selection; editing and

coding procedures; weighting and nonresponse adjustment methods;

estimation and variance estimation equations and statistical

analyses used; and quality control methods applied.

 

The final project report should include survey results, analyses,

conclusions, recommendations based on the results, and suggestions

for additional research.  Project specialists, such as subject

matter specialists and statisticians, should be involved in

preparation of the final report.

 

 

TECHNICAL INPUT AT VARIOUS STAGES OF CONTRACTING FOR A SURVEY

 

As mentioned earlier, substantial technical input is needed at each

of the following three major phases of contracting for a survey:

project planning and preparation of the RFP; evaluating proposals

and awarding the contract; and monitoring the contractor's

performance.

 

                                12



 

 

 

 

 

Unfortunately, the importance and/or extent of technical input

needed at these stages is often underestimated, thus diminishing

the quality of the survey.  The required inputs are discussed below

for each of these major phases.

 

 

Project Planning and Preparation of the RFP

 

Right from the start of the process of contracting for a survey, it

is important to have technical input and program and subject matter

expertise.  Subject matter experts and statisticians can help

define specific project goals and objectives that will satisfy the

purpose of the study and yet will be achievable, within the time

and budget available for the project, by applying appropriate

survey and statistical methods.

 

A statistician should participate in the development of the RFP so

that it will be complete and accurate in terms of the statistical

requirements of the survey.  In cooperation with the project staff,

the statistician should prepare or assist with the preparation of

the following portions of the RFP;1

 

     -    Sample size specifications, if any, for the target

          population and/or for population subgroups,

     -    Precision requirements for one or more key survey

          estimates, including any requirements for population

          subgroups,

     -    Response rate required and/or number of callbacks

          specified,

     -    Requirements for nonresponse imputation procedures,

     -    Quality control procedures,

     -    Variance estimation requirements, and

     -    Specification of particular population or subpopulation

          estimates required.

 

If a statistician does not participate in the preparation of those

portions of the RFP that address the above topics, it is virtually

certain that some of these specifications will be omitted from the

RFP, and some of those included will not be properly stated.  In

such a case, offerors would be confused and may prepare proposals

that miss the real objectives or which are somewhat vague,

incomplete, or misdirected in terms of important technical aspects

of the proposed survey.  This, in turn, would make it more

difficult to compare the proposals and to accurately evaluate them

in terms of technical content.

 

 

Judging and Selecting Among Offerors

 

Technical input is essential in the evaluation of the proposals. 

In addition to subject matter specialists, at least two persons

with varying experience in the different aspects of a survey should

be included on the proposal review panel.  At least one of these

persons should be a statistician.

 

A major focus of the technical review of proposals is the offerors'

responsiveness to the survey specifications, such as frame

construction, sample design and selection, quality control plans,

and estimation and analysis plans.  Just as important as evaluating

these technical responses is the assessment of the offerors'

abilities to fulfill the statements and claims made in their

proposals.

___________________________

 

1For a general type of RFP the technical input discussed above is

still needed.  However, some of these items would not usually be

specified.

 

                                13

 

 

 

 

 

For example, if they claim that they would achieve an 85 percent

response rate, is there sufficient evidence to support this? The

basis for this type of evaluation would primarily be the previous

experience and perforMance of each offeror and the technical skill

and experience of the proposed project staff.  Of course, the

survey plans in the proposal would also be used to help assess the

offerors' potential to achieve the proposed statistical standards.

 

 

Monitoring the Contractor's Performance

 

After a contract is awarded and the project begins, technical input

is still very important to the success of the project.  The

contractor will presumably be describing details of their survey

plans at meetings and in written documents.  Technical personnel

representing the agency should attend all project meetings which

involve technical issues and should review all project documents to

ensure that specified technical standards are met.  Even if the

contractor's project staff appears to be technically sound, it is

necessary for the agency to continue technical input.  This should

ensure that the standards of the contract are met and may avoid

technical problems that might have appeared near the end of the

project if technical input had been interrupted.

 

Any problems on technical issues between the agency and the

contractor should be discussed and resolved.  If resolution of

these problems requires changes in the specifications, the changes

should be proposed to the contracting officer for decision and

formal action.  To facilitate resolution of problems, there should

be an open line of communication between the contractor's

statisticians and other technical staff personnel and a liaison for

he agency's project staff.  Good communication between contractor

and agency staffs can prevent substantial misunderstandings on

technical issues that may arise.  Ideally, there should be enough

discussion of technical issues between the agency staff and

contractor staff so that when the contractor submits a technical

document, such as a sampling or analysis plan, agency personnel

will be aware of the major technical aspects of the document.  If

communication has not been good and major problems with technical

issues arise, there may be substantial time delays in the project

work.

 

In addition to participating in meetings and reviewing documents,

in some surveys it is useful for agency technical personnel to make

site visits.  The purpose of a site visit might be to observe and

review the implementation of sampling procedures, interview

activities, coding and editing, or quality control activities.

 

 

SOME TECHNICAL ASPECTS ASSOCIATED WITH CURRENT PRACTICE

 

Use and Misuse of Technical Advisors

 

As expressed previously, the use of technical advisors in

contracting for a survey is vital to the success of the project. 

Although project managers generally recognize this need, technical

advisors are not always used properly.

 

In the previous section it was pointed out that it is important to

have technical input and program-specific expertise from the

beginning of the process.  Often the technical advisors are added

to the project team too late to assist with the preparation of the

RFP.  In such a case, the specifications of the survey may be

inadequately defined.  For example, the precision requirements for

the survey may be unclear, causing confusion and apprehension among

offerors.  As another example, the minimum acceptable response rate

may have been omitted, which would probably cause problems in

comparing proposals.

 

                                14

 

 

 

 

 

Technical advisors are generally called upon to review and judge

the technical proposals submitted.  However, in addition to rating

proposals on technical aspects with which they have knowledge and

or experience, members of the review panel are also asked to rate

proposals with respect to technical aspects with which they are

less "familiar, or even unfamiliar.  For example, a statistician

who specializes in sample design may be asked to rate the proposals

with respect to questionnaire design and quality control

procedures, in addition to sampling aspects.  The reviewer's

experience may be rather limited in areas other than sampling.  As

a result, ratings of these aspects of the proposals might be

inappropriate.  To try to minimize this problem, individual

reviewers, during the discussion and review of ratings, may revise

their original score on an item to reflect the opinion of another

reviewer with particular expertise in the area.  To allow for this

type of rating adjustment, it is advisable for an agency to attempt

to balance the members of the review panel with respect to the

important technical aspects of the proposals that need to be rated.

 

 

Use of Outside Help to Supplement in-house Technical skills

 

For many surveys being contracted, the in-house technical skills

needed to design the survey, to judge offerors, and to monitor

survey progress, are not fully adequate.  In such cases technical

advisors from other Federal agencies or nonfederal consultants

should be added to the agency's project team.  For example,

advisors specializing in sample" design, questionnaire design, or

field operations may be needed. it is, of course, important that

the advisors required are added to the project team early in the

development of the survey.

 

As was indicated earlier in this chapter, the agency should have

its own technical advisors to provide the technical expertise in a

particular area, rather than relying solely on the contractor's

expertise.  Even if outside consultation is needed, it is important

to have the appropriate technical advisors on the agency's project

staff to protect the agency's interests.

 

There are a number of Federal agencies that have technical staff

who should be able to provide assistance directly or to suggest

others with appropriate skills.  These agencies include the

following: Bureau of Economic Analysis, Department of Commerce;

Bureau of Labor Statistics, Department of Labor; Bureau of the

Census, Department of Commerce; Bureau of Justice Statistics,

Department of Justice; National Center for Education Statistics,

Department of Education; National Center for Health Statistics,

Department of Health and Human Services; Office of Research and

Statistics, Social Security Administration, Department of Health

and Human Services; Statistical Reporting Service, Department of

Agriculture; Statistics Division, Internal Revenue Service,

Department of the Treasury.

 



                                15

 

 

 

 

 

       Chapter 3. CONSIDERATIONS IN THE PREPARATION OF RFPS

 

 

INTRODUCTION

 

This chapter covers some considerations in the preparation of a

Request for Proposal (RFP) for a survey.  These comments and

suggestions are not all-inclusive; additional considerations will

be warranted for some RFPS.  For example, certain types of

surveys - such as repetitive or longitudinal surveys - pose special

problems that are not addressed here.  All of the possibilities

cannot be covered because of the wide range of purposes and survey

designs encountered in surveys sponsored by the Federal Government. 

Emphasis is placed on elements which are included in many of the

RFPs prepared in Federal Government agencies and on those portions

of the RFP to which the project officer or other technical

personnel are most likely to contribute.1  On the other hand, the

comments and suggestions which are presented are not likely to be

applicable to every RFP, either because of the nature of the

proposed surveys or because of variations in regulations, policies

or procedures among government agencies.

 

 

INITIAL CONSIDERATIONS

 

Because of the variations noted above, it is essential to contact

the procurement off ice that services the sponsoring agency at the

conceptual stage when a survey contract is contemplated.  Early

discussions with the contracting office will enable determination

of the contracting options, if any, which are available, the input

expected from the program staff, the steps which should be followed

from development of the RFP and a list of potential offerors

through award of the contract, and the amount of time each step is

likely to take.

 

 

Payment Provisions

 

One option that might be available concerns the payment provisions

of the proposed contract.  A contract may stipulate a fixed-price

which is agreed upon in advance by the sponsor and the contractor,

or it may provide for reimbursement of allowable costs (up to a

preset maximum) incurred by the contractor in performing the

survey.  The decision about which approach to use is usually made

by the procurement office based on technical contracting

considerations, not by the program staff. Some agencies use only

one of these financial arrangements for all of their contracts;

some agencies use one or the other, depending" on the nature of the

survey to be undertaken.  The RFP should indicate whether the

proposed contract will stipulate a fixed price or will provide for

reimbursement of costs incurred by the contractor.

 

There are advantages and disadvantages to each of these payment

mechanisms.  For example, fixed-price contracts assure, in advance,

that sufficient funds are available for completion of the work and

keep the sponsor's administrative audit expenses to a

___________________________

 

1A complete RFP contains a number of standard provisions or

sections originating in the contracting office to which the program

staff have little or no input.

 

                                17



 

 

 

 

 

minimum.  However, some qualified potential offerors will not

submit a proposal for a fixed-price contract and competition is

therefore reduced. it is particularly important that fixed-price

contracts contain comprehensive, detailed statements of the

requirements for the work to be performed by the contractor to

avoid misunderstandings about what the agreed-upon price actually

covers.  These requirements should be based on a combination or

merging of the specifications in the RFP and the contents of the

successful offeror's final proposal.

 

If there are many unknown or undecided factors in the project plan

that affect the predictability of costs, a cost-type contract may

be preferable, provided adequate funding will be available.  A

"dollar cap" or maximum cost can be included in a cost-type

contract.  If a cap is specified, flexibility should be built into

the survey design so that if avoiding cost over-runs, which

jeopardize completion of the survey, is not possible, a limited but

still useful product can be obtained.

 

 

Phased Surveys

 

If there are any serious doubts about the feasibility of key

aspects of the survey plans because of unknown factors such as the

ability or willingness of designated respondents to provide the

information sought, it may be prudent to limit the scope Of the

project to which the sponsor will be committed.2

 

One approach which might be appropriate in this situation is to

issue an RFP solely for an evaluation of the proposed methodology,

with the stated intention of issuing another RFP for a full-scale

survey if the initial effort indicates the methodology is feasible. 

Obviously, this approach would consume more time because of the two

RFPs involved.

 

Another possible approach is to specify work segments which may be

done during the course of the contract if the concerns are resolved

in the earlier stages of the contract.  If a phased survey is

deemed advisable, early discussions with the Procurement office

will indicate what contracting alternatives are available for

consideration and their advantages and disadvantages.

 

 

Survey Design Specification

 

RFPs may state all, some, or none of the details of the survey

design.  The level of survey design specification found in RFPS, in

practice, ranges from "This is exactly what we want, including the

questionnaire to be useful" to "We have a problem and need

information to help solve it; what do you suggest?" The choice of

the level of specification is dictated by a number of

considerations such as the nature of the survey to be conducted,

the applicable regulations and policies, and the technical skills

available among the sponsor's staff or advisors.

 

Detailed survey design specifications may be appropriate if the

sponsoring agency has highly qualified and experienced statistical

and survey staff available to develop the RFP who are aware of

potential contractors' capabilities and resources, and who know

exactly how the agency wants the survey to be conducted.

___________________________

 

2Although the general provisions incorporated in Federal survey

contracts usually provide for termination of the contract for the

convenience of the Government, exercising provision can be

difficult, complex, and time consuming.

 

                                18



 



 



 



 



 



Such an RFP may be more difficult and time consuming to develop



than one that calls for offerors to suggest survey designs. 



Maximal specification of the survey design will indicate the



quality standards of the sponsor to potential offerors, and will



also permit them to repeat the requirements in their proposals



without demonstrating capability or understanding of the problem. 



It may be easier and quicker to evaluate the responses to a



detailed RFP, but more difficult to determine the technical



capabilities of the offerors.  Selection is therefore more likely



to be made primarily on price.  Any unnecessary constraints imposed



on the survey design would preclude desirable flexibility and



innovation by offerors which could improve the quality of the



survey or decrease its cost.  Furthermore, some of the more



competent survey research organizations dislike this approach and



are reluctant to respond to this type of RFP.



 



Less detailed RFPs which, invite offerors to make suggestions or



develop survey designs, allow them to apply their knowledge and



demonstrate their ingenuity or creativity.  However, minimum design



criteria or quality standards should be stated, when known, even if



the offerors are asked to develop the survey design.  Evaluating



the proposals received in response to a less detailed RFP is likely



to be more difficult and time consuming.  When an RFP does not



specify the details of the survey design, the successful offeror's



explicit statement of the survey design specifications is commonly



incorporated in the contract awarded.



 



An alternative that incorporates many of the advantages of both the



detailed and less detailed survey design specification approaches



has been used effectively by some agencies.  In this procedure,



potential offerors are invited to submit proposals based on



detailed specifications but are also encouraged to suggest



modifications to the detailed plans which they believe will be



advantageous to the sponsoring agency.



 



 



Bidders' Conferences and Due Dates



 



Some RFPs - usually only those for large or complex surveys -



include provision for a Bidders' Conference on a specified date to



which prospective offerors are invited.  The conference should be



held after prospective offerors have had the opportunity to study



the RFP, but well before the due date for proposals.  They may be



asked to submit their questions beforehand so that answers can be



prepared in advance for the conference.  At the meeting, technical



and procurement personnel from the sponsoring agency answer



questions raised by potential offerors about the RFP or arrange to



provide answers later for any issues which cannot be resolved



during the conference.  They may also review contractual



administrative requirements.  Discussions at a Bidders' Conference



will benefit the sponsoring agency if they identify RFP



specifications which are misunderstood or which appear to potential



contractors to be inconsistent, inadvisable or unacceptable.  Any



additional information or material developed as a result of the



Bidders' Conference should be mailed to all requesters of the RFP.



 



It is very important in any RFP to allow a reasonable amount of



time for preparation and submission of proposals - at least 30



calendar days after the RFPs are mailed and more than 30 days if



the proposed survey is complex or offerors are asked to develop



substantial portions of the survey design.  The availability of an



RFP is announced through the Commerce Business Daily. it is also



advisable that the names and addresses on mailing lists of those to



receive the RFP are current.



 



                                19



 



 



 



 



 



POSSIBLE CONTENTS OF RFP's



 



When developing an RFP, it may be helpful to refer to RFPs for



similar surveys conducted by the sponsor or by another Federal



agency as sources of ideas for content and wording.  However, these



RFPs should be used together with comments or suggestions from



persons familiar with those surveys about problems or



misunderstandings which arose during the contractor selection



process or the conduct of the study - particularly difficulties



which might have been forestalled if the RFP had been written



differently.



 



The goals of the procurement process should be kept in mind when an



RFP is written.  Ideally, an RFP does the following:



 



     -    Requests the development of or specifies a technically



          sound survey design which will meet the sponsor's



          information needs and is realistic, considering the time



          and funds available for the project and the capabilities



          and resources of potential contractors.



 



     -    Conveys all the information needed by potential



          contractors to prepare a responsive proposal, including



          all the terms, conditions, and provisions that the



          sponsor intends to incorporate in the contract.



 



     -    Asks for all the information that procurement and



          technical personnel will need to conduct an accurate and



          equitable evaluation of the proposals received.



 



     -    Attracts responsive proposals from enough of the



          qualified potential contractors to provide adequate



          competition and to permit a suitable selection.



 



Some agencies include in the RFP a table of contents for responses



to be followed by all offerors and stipulate that staffing



requirements should be specified in a common measure (for example,



hours).  This simplifies reading and comparison of proposals.



 



Specific suggestions for developing an RFP that meets these



criteria are presented in the remainder of this chapter.



 



 



Statement of Survey Objectives



 



Every RFP should contain a clear statement of the specific



objectives of the survey to be conducted under the ensuing



contract.  Discussing the proposed research in the larger context



of the mission of the agency may also be advisable, particularly if



the project is part of an ongoing integrated program.  The



statement of the survey objectives should also indicate what groups



(such as government agencies, congressional committees, state or



local governments, trade associations, and business firms) will be



the key users of the findings.  If data from the survey or an



interpretative report of the findings must be provided in time to



meet a deadline, the date of and reason for the deadline should



also be stated.



 



Any germane reference material that would be helpful to potential



offerors in planning the broad outline of the proposed survey or in



estimating time, skill, or cost requirements should be cited and



also included in the RFP package.  If the material cannot be



provided with the RFP, information on where and when it can be



accessed or obtained should be supplied.  Appropriate reference



material might include such information as estimated numbers of



eligible respondents, geographic distribution of eligible



respondents, and the methodology and results of any related



research that has been conducted by or is known to the sponsoring



agency.  However, materials which



 



                                20



 



 



 



 



 



would be of interest only to the successful offeror should not be



included with the RFP.



 



Although RFPs sometimes state anticipated level of effort in terms



of person-years or similar proxies for dollar estimates, the



Subcommittee's interviews with contractors indicated that these



proxies are less helpful to them than dollar estimates would be



because of wide variations from agency to agency in the dollar



amount one person-year signifies.  If agency procurement policies



permit, it may be desirable to inform potential offerors of the



approximate amount of funds available for the contract.  Providing



offerors with this information would help avert some of the



difficulties that might otherwise be encountered in the contractor



selection stage, such as the problems that arise when the cost



estimates for all of the technically sound proposals received by



the sponsoring agency substantially exceed the funds available for



the contract.  If some but not all of the technically sound



proposals fall outside the anticipated price range, this reduction



in competition may result in a real but less obvious disadvantage



to the sponsoring agency.



 



 



Request for Information About the Offeror



 



The RFP should request the relevant information about the offeror. 



It is not sufficient for an RFP to simply define the requirements



for the technical proposals.  Before a contract is awarded, the



reliability of the contractor should be established.  When



appropriate, the following specific requests should be responded to



by the offerors before a contract is awarded.



 



Description of experiences.  The following list contains areas



where it could be important to obtain specific descriptions of the



offerors' experiences:



 



-    frame refinement processes,



 



-    probability sample designs, especially for large-scale



     surveys,



 



-    data collection methodology (including callback procedures),



 



-    questionnaire development and testing,



 



-    estimation procedures, including nonresponse adjustments,



 



-    variance computation procedures, especially for complex



     samples,



 



-    training and supervision of interviewers, coders, and others,



 



-    analysis of data from large-scale surveys,



 



-    data processing,



 



-    quality control, and



 



-    publications.



 



 



Examples of previous relevant work.  The examples of offerors' work



should include questionnaires, sampling documentation, instruction



manuals, and reports containing the results of the projects.  The



agency should request information on all previous work pertaining



to the subject matter areas for a specified period of time.  The



time period should be such that enough material is received but



that the offeror does not inundate the agency with paper.  In most



cases, five years should be a reasonable period for both the



offerors and the agency.  Included with this information should be



the name and current telephone number of the sponsor's



representative (or government project officer).  At least a sample



of these references should be verified by the agency before the



award of any contract.



 



                                21



 



 



 



 



 



Staff experience.  This should include the experience of the



current staff.  This could differ substantially from the company's



experience.  Their background in statistical methodology, previous



contracts, and subject matter areas, especially the ones related to



the RFP, should be stated.



 



Key project staff.  The names of key project staff members should



be required along with the percent of time each one will work on



the project.  Their resumes should be included with the proposal. 



It may be desirable to have a key personnel clause stating which



staff members cannot be replaced on the project without government



approval.  Letters of intent should be provided by key persons who



are not members of the offeror's firm.



 



Field staff.  The interviewers' field operations background in the



primary sampling units (PSUs) in the frame from which the sample



will be drawn should be requested.  Also their experience in



handling nonrespondents should be stated.  The background of



supervisory and clerical field staff should also be requested.



 



Conflicts of interest.  The RFP should contain a section requesting



information that will allow the agency to determine if any possible



conflicts of interest exist.  This section could include questions



about the offeror regarding other clients, activities of staff



members, subsidiaries, or parent companies.



 



Facilities.  In the RFP, the agency should request information



about the potential offerors! facilities in order to make certain



they are adequate to complete the contract.  This is especially



true of computer facilities.  The offeror should have the computer



capabilities to handle all data processing requirements.  If



necessary, this should include a requirement of compatibility with



the agency's data processing facilities.



 



Subcontracting.  If an offeror plans to use a subcontractor in any



aspect of performance under the contract, detailed information



should be provided on their experience, resources, and facilities. 



If subcontracting is anticipated but there is no identified



subcontractor, standards for subcontractor selection should be



specified.



 



 



Survey Design Checklist



 



The purpose of this section is to present a descriptive checklist



which contains topics that should be addressed or considered when



writing the survey design portion of an RFP.  As was stated



previously, the amount of specification is dependent on many



factors.  Ideally the amount of detail that each item in the



checklist requires will depend on the type of study to be



conducted.  For this reason, the following list should be used only



as a guide and the appropriateness of each item should be judged



separately for each RFP.  However, if a design topic is not



specified in detail, it should be mentioned as an item to be



discussed by the offeror.  If an item is not mentioned in the RFP,



the agency might not be able to use it as a criterion for



evaluating the proposals.  In addition, the importance of



consulting a qualified sampling statistician before proceeding with



this section cannot be overemphasized.



 



 



Design and selection



 



     Definition of the target population and key subgroups.  The



     objectives of the survey should define the population that is



     intended to be covered.  However, it may be necessary to



     eliminate certain segments of the population and thereby



     redefine the scope of the survey because of practical



     problems.  Examples would be excluding Alaska and Hawaii



     because of the high cost of collecting information by personal



     visit, excluding nonrepriceable items in the production of a



     price



 



                                22



 



 



 



 



 



     index, and excluding nontelephone households when telephone



     interviewing is the primary method of data collection.



 



     The survey results will apply only to the population sampled. 



     However, it might be possible to obtain estimates for the



     excluded populations by using alternative collection methods



     or by using auxiliary data which are correlated to the



     information gathered in the survey.  The RFP could require



     that proposals include a method of estimating for the excluded



     population.  Any conclusions for these groups would have to be



     stated separately and their limitations would have to be



     noted.



 



     A description of any subpopulations for which estimates are



     desired should be included.  Examples could be geographic and



     demographic detail.



 



     Sampling frame.  Ideally the sampling frame should fully cover



     the target population.  Every effort should be made to.ensure



     that the source is as current and as complete as possible. 



     Consideration should be given to requiring procedures for



     updating the available source, including merging information



     from different records and removing duplicates.  If more than



     one frame is available, a decision must be made on which one



     to use or whether to use multiple frames in the survey.  If



     multiple frames are used, a method to identify and handle



     overlap in the frame must be developed.



 



     Sample selection.  The RFP should specify that a probability



     sample be selected for the survey and that the sample design



     be adequately defined.  This includes a description of the



     proposed sampling plan for each stage of sampling.  The



     details of     this description should include the following



     topics for each stage of selection:



 



     -    Definition of the sampling unit.  A sampling unit is a



          cluster of one or more elementary (or observation) units. 



          Each elementary unit should belong to exactly one



          sampling unit.  In some RFPs the sampling units for some



          of the stages of selection (for example, the primary



          sampling units or PSUS) are specified.  Any sampling



          units that are not specified in the RFP should be clearly



          defined in each proposal.  Examples of sampling units are



          counties, enumeration districts, households, persons,



          companies, and retail outlets.



 



     -    Number (or expected number) of sampling units to be



          chosen.  In many cases the RFP gives the target final



          stage sample size (for example, 5,000 completed



          interviews) and sometimes the first stage target sample



          size (for example, 100 PSUS).  A proposal should indicate



          all selection stage sample sizes not specified in the



          RFP.



 



     -    Specific method of selecting units from the frame.  The



          method of selecting sampling units at a given stage (for



          example, simple random sampling, systematic sampling, or



          stratified sampling) should be clearly specified so that



          the probabilities of selection of the sampling units can



          be derived.  In addition, any variation from strict



          probability sampling that an offeror feels is required



          should be clearly described and justified in the



          proposal.  Examples of cases for which variations from



          probability sampling may be appropriate are unavoidable



          coverage deficiencies, or the selection of a major brand



          to represent a product line in a price survey.



 



          Offerors should base their choices of sampling units,



          sample sizes, and selection methods for the various



          stages of a sample design on optimization. considerations



          (that is, maximum precision per unit of cost).  The RFP



          should require that proposals include a justification for



          all the sample design choices made.



 



                                23



 



 



 



 



 



          It is essential that at least one sampling statistician



          with experience in the particular area of interest assist



          in the preparation of the sampling portions of the RFP.



 



          It should be noted that unless probability sampling is



          required in the RFP, a proposal cannot be considered



          technically unacceptable just because it allows for some



          form of nonprobability sampling.  This could cause



          problems during the contractor selection procedure.  In



          addition, the Office of Management and Budget (OMB)



          requires justification if probability sampling is not



          used.



 



     Sampling error.  Two different approaches are commonly used in



     RFPs for sample surveys to specify the level of effort desired



     by sponsors to provide estimates that are sufficiently



     reliable for their purposes.  One method is to specify the



     sample size; the other is to specify the precision (that is,



     desired minimum level of sampling error) for estimates of one



     or more key variables.  It is not desirable or practical to



     specify both sample size and precision in the RFP, since the



     desired precision may be impossible to achieve with the



     specified sample size.  Whichever is specified, the other



     should be an important factor in evaluating the proposal.  The



     justification for using one method over the other is dependent



     on the study to be done and the resources available.



 



     The RFP often specifies the target sample size, usually in



     terms of the number of completed questionnaires or interviews. 



     The derivation of a target sample size may be based on cost



     considerations or on the approximate precision levels desired. 



     If a sample size is specified in the RFP, it is important to



     clearly define the final stage unit and whether or not the



     sample size is in terms of original selections, inscope



     selection, or completed interviews.  Any required sample sizes



     for population subgroups should be clearly stated.  Because of



     the different components of the sampling error, it may be



     necessary to require a minimum number of sampling units at



     each stage of selection.  Otherwise the resulting sample,



     although meeting all the specified criteria, might not yield



     acceptable results.  Finally, if the sample size requirements



     are in terms of completed interviews, the RFP should include a



     precise definition of what data must be collected before an



     interview is complete.



 



     If the sampling error is to be specified then a decision must



     be made whether it will apply to one key variable or more than



     one key variable.  If it applies to more than one, will there



     be a controlling variable, that is, the design which meets



     requirements for that variable will exceed requirements for



     all other variables.  In addition, it must be made clear



     whether the precision requirements are in terms of the



     standard error, the coefficient of variation (relative



     standard error), or 95 percent confidence levels (+ 2 standard



     errors).  Any misinterpretation of these requirements can lead



     to gross differences in the proposed sample sizes.  Also it



     must be made clear whether the required minimum sampling error



     will apply to specific subpopulations "or just the overall



     population under consideration. A qualified sampling



     statistician should be consulted to assist with the decisions



     made in these areas.



 



 



Data collection, questionnaire development and testing



 



This section of the RFP, more than any other, will require input



from the.people who handle the agency's OMB clearance process.  In



addition to knowing the agency's procedures regarding data



collection they are familiar with OMB's regulations. (See appendix



11 for a discussion of OMB's clearance and, Information Collection



Budget



 



                                24



 



 



 



 



 



requirements.) Their assistance can help in making the clearance



process go as smoothly as possible.



 



When preparing an RFP, the following issues concerning data



collection, questionnaire development, and testing, should be



considered.  While they may not apply in all cases, when they do



they will have an effect on the contract price.  Therefore, in an



attempt to avoid any misinterpretation by potential offerors, these



issues should be addressed when applicable.  Depending on the



situation, they can be either explicitly defined or stated in



general terms and then used in the evaluation of the proposals.



 



     Method of data collection.  The three most common types are



     personal (face-to-face) interview, telephone interview, and



     self-administered (primarily mail) questionnaires.  Each has



     specific advantages and disadvantages which must be



     considered.  They may also be used together in a survey.  An



     example would be a self-administered questionnaire and a



     personal interview follow-up.  It is important to make certain



     that the desired information can be collected successfully or



     more accurately and that the target response rates can be



     achieved by the proposed method or combination of methods. 



     Mail questionnaires seldom produce adequate response rates



     without some form of telephone or field follow-up.



 



     Data specifications or information to be obtained.  They



     should be listed in sufficient detail to indicate what the



     questionnaire content should be and what the amount of effort



     required in developing and testing the questionnaire will be. 



     They should be complete and realistic.  When the survey is



     covering a new topic, the sponsor should identify the



     variables for which data are needed, but should allow the



     contractor, once selected, to play a major role in developing



     the specific questions needed to get data on these variables.



 



     Pretest.  The need for a pretest is paramount in any survey. 



     Without it the chances of achieving good results are minimal. 



     The three following primary issues arise in a pretest which



     must be considered in any survey: sample size, probability



     sampling, and the survey time schedule.



 



     Sample sizes should be sufficiently large to learn about



     significant problems that may be encountered.  For this



     reason, specifying a pretest of nine respondents so that the



     pretest does not have to be cleared through OMB should be



     avoided.  Furthermore, clearance at an early stage of



     development may identify possible overlap of the planned



     survey with work of other agencies or may bring to light



     experience in other surveys that will help to avoid repetition



     of past mistakes.  It is better to determine the sample sizes



     needed without reference to clearance requirements and to



     ensure that the survey development schedule allows enough time



     for necessary clearances.



 



     While probability sampling at all levels of selection is



     desirable for nearly all surveys, it may not always be



     indicated for pretests.  Purposive sample designs are



     frequently used in pretests to ensure inclusion of important



     subpopulations.  Probability selection methods at the final



     stages (that is, households within block) are still desirable



     to avoid selecting the "easy to get" units.



 



     The survey time schedule should allow for adequate time and



     professional staff to both conduct and analyze the findings



     from the pretest and to use the findings effectively in



     subsequent stages of development.



 



                                25



 



 



 



 



 



     The following is a minimum list of topics which should be



     addressed in the planning of a pretest:



 



     -    Should exploratory interviews be conducted? The purposes



          might include identifying items that are difficult or



          sensitive to answer, determining if the data



          specifications are complete, or ascertaining what



          records, if any, are needed and if they are available.



 



     -    Will the pretest be conducted in waves? That is, will the



          pretest sample be sent out in parts, each of which



          contains questionnaires modified based on the findings in



          the previous part?



 



     -    Will the effect of the sequence of the questions be



          considered?



 



     -    How long is the average time for pretest interviews



          likely to be?



 



     -    Will the pretest determine the required experience level



          of the interviewers or is this one of the requirements in



          the RFP?



 



     -    What records, if any, will the respondent need to refer



          to and will they be reasonably accessible?



 



     -    What results are necessary before advancing to the next



          stage of the study?



 



 



     Respondents' obligation to reply.  Any information about the



     respondents' obligation to reply that would help the offerors



     should be included in the RFP.  Most surveys are voluntary. 



     However, if the survey is mandatory, if a response is required



     in order to obtain or retain benefits, or if informed consent



     is required, this could greatly affect the proposals and



     should be included.



 



     Length of interviews.  If the contractors are to develop the



     questionnaire, a ceiling on the interview time should be set. 



     A respondent may agree to cooperate only if this time is kept



     to a minimum.  Care should be taken to obtain the required



     information in this limited time.  To do this, a determination



     of the minimum number of questions needed to accomplish the



     survey's purpose should be made.



 



     Confidentiality.  Any questions regarding confidentiality or



     the Privacy Act system of records should be addressed in the



     RFP.  If there is a promise of confidentiality in the survey,



     the contractor may have to take special provisions to allow



     for it.  These could include locking up data and only allowing



     access by authorized personnel.  Special provisions regarding



     the security of the questionnaires might also be needed.



 



     Payments to respondents.  If there is to be financial



     compensation to respondents for completing a questionnaire,



     the offerors should not only be made aware of this fact, but



     also the reason for it.  Following are several reasons (each



     of which could have different effects on the bids) are:



     studies have shown that the success of the project depends on



     the agency providing compensation, or the data collection



     requires a significant amount of work on the part of the



     respondents and they should be compensated for their time.



 



     In addition, OMB and many Government agencies require a



     description or justification of any remuneration to persons



     supplying information.  If the success of the survey depends



     on these payments, approval should be obtained from both the



     Department and OMB before the contract is signed.



 



     Advance notice to respondents.  Many surveys require advance



     notice to the respondents.  This is so they can consult their



     records and documents, thereby



 



                                26



 



 



 



 



 



     making the interview as useful as possible.  If this time is



     needed, the offerors should be made aware of it so they can



     plan their resources accordingly.



 



     Dates for data collection.  There may be reasons that specific



     beginning and ending dates for the data collection must be



     set.  These reasons could be that the particular event being



     measured will only occur during those times or that the



     collection is not allowed while the decennial census is being



     conducted.  Make certain the reasons are justified.  Moving up



     the time schedule for a survey is usually not a sufficient



     justification.



 



     Training.  It must be determined who will be responsible for



     training interviewers and preparing the necessary materials. 



     In addition, what specific types of training requirements



     exist and will the sponsor participate in the training.,



 



     Response rate.  The required minimum response rate should be



     specified along with the method which will be used to



     calculate it.  Included with this information should be the



     number of completed questionnaires required and the criteria



     for defining a completed questionnaire.  The procedures for



     field follow-ups should be stated.  These should include any



     time-of-day and day-of-week requirements along with the number



     of follow-ups and a requirement that the contractor complete



     all specified follow-ups, even if the required minimum



     response rate is met.  In addition, there should be a



     provision that the contractor must exceed the minimum number



     of callbacks if they are required to meet the minimum response



     rate.  Of course, any additional field follow-ups needed to



     meet the response rate will have a cost associated with them. 



     This tradeoff should be considered while decisions on the



     survey requirements are being made.  No attempt is made in



     this report to determine what the minimum response rate should



     be for every survey; each survey will have factors that will



     have an effect on this number.  The best advice is to check



     current policy and consult a survey statistician before



     writing this section of the RFP.



 



     Quality control.  The quality controls that will be required



     in the data collection process and the validation requirements



     must be addressed in the RFP.  Who will be conducting these



     tests must also be specified.  It is usually best to indicate



     that quality control is a necessary part of the survey and let



     the offerors indicate how they propose to control quality at



     each stage.



 



     Interviewer qualifications.  If the interviewers must possess



     certain qualifications because the survey has special



     circumstances that require them, they must be specified in the



     RFP if they are to be used in the evaluation process.  These



     could include expertise in the specific field of study,



     knowledge of a foreign language, previous data collection



     experience, or knowledge of survey sampling.



 



 



     Additional questionnaires.  If there are to be different



     versions of the questionnaire , these requirements must be



     specified along with the reasons for needing them.  They could



     be necessary due to the questionnaire must be tailored to the



     various regions of the country because of content or other



     considerations.  Examples are asking about citrus fruit



     production only in the appropriate areas or having a foreign



     language version of the questionnaire in areas where English



     is not the predominant language.



 



 



Tabulations and reports



 



The final products and quality of the completed work should be



monitored very closely.  To facilitate the monitoring process,



certain deliverables should be



 



                                27



 



 



 



 



 



specifically requested in the RFP.  A schedule should be set up and



rigidly followed.  The following list contains several of the more



important deliverables.



 



     Progress reports.  Progress reports should be required



     according to an agreed upon plan.  The p!an should specify



     content as well as time of delivery.



 



     Draft reports for agency review.  The RFP should indicate that



     a draft of each required task report will have to be submitted



     for approval before the final report is written and the task



     is considered complete.



 



     Tabulations required.  When it is feasible, the format and



     content of all required tables should be included in the



     proposals.  The RFP should also state whether the contractor



     will be expected to perform analyses of the data and in what



     form they must be delivered.



 



     Data descriptions.  All data supplied should be accompanied by



     sufficient documentation.  For example, the formats of any



     data tapes must be required and the formula used to calculate



     any weights must be supplied with the weights.



 



     Survey documentation.  The RFP should require a complete



     detailing of the methodology used.  This report should include



     the contractor's impressions of the effectiveness of the



     survey procedures, suggestions for improved methods in future



     surveys, and all appropriate technical materials.  Included



     would be the following: copies of all forms; documentation of



     all (especially sampling and data collection) procedures in



     all phases; all adjustments for nonresponse and missing data;



     survey estimates and estimates of sampling errors; full



     accounting of all data collection results, including response



     rates and interviewer validation results; coding error rates,



     along with all quality control procedures used; methodological



     findings; and a description of the procedures used in handling



     confidential data.



 



     Analytic reports.  If the agency wants the contractor to



     produce one or more reports analyzing the findings of the



     survey, such a report should be requested in the RFP. 



     Frequently this analysis and the resulting report are done in-



     house.  If this is to be the case, it should be stated in the



     RFP.  Analytic reports should contain a complete analysis of



     the survey data along with any appropriate conclusions and



     recommendations.



 



 



ROLE OF THE PROJECT OFFICER



 



Monitoring functions of the project officer that directly affect



the performance of the contractor should be clearly discussed in



the RFP.  This allows the offerors to know in advance how the



project officer intends to monitor the contract.



 



Following are several important points which should be defined. 



The RFP should list all areas where reviews and approvals will be



required.  These could include sampling plans, questionnaires,



training materials, and data processing requirements.  If site



visits are planned or meetings are to be scheduled, they should be



mentioned in the RFP.  The RFP should clearly specify the role of



the Project Officer regarding attendance at any training that is



planned.  Detailing the role of the project officer in the RFP



allows the offeror to plan for additional resource requirements and



permits a smoother relationship in the performance of the contract.



 



                                28



 



 



 



 



 



EVALUATION CRITERIA



 



Every RFP must include the criteria to be used in evaluating the



technical merits of the offeror's proposal.  These will be listed



in descending order of points to be allocated to the item and may



show the actual number of points.



 



The technical evaluation criteria need to cover the critical



elements to be used in determining the acceptability of the offeror



and the proposal submitted for the contract survey. 



Differentiation between acceptable and unacceptable proposals and



offerors is made on the basis of these criteria.  If a vital



element of approach or capacity is not reflected in the evaluation



criteria, it may not be possible to distinguish among offerors or



their ratings based on the issue.  All too often program staff



members concern themselves with technical aspects of the survey



(that is, sampling, interviewer training, field work, editing and



coding) and ignore the need to translate these specifications into



the criteria that will be used to evaluate the proposals received. 



The criteria will be used in the selection process.  The survey



specifications are the basis on which the offeror provides the



proposal that will be the subject for evaluation.  Proposals can



only be evaluated based on the published criteria.



 



                                29



 



 



 



 



 



 



 



 



              Chapter 4. CONTRACTOR SELECTION PROCESS



 



 



INTRODUCTION



 



The work to be done under the contract will have been described in



the Request for Proposal (RFP) as well as the material and



presentation to be submitted by the offeror as a basis for



evaluating the response.  The criteria and scoring to be used in



this evaluation are also included in the RFP, thus giving each



offeror the same knowledge about evaluation of the technical



proposal.  It is in this sense that contractor selection starts



when the decision is first made to issue a competitive RFP.  By



that time, there should have been consideration of the possibility



of doing the work in-house or through another Federal agency, of



contracting with an 8(a) firm, or of limiting competition to small



business firms under the set aside program.



 



The decision by the contracting officer to issue an open



competitive RFP instead of taking one of these other options



involves decisions and justifications within the agency as to the



conduct of the work or some basic characteristics that potential



contractors must have.  These may relate to performance or capacity



requirements or experience or reputation requirements of potential



contractors.  These characteristics should be part of the



specifications included in the RFP.  Since the agency has made the



decision to go competitive based on specific requirements of the



proposed contract as these relate to capacities of potential



offerors, it is incumbent upon the writers of the RFP to make this



information available to potential offeror organizations.  It will



be useful to them in making their decisions as to whether to



respond to the RFP.  The mechanism for doing this is the RFP



itself.  An adjunct to this that is sometimes used is a Bidders'



Conference as discussed in Chapter 3.



 



The thinking that resulted in a competitive RFP should provide the



framework of offeror capacities and expected activities performance



that will be used to evaluate responses.  The remainder of this



chapter is based on the premise that the RFP discussed what was



being looked for in the contractor selection process.



 



 



COMPONENTS OF THE SELECTION PROCESS



 



The response to an RFP is submitted in two separate parts, the



Technical Proposal and the Business and Cost Proposal.  These are



evaluated separately.  Evaluation of the Business and Cost Proposal



is the responsibility of the contracting officer and is done in



terms of the appropriateness and documentation of costs relative to



resources to be devoted to the effort.



 



The technical proposal is evaluated in terms of the material on



project conduct and management, the staff and other resources to be



devoted to the project, and the qualifications of the staff.  If



the RFP was specific in the description of the work to be done, the



capacity and experience of the organization and staff will probably



be weighted more heavily in the evaluation criteria.  If the RFP



was more general in the description of the work to be done, the



approach to conducting the project proposed by the offeror will



probably be weighted more heavily.  In any case, the RFP should



 



                                31



 



 



 



 



 



have specified what the offeror was to submit as a basis for



technical evaluation and the criteria to be used.



 



The contracting officer has the responsibility to ensure that the



technical evaluation is conducted in a way that satisfies the



Federal Procurement Regulations and agency requirements.  The



program office bears responsibility for the conduct of this review. 



The guidelines in the regulations governing technical review



require establishing a selection process that will insure fair and



impartial treatment of all offerors, the selection of sources whose



performance is expected to meet the contract requirements at a



reasonable cost or price, and avoidance of arbitrary or capricious



behavior, inequitable treatment, or undue influence.  It is



essential that persons involved in the evaluation process have no



conflict of interest with this function and that no information



concerning the evaluations be released to another individual except



one who is participating in the process.



 



The project officer for the contract is responsible for proposing



the names of individuals to be on the technical evaluation panel to



his/her superiors within the program office.  The project officer



is expected to be a member of the panel and may serve as its



chairperson.  The panel is usually composed only of government



personnel.  Outside evaluators are used only under unusual



circumstances, such as when required expertise for evaluation is



not available within the government.



 



The project officer should consult with the contracting officer,



who bears official responsibility for the overall contracting



process, on aspects of the technical review process and current



standards and requirements of the Federal Procurement Regulations



and agency regulations on technical evaluation.  In general,



project officers are given a significant amount of flexibility and



latitude in establishing panels and review plans and in conducting



technical evaluations.  However, a list of persons to serve on the



technical evaluation panel and a plan for the technical evaluation



must be submitted to the contracting officer before the technical



proposals are released to the project officer for panel evaluation. 



To the extent possible, members of the project team should be



included on the panel.  The contracting officer should attend the



initial panel meeting to clarify the basic rules and considerations



under which the technical review will take place.  Beyond that, the



prime requirements are that the process be equitable, that the



basis for the evaluation of each proposal be documented, and that



the evaluation be conducted in terms of the criteria published in



the RFP for contractor selection and the evaluation plan.  The



purpose is to make the contractor selection process as fair and



open to competition as possible.  The process is designed so that



all potential offerors have the same base of information on which



to assess their qualifications for competition for award of the



contract, to know the basis on which a selection will be made and



to have the selection based on preset criteria.  The goal of this



approach is to eliminate the actuality and, to the extent possible,



the appearance of bias and preselection.



 



In assembling the technical review panel, the project officer



should try to obtain a balance of areas of survey expertise and



subject matter interest as related to the purpose of the contract



and the needs of the agency.  For example, this might include



experts on survey methodology and sample design and experts



representing interrelated content and program areas.  Expertise and



experience available in other agencies should be used as a



resource.  Representation of other agencies is also advantageous



when there is overlapping program interest.  A panel that is biased



in composition toward a given expertise may distort the contractor



selection in that direction independent of the balance built into



the technical evaluation criteria and" plan.  A minimum of three



persons should be on the panel to allow for more effective



 



                                32



 



 



 



 



 



discussion and resolution of disagreements.  The project officer



should arrange for a meeting place where panel members can discuss



the merits of each of the proposals in private.  It is important



that there be continuity over the full process of proposal review



and contractor selection.  Individuals selected for the panel



should be available to review later modifications and additions as



well as the initial responses to the RFP.



 



 



TECHNICAL EVALUATION OF CONTENT AND CAPABILITY



 



The technical evaluation process is designed to carry out the



objective consideration of the technical merits of proposals



submitted in response to an RFP in terms of the published



evaluation criteria contained in the RFP.  By the end of the



technical review process each proposal will be identified as being



either acceptable or unacceptable.



 



A proposal can be determined at the outset to be technically



unacceptable without detailed review if it fails to meet a specific



expressed requirement of the RFP.  As an example, if the RFP



requires probability sampling for the planned statistical survey



and an offeror proposes only an alternate sampling approach, the



proposal can be considered an inappropriate response to the RFP and



technically unacceptable.



 



A technical evaluation plan should be prepared by the program



office.  It should include information for the panel members based



on the specific criteria, as included in the RFP, to be used in



rating proposals.  The evaluation plan should provide more specific



guidance for the panel on details to be looked for within an



offeror's proposal in scoring it under various elements of the



evaluation criteria.  The plan will specify the number of points to



be allocated to each part of the evaluation criteria.  In general,



these criteria should deal with the offeror's approach to the



contract tasks and the organizational ability to perform those



tasks.  The evaluation criteria for ability or capacity to perform,



which should have been built into the RFP, should include some



forms of the following considerations: demonstrated organizational



expertise in the subject area, staff availability and their



expertise in the subject area, past experience in surveys under



contract, ability to meet special requirements, and, availability



of needed resources.  The references for past surveys under



contract should be checked.



 



Before discussing the proposal in a group, each member of the



evaluation panel should individually read each proposal and prepare



a tentative indication of its strengths and weaknesses and a



preliminary score on each evaluation criterion using the rating



sheets in the technical evaluation plan.  Adequate time should be



allowed for this key aspect of the process.  After this has been



done, the panel members should meet to discuss the relative merits



of each proposal and try to arrive at a consensus opinion on the



strengths and weaknesses of each proposal and the possibilities for



correcting the weaknesses.  The evaluators should then individually



rate each proposal.  The individual scores of the panel members



have to be combined into a single ranking of all of the proposals. 



While a numerical average of the panel members ratings for each



proposal is generally used in developing a ranking, this is not a



requirement and depends on the rating plan used.  The panel then



should identify each proposal as acceptable or unacceptable. 



However, this can not be done by the use of predetermined cutoff



scores.



 



The definition of a proposal as acceptable or unacceptable is based



on technical consideration of the offeror's proposal in terms of



the requirements of the RFP "as assessed, using the evaluation



criteria published in the RFP.  A determination of



 



                                33



 



 



 



 



 



unacceptability must be supported by concrete factual statements



consistent with the evaluation criteria and indicate that the



proposal's deficiencies are so major it could not be brought up to



an acceptable level without the equivalent of a new proposal being



submitted.



 



The technical evaluation panel members will prepare and sign a



report to the contracting officer showing the ranking of the



proposals and identifying each as acceptable or unacceptable.  The



individual rating sheets should be included.  A narrative



assessment of each proposals strengths and weaknesses must be a



part of the report.  The report should also include any



considerations related to the selection of sources for negotiation



and award and any specific issues and questions for discussions.



 



 



DISCUSSIONS WITH OFFERORS IN THE COMPETITIVE RANGE



 



After the technical evaluation and analysis of the business



proposals, the contracting officer will determine which proposals



are in the competitive range as a basis for conducting written or



oral discussions.  Discussions are communications essential for



determining a proposals acceptability or to provide the offeror



with the opportunity to revise or modify its proposal based on



deficiencies or weaknesses noted in the evaluation process.  More



than one round of discussions may be held by the contracting



officer with offerors depending on the situation.  The



determination for inclusion in the competitive range will be based



on price or cost and technical and other important factors as



stated in the RFP.  The competitive range will include all



proposals that have a reasonable chance of being selected for



award.  In making this determination, the contracting officer may



request the technical evaluation panel to review the cost or price



data, or request comments from the project officer on labor and



other resources needed to conduct the project.  Discussions will be



held with all offerors in the competitive range.



 



The competitive range may be narrowed after initial discussions by



dropping an offeror if a revised proposal is determined to no



longer have a reasonable chance of being awarded.



 



All discussions with offerors are under the control of the



contracting officer.  Offerors are given the opportunity to correct



deficiencies, resolve uncertainties and errors, and to submit



revised material based on these discussions.  However, no



information is to be given to offerors in these discussions on



other offerors' proposals or the evaluations, or on competitive or



estimated prices other than to indicate that the Government



considers its price too high.  No specific help should be given an



offeror in rewriting the proposal.



 



At the end of the discussions with all offerors in the competitive



range, each offeror will be notified that the discussions have



ended and that they have an opportunity to submit a "best and



final" offer by a common cutoff date, including a revision of an



earlier offer or a confirmation that a prior offer is a "best and



final" offer.  These "best and final" technical and business



proposals are subject to evaluation on all significant factors by



the contracting officer and project officer and, if necessary, a"



full technical rescoring and reranking by the technical evaluation



panel.  At the end of this process, the contracting officer will



select for award the proposal that offers the Government the



greatest advantage, price, and other factors considered.



 



While the final selection decision is the responsibility of the



contracting officer, this' decision is usually made after



consultation with the project officer.  Since it will be



 



                                34



 



 



 



 



 



the project officer's responsibility to monitor technical



performance, he or she should be part of the assessment of the



relative importance of technical quality and price differences. 



The analysis of business management and price factors is the



contracting officer's domain.  Although price or cost is one factor



in the final decision, whether certain items are allowable and the



assessment of the basis for estimating cost is an administrative



issue.  Usually, whether a contract is to be a fixed-price or a



cost-type is specified in the RFP.  However, the final decision to



use a fixed-price or a cost-type contract is a contracting officer



decision and the ability to estimate certain costs is a factor in



this decision.



 



 



THE FINAL AWARD PROCESS



 



.After the successful offeror has been selected, the contracting



officer may hold limited negotiations with this offeror.  However,



these negotiations cannot relate to any factor which could have an



effect on the Selection process.  It is the Contracting Officer's



responsibility to ensure that negotiations do not change the RFP,s



requirements nor make any other changes that would impact on the



selection process.  If there is a material change in requirements



because of negotiations, the competition must be reopened to all



offerors who were in the competitive range.  A significant increase



in the offeror's cost proposal may also require reopening of the



competition.



 



After any negotiations are held, the final contract document is



prepared.  In preparing the contract document, including all the



agreed upon terms and conditions, the option should be considered



of specifically incorporating the offeror's final proposal to avoid



any question as to what was offered and accepted.  This can be of



particular value when the RFP specified only an area of survey



concern and not the specific approach to be used.



 



After the contract has been awarded, the unsuccessful offerors will



be notified.  Unsuccessful offerors can, and often do, request



debriefing.  The project officer and possibly other members of the



technical evaluation panel may be. called upon in a requested



debriefing of an unsuccessful offeror.  It is at this point that



the documentation of the basis for technical evaluation can be of



critical importance in explaining why the offeror's proposal was



not selected for award and in establishing that the competitive



process was appropriately conducted.  Unsuccessful offerors may



formally appeal a contract award.  However, a well documented and



unbiased basis for selection will reduce such appeals of the



selection process.



 



                                35



 



 



 



 



 



 



 



 



 



                 Chapter 5. POST-AWARD ACTIVITIES



 



 



INTRODUCTION



 



Following award of a contract, both the contractor and the



Government have responsibilities that must be fulfilled.  Post-



award activities are all activities that occur or should occur once



the negotiated contract has been signed by both parties. it is the



joint responsibility of the contractor and of the Government to see



that work under the contract is accomplished in a timely and proper



manner.  Government policy and procedures designed to ensure that



the contractor and the Government staff understand their mutual



responsibilities after the contract is awarded are set forth in the



Federal Procurement Regulations.



 



The major elements of post-award contract administration include



monitoring of performance (including review and approval of



products, services, or reports), contract modifications, approval



of subcontracts, property administration, and contract closeout. 



Aspects of these elements may be performed by various Government



personnel, but the two principal persons are the project officer



and the contracting officer.  It should be made clear at the outset



of the contract exactly what the respective roles of these two



individuals are with respect to contract administration and



monitoring.  Typically, the contracting officer, the person who



actually signed the contract on behalf of the Government, delegates



to the project officer responsibility for monitoring technical



aspects of work under the contract.



 



In most instances, a contract identifies tasks, subtasks,



schedules, review processes, criteria identifying skills,



techniques and methodologies to be used, and a schedule of



deliverables.  These requirements are usually included in the



Request for Proposal (RFP) and are addressed specifically in the



contractor's technical proposal, which is often incorporated into



the final negotiated contract.  The project officer must monitor



closely all aspects of work to ensure that it is carried out as



specified.



 



In addition to technical and procedural requirements for conduct of



the work associated with the survey, the contract should also



include performance requirements and a description of measures to



be used in determining if certain requirements, such as the



specified response rate, have been met.  The contract is the basis



for monitoring and assessing contractor performance.  A project



officer should have a complete copy of the final negotiated



contract for a survey, should read it completely to ensure that it



contains all that it should contain, and should reread it



periodically to insure that all of its terms and specifications are



being followed as specified.



 



Most post-award responsibilities of the project officer relate



directly or indirectly to technical aspects of the work.  The role



of the project officer is discussed in the following material,



which emphasizes activities related to insuring that the product



resulting from the contract is of the highest quality possible.



 



                                37



 



 



 



 



 



GENERAL TECHNIQUES FOR MONITORING CONTRACTOR PERFORMANCE



 



Meetings with the Contractor



 



To the extent possible, there should be regular meetings between



the project officer and the contractor.  For a large, complex



contract such meetings may have been included in the statement of



work and travel funds provided for contractor participation.  If



such meetings are not possible or are only possible on an



infrequent basis, conference telephone calls may serve as a



reasonable, though less satisfactory, substitute.  These meetings



provide a forum for structured, though somewhat informal, review of



activities and can offer an early alert to technical problems or to



slippage of the project schedule.



 



 



Maintenance of Files



 



Ideally, the same project officer will serve from the inception of



the project through its completion.  Experience shows, however,



that this is often not the case and that project officers may



change several times during the course of a project.  Thus, it is



essential that the project officer maintain complete, organized



files pertaining to the project.  Included in the files should be



the following: all formal contract documents, including the



negotiated, signed contract and any modifications to it; copies of



all correspondence relating to the project between the contractor



and the project officer; notes or minutes of any meetings or



substantive telephone conversations with key contractor staff;



copies of all progress reports and other materials received from



the contractor; copies of or notes on all communications with the



contracting officer; and documentation of any other significant



materials, including various approvals such as required



departmental clearances and Office of Management and Budget



approval.  Project files are also important as a source of



documentation if there is a disagreement between the contractor and



the project officer about whether information was communicated, and



if so how and when.



 



 



ADMINISTRATIVE RESPONSIBILITIES OF THE PROJECT OFFICER



 



Review of Contractor Progress Reports and Submissions



 



The project officer should, as soon as possible upon receipt, read



and provide comment, preferably in writing, to the contractor on



all interim progress reports and other documents received.  No



matter how often there are conversations between project officer



and contractor, they do not replace the formal progress reports



that should be required by that contract.  These reports are the



appropriate forum for the contractor to officially inform the



project officer of project accomplishments and problems.  Reports



should contain sufficient information to determine that work is



being accomplished on or ahead of schedule and that it is being



completed in a satisfactory manner.



 



It will facilitate the project officer's review, and perhaps even



the contractor's preparation, of progress reports if a mutually



agreed upon outline is followed for each periodic progress report. 



Such an outline may even have been specified in the contract.  In



addition to a description and summary of activities since the last



report, topics to be addressed might also include a statement of



any problems encountered and their resolution, personnel changes, a



summary of expenditures to date, and planned activities for the



coming reporting period.



 



When data collection is involved, the contract should also specify



provision to the project officer, for review and approval, of all



data collection plans; instruments and



 



                                38



 



 



 



 



 



associated materials, including sampling procedures; letters to



respondents; interviewer or respondent instructions for completion



of the questionnaire; training materials for data collection;



coding manuals; and data processing specifications.  If the project



officer does not have the requisite expertise to review and approve



these materials; one or more technical advisors on the project team



should have such expertise and should be asked to conduct the



review, giving advice on appropriate action.



 



Should the contractor fail to submit required reports or other



deliverables in a timely or technically acceptable manner as



specified in the contract, the project officer should notify the



contracting officer so that determination of appropriate action can



be made.  Unless the delay is minor or affects only a small,



inconsequential portion of the contract, it will probably be



necessary for the contracting Officer to extend officially the



contract schedule or terminate the contract completely.



 



 



Review of Financial Reports



 



The contracting officer will, particularly in the case of cost-



reimbursement contracts, often ask the project officer to review



the periodic vouchers submitted by the contractor for payment of



costs incurred.  For a large or complex project, it may be helpful



to have the information for the financial reports provided in



specified categories.  This will facilitate the assessment of



whether the project is likely to be completed within the budgeted



resources and will be useful if the survey is ever repeated.  In



addition to a report of the current status of expenditures, it may



also be desirable to require projections of expenditures for the



coming project period and perhaps for the remainder of the project.



 



Review of expenditure information in conjunction with technical



progress reports will enable the project officer to assess



likelihood of cost overruns and to take early steps to avoid or



minimize them.  The project officer must determine that



expenditures are in keeping with technical progress before



recommending to the contracting officer that payment be made.  If



any problems are identified, the project officer should inform the



contracting officer and effort can be initiated to take remedial



action.



 



 



Modifications to the Contract



 



When the contractor or project officer identifies the apparent need



for a change in the contract specifications, they may discuss it



informally between themselves, but no deviations from the



specifications of the written contract may occur until the contract



has been formally modified in writing by the contracting officer. 



If there is a need to change aspects of the work requirements,



deliverables, or schedule for completion of activities, formal,



written modification to the contract will be necessary.



 



 



Approval of Subcontracts



 



It is not unusual for a contractor to engage a subcontractor to



complete one or more tasks under a contract.  It is important to



the Government that the capabilities of the subcontractor be such



that a high quality job will result. if it is known that a



subcontract will be necessary, it and the proposed subcontractor



should be identified before the contract is signed.  However, the



need for a subcontract is sometimes not identified until after the



contract has been awarded.  Therefore, the negotiated contract



should contain a clause requiring approval by the contracting



officer of any subcontract.



 



                                39



 



 



 



 



 



Before entering into a subcontract, the prime contractor should



provide information about the work to be done under the



subcontract, the anticipated costs and dates for completion of



work, along with identification of the proposed subcontractor and



information about why and how the subcontractor was selected.  Upon



receipt of notification of plans to subcontract, the contracting



officer will provide the information to the project officer for



concurrence that the proposed subcontractor has the requisite



capabilities to perform in a technically acceptable manner.  The



project officer should monitor the prime contractor's management of



work under the subcontract.



 



However, if the work of the subcontractor or the management of the



subcontractor by the prime contractor is not satisfactory, the



project officer should not take direct action.  Instead, the



contracting officer should be notified and asked to take



appropriate action through the prime contractor.  Only under rare



circumstances would the project officer engage directly in contact



with the subcontractor.



 



 



Administration of Government Property



 



The contractor will usually have all equipment and other resources



necessary for conduct of the work under a contract.  However, a



contract may specify that the Government will provide certain



materials necessary to complete the work.  If data collection is



involved, this might be a sampling frame based on administrative or



other records that are in the possession of the sponsoring agency. 



It might be access to Government computer facilities, or it might



be acquisition of specialized equipment specifically for the work



of the contract.  The project officer is responsible for seeing



that the appropriate arrangements are made for contractor access



to, acquisition of, or use of such equipment or materials.  The



contracting officer is also likely to seek the advice of the



project officer on disposition of any equipment purchased under the



contract.



 



 



Closing Out the Contract



 



Upon completion of all work under the contract and provision to the



Government of all required reports, materials and information, and



review and approval of all specified deliverables by the project



officer, the final payment will be made to the contractor. 



Although contract closeout is the responsibility of the contracting



officer, the project officer may be called upon to assist.



 



 



QUALITY ASSURANCE RESPONSIBILITIES OF THE PROJECT OFFICER



 



It is implicit throughout the following discussion that the project



officer will probably not have the full range of technical skills



needed to monitor all aspects of the work.  As appropriate, other



members of the project team should be called upon to participate in



post-award monitoring activities.  These persons should have the



necessary technical knowledge, and familiarity with and



understanding of the objectives and specifications of the contract



to provide advice to the project officer.



 



Every survey should include a program of built-in quality assurance



activities.  It is the responsibility of the project officer to



ensure that appropriate steps are taken to quantify and document



the quality of survey data.  Provision should have been made in the



contract specifications for inclusion of various quality assurance



components, and the contractor's technical proposal should have



addressed them.



 



This section of the chapter presents an overview of quality



assurances that should, as appropriate, be built into any survey. 



The discussion is intended to apprise a project



 



                                40



 



 



 



 



 



officer of the variety of ways in which data quality can be



maximized.  Not all of these measures will be appropriate or



necessary for every survey.  In general, large studies require all



or most of these quality assurance procedures; smaller, less



complex studies may require only some of them.



 



The project officer should be thoroughly familiar with the required



and proposed steps to be taken by the contractor to measure data



quality and achieve a satisfactory level of quality at all steps



from instrument development through data collection, data



reduction, and final data tape preparation.  If no quality



assurance program is specified in the contract the project officer



should discuss plans for one with the contractor's project director



and agreement should be reached and documented on a step-by-step



basis so that there is no room for misunderstanding; a formal



contract modification may be made to make the quality assurance



program a contract requirement.



 



A key element in assurance of data quality is the active



involvement of the project officer at all steps in the data



collection and data reduction processes.  Active participation by



the project officer and other members of the project team will



often involve costs for travel and other activities.  There should



be provision for these costs in the agency's operating budget over



the period of the project.



 



The role of the project officer is sometimes specified in the



contract itself; the project officer should make clear to the



contractor's project director as early as possible the intended



nature and extent of this involvement.  Following is an enumeration



and discussion of quality assurance activities in which the project



officer should take an active role.



 



 



Sample Selection, Questionnaire Design and Data Collection



 



Data collection activities conducted under contract usually require



approval by the Office of Management and Budget.  It is the



responsibility of the project officer to learn about these



requirements, determine how they pertain to a particular project,



oversee the development of necessary descriptive materials, and



shepherd the submission through the review and approval process. 



Appendix 11 gives an overview of



 



these requirements but does not replace early consultation with the



agency reports clearance officer.



 



In addition to reviewing and approving sample selection



specifications, interviewer and coder manuals, and training



materials, the project officer should attend and, as appropriate,



participate in interviewer and coder training. project officer



involvement in activities relating to sampling, questionnaire



design, interviewer training, and field procedures during the



pretest and pilot surveys is particularly important.  Sample design



and selection are critical survey underpinnings.  Questions for a



survey must be carefully worded to ensure that what is conveyed to



a respondent is what is intended, and responses must be recorded



(either verbatim or in predetermined categories) so that the



planned analysis can be adequately accomplished.  Similarly, the



interviewer training and actual data collection activities



constitute survey activities in which errors or oversights that



occur can negate the success of the entire project.



 



Sample selection.  The process of sample selection is critical and



should be carefully monitor d by both the project officer and the



contractor.  Depending on the type of survey and whether the



sponsor provides a sampling frame, it may be necessary for the



contractor to obtain, create, or refine the frame from which sample



units for the survey are to be selected.  It is important to



assess, prior to selection of the sample,



 



                                41



 



 



 



 



 



the completeness of coverage of the sampling frame, the currency of



the frame and its contents, and the associated problems arising



from limitations of coverage and currency.  These steps are



essential even if the survey is a census of the universe of



eligible individuals or other units.



 



Once the frame has been determined to be satisfactory, it is



necessary to ensure that the specified procedures are followed in



selection of the sample.  This should involve a monitoring and



review of the selection that takes place at each stage of sampling. 



If the sample procedures are being carried out incorrectly at one



or more stages of selection, biases in survey estimates will



probably be introduced.  If independent population and



subpopulation counts (or good estimates of them) are available from



a census or large sample, these should be used to help evaluate the



quality of the sample.  For example, suppose that a national



probability sample of persons age 18 years and older. is selected. 



Estimates of population counts by age, race, and sex cross-



classifications from the sample should be compared to corresponding



census counts; any substantial variations could indicate that the



sample is not being selected correctly.



 



Questionnaire development.  On occasion an agency provides the



contractor with a completely developed questionnaire or set of data



collection instruments.  But even when this occurs, there is



usually the need for a pretest of the questionnaire because seldom



has it been used exactly as intended in the present effort.



 



More typically, the contractor is responsible for development of



the data collection instruments.  The advice and experience of



subject matter experts are usually the basis for instrument



development, and to the extent possible, previously developed



questions whose efficacy has been demonstrated should be used. 



Questionnaires should receive several levels of pretesting, the



least formal of which is testing on office staff or friends.  This



type of testing offers an opportunity to identify difficulty in



understanding the intent of the question and question flow. 



Subsequently, there should be more formal tests of the



questionnaires.  The project officer should not only review the



questionnaires and any accompanying instructions for their



administration or completion, but should also test the instructions



and questionnaires by administering or completing them.  Only in



this way can first-hand experience be gained with the instruments



and the problems associated with their administration.



 



Interviewer and coder training.  The project officer is responsible



to monitor training for data collectors in order to be certain that



instruction, given is accurate and thorough.  If possible, the



project officer should attend the training sessions; at a minimum,



the project officer can explain the objectives and importance of



the study to provide project staff with an understanding of why the



data are needed and how they will be used.  This is also an



opportunity for direct acquaintance with the data collectors and



the level of project understanding with which they embark upon



their work.



 



Interviewer observation.  Throughout the data collection process,



as circumstances permit or dictate, the project officer should



participate in an interviewer field observation program.  Survey



firms usually have a program for observation of each interviewer



early in their field assignment or for monitoring telephone



interviewers throughout the survey.  This observation or monitoring



is typically done by a field supervisor and is intended to



determine how the interviewer presents himself or herself and the



survey to respondents, whether procedures are followed as specified



(including asking the questions exactly as they are worded on the



questionnaire), and what problems are encountered in the field



(including efficient use of time in the field and correct field



editing of completed data collection instruments).



 



                                42



 



 



 



 



 



The project officer should try to observe or monitor several



interviewers, possibly identifying at the training session those to



accompany to the field or otherwise monitor.  Interviewers are



seldom perfect in their work, but by observing several the project



officer will be able to assess strengths and weaknesses in the data



collection process.  Observers, however, must remember that their



role is a passive one and that no active role should be taken



during the interview to either converse with or correct the



interviewer during the interview itself.



 



Reinterview program.  Many surveys have as a component of their



quality assurance program a reinterview program in which a



subsample of the respondents and nonrespondents are recontacted to



validate selected information reported by the interviewers. 



Reinterview programs are usually developed to check on the work of



individual interviewers in order to identify interviewers who are



reporting imaginary interviews.  For some surveys, reinterviews are



used to try to measure response bias and variation.  Some survey



organizations routinely send respondents a postcard to verify that



the interviewer contacted them and the length of the interview. 



This augments but does not replace a personal recontact with a



sample of the original respondents for each interviewer in which a



brief reinterview is conducted to verify that the original



interview was, in fact, conducted.  Effort is made to conduct the



reinterview with the initial respondent; typically included are a



few basic questions about household or respondent characteristics



needed to verify the completeness and accuracy of information about



their eligibility for the survey.



 



Response rate.  It is important that the project officer monitor



closely the response rate being achieved by the contractor. 



Several aspects of the response rate should be calculated by the



contractor on an ongoing basis and should be communicated to the



project officer on a regular basis, perhaps weekly.  Overall



response rates, interviewer-specific response rates, and response



rates for key subgroups or characteristics of the study population



should be monitored.  Problems of failure to locate sample units



and to complete interviews or obtain completed questionnaires for



sample units must be identified as quickly as possible in the data



collection effort so that corrective action can be taken.  It is



important to achieve acceptable response rates for specific



subgroups as well as for the entire population.  Poor response



rates can lead to substantial nonresponse biases that cannot be



estimated.  Furthermore, if there are large differences in the



nonresponse rates among population subgroups, there may be



differential nonresponse biases in subgroup estimates. 



Consequently, there may be some misleading subgroup comparisons due



to differential biases.



 



If there is more than one stage to data collection, such as a



screener questionnaire to identify those eligible for a more



detailed questionnaire, the response rate for the screener is



particularly critical and must be given special attention.  The



nonresponse rate at the screening stage is part of the survey



nonresponse rate.  Furthermore, a high screener nonresponse rate



could create a substantial bias in the estimate of the size of the



target subpopulation.



 



 



Data Reduction



 



Tolerable error levels for coding and data entry must be



established and may have been specified in the contract, but there



must be a monitoring system to assure that they are met.  These



levels are often specified in the contract.



 



As for interviewers, training is very important for coders.  The



project officer should consider observing at least some of the



coder training to ensure that it is comprehensive and clear.



 



                                43



 



 



 



 



 



Coding.  Questionnaires are usually processed in batches through



the coding and data entry phases of the survey with a single



individual having responsibility for them at each stage.  For



example, early in a project all of a coder's work is usually



checked or verified on a 100-percent basis.  Then, if the person is



operating within the limit of tolerance for error for the project,



this recheck is cut back to a fraction of the completed work. 



Should it occur that the acceptable error level is exceeded for a



batch, the entire batch may then be checked, the coder may be given



special retraining if appropriate, and the coder's work is again



verified on a 100-percent basis until tolerable accuracy is



achieved or the person is removed from the project.  A similar



approach is used for data entry, but data entry is often verified



on a 100-percent basis for the duration of a project.



 



Surveys often include open-ended items that allow the respondent to



describe or explain something in his or her own words.  Such



information is then coded into predetermined categories by



specially trained persons.  It is common practice to have two



persons code the responses to some or all of the open-ended



questions on a survey.  This is known as independent coding when



the second coder or verifier works without knowing how the first



person coded the information.  When the two coders disagree, there



may be adjudication.  In highly complex coding, there may be three-



way independent verification in which two verifiers independently



code a sample of a production coder's work with a majority rule



applying when two or more coders agree on a code.  Independent



coding may be continued throughout a project.



 



For large scale surveys it is not unusual for the contractor to



turn over to the sponsoring agency, for totally independent coding,



a small percentage of completed records.  These should be selected



in such a way that they can be considered representative of the



completed work done by the contractor's staff.  Independent coding



is done by agency staff or another contractor using the detailed



coding manual (including documentation on coding decisions made on



difficult and unusual cases).  Any discrepancies are then reviewed



to determine their nature, reasons, and impact on the data base and



analyses.



 



Data cleaning.  Vigilance to ensure high quality survey results



must Continue through the data processing phase of the project.  To



ensure that computer programs are actually doing what they are



specified to do, it is prudent to put through the system test data



for which the desired values are known through hand or other



independent calculation.  This set of test data should represent



the full range of operations expected of the complete program. 



Then, if the expected results are not obtained, it is clear that



there is at least one problem with the purportedly debugged



program.  Even though the contractor may think the programs are



functioning, it is wise for the project officer to make certain



that it is indeed the case that only valid codes are read and that



data manipulation is occurring properly.



 



It is important to be certain that only the admissible codes are



contained for each item on a survey record.  However, it is



possible for errors to be present on the final data tape.  In



addition to checking that only reasonable, allowable codes are



present for each item, there are usually between item checks made



to test for consistency within a subject's data record.  For



example, a medical procedure code for hysterectomy can be a valid



code in studies of hospitalizations, but such a code should not be



valid if the person involved is a male.  Such a combination of



codes on a record requires checking back to determine the source



and nature of the error.



 



                                44



 



 



 



 



 



Data Documentation



 



The project officer should be certain that complete documentation



of all sample selection, data collection, reduction, and processing



activities is created, reviewed, and kept up to date so that it can



be provided with the data tape or tabulations that are received



from the contractor.  This documentation should include information



about sample design specifications, sample frame construction,



sample selection, calculation of respondent weights, field



procedures for initial respondent contact, follow-up of



nonrespondents, and out-of-scope cases.  Any and all decisions



about coding, resolution of apparent inconsistencies in data items



for a given respondent, imputation for missing data, and other



nonresponse adjustments, as well as full documentation of all codes



used in the processing of data and preparation of the final data



tape, must be documented and made available to anyone who will be



involved in analysis, reanalysis or review of data from the survey. 



The data tape received from the contractor should include "flags"



to indicate any imputed values so that analysts will know which



data were actually reported by respondents and which were



statistically derived.  Without this information, along with



documentation of how sampling weights and estimation procedures are



derived and used, the value and utility of the survey data will be



greatly diminished.



 



 



Data Analysis



 



If the contractor is responsible for data analysis, care should be



taken by the project officer that there is a good understanding of



the specific tabulations to be provided by the contractor.  The RFP



for a large, complex survey often includes either actual or



representative table shells for tabulations to be provided by the



contractor.  Manipulation of large and complex data bases is a



difficult and costly task and provision should be made well in



advance for it.



 



In addition to the tabulations to be produced, attention should be



given to the analytic procedures and tests to be used in data



analysis.  If certain assumptions (for example, a random sample)



must or can be made about the selection of the sample in order to



use existing software packages, it is important to be sure that the



requisite assumptions are reasonably well met by the sample.  It is



recognized, however, that for some purposes a systematic random



sample may be equivalent to a simple random sample.



 



Appropriate statistical tests to be applied in determining whether



differences between various estimates are significant should be



agreed upon in advance if the contractor's staff is to perform



analysis of the data.  If the contractor performs data analysis,



information on testing procedures used should be included in the



final documentation for the project provided by the contractor.



 



                                45



 



 



 



 



 



 



 



 



Appendix 1.    INFORMAL SURVEY OF AGENCIES AND CONTRACTORS



               CONCERNING CONTRACTING FOR SURVEYS



 



 



INTRODUCTION



 



When the Subcommittee on Contracting for Statistical Surveys set



out to do its work, the members believed it would be important to



solicit facts and judgments from persons who were directly involved



both from the point of view of Federal agencies and from the point



of view of contracting firms.  The Subcommittee realized that,



within its frame of reference, it could not field a formal



statistical survey of Federal agencies and contractors concerning



the broad spectrum of problems encountered in contracting for



statistical surveys.  On the other hand, the Subcommittee could and



did adopt some procedures which would assure then that its report



reflected more than just the views of its members and their



immediate contacts.  It should be emphasized that the results are



not the careful findings one would encounter in a scientific sample



survey.  They are the considered judgments of a selection of



individuals who are personally involved in the contracting process



both from the Federal agency and private contractor point of view.



 



The Subcommittee devised two interview schedules - one for



representatives of contracting firms and one for representatives of



Federal agencies which use contracts for statistical surveys. 



Respondents were usually groups of individuals who were interviewed



simultaneously; in a typical agency interview, for example, a



subject matter specialist, contracts expert and administrative



official would be present.



 



The agencies were selected first and then a list of their



contractors was obtained.  The sample of contractors was selected



from that list.  This procedure offered some assurance that the two



groups of interviewees were both concerned with similar Federal



contracts, although from different points of view.



 



All the contractors were engaged in the competitive procurement



process with the exception of one contractor selected from those



largely involved in noncompetitive procurements.  For the



competitive contractors, the first step in the sample selection



process was to review the contract awards of the selected agencies



in the statistical contracting area.  All contracts that seemed to



be within the area of review were identified.  Next, the sample was



drawn and the contracts were reviewed more carefully to ensure that



they were consistent with Subcommittee judgments regarding



statistical contracts.



 



For the non-competitive contractor group, a list of such



contractors was obtained.  The list was reviewed by the



Subcommittee.  Most were determined not to engage in statistical



contracting.  One contractor was selected from the remaining few



based on Subcommittee members' knowledge of the contractors and the



type of work they typically performed.



 



Interviews were conducted by Subcommittee members. The interview



schedule was comprised mostly of open-ended questions which would



elicit principally qualitative information.  Results were compiled



at the Office of Management and Budget (OMB),



 



                                47



 



 



 



 



 



and Subcommittee members used the results as background information



in preparing this report.



 



 



FEDERAL AGENCY RESPONSES



 



Eleven interviews were ultimately conducted with Federal agencies,



two in the nature of pretests.  The results described below do not



always pertain to the same number of agencies and contractors



because the pretest schedule differed in some respects from the



schedule finally adopted and because the interviews were



unstructured.  Seven of these agencies permit contractors to use



subcontractors, although some restrictions may be placed on the



extent of subcontractor involvement.  The number of statistical



survey contracts awarded by these agencies during fiscal year 1977



and fiscal year 1978 ranged from a low of 2 in one agency to a high



of "25 or more in another."  Four agencies used cost-type



contracts; one used fixed-price contracts; and three used both. 



Contract amounts varied between $40,000 for 1 year to $20 million



over a 3-year period.



 



 



Developing RFPs and Soliciting Bids



 



Typically, the first step for agencies is to develop a Request for



Proposal (RFP) which provides prospective contractors with



information concerning the specifications of the proposed survey. 



Usually, no formal guidelines or checklists are used in preparing



the RFP.  On the other hand, the respondents in the agencies agreed



as to features which should appear in all RFPS.  More than half of



the interviewees stated that the following items should appear in



RFPS: proposal evaluation weights, population to be studied, level



of effort, key content variables, data collection method, sampling



frame, required response rate, analysis of data, and sampling



design.  Items such as data processing procedures, sampling



reliability, level of response, questionnaire development, trade-



offs between cost and quality, and quality control procedures were



believed to be less important.  Sometimes agencies consult OMB



concerning the contents of a survey RFP before issuance; sometimes



they ask other agencies for review and comment.



 



The availability of the RFP is usually announced in the Commerce



Business Daily as a formal notification to potential contractors. 



Advertisements may also be put in trade papers or announcements



placed in the Federal Register.  Mailings are also made to lists



derived from past agency experience or from the expertise of



persons in the industry.  A few agencies utilize sources-sought



announcements as a way to stimulate offeror interest.



 



Of ten agencies make use of Bidders' Conferences to explain the RFP



and answer technical questions.  Sometimes these conferences result



in changes, or clarifications in the RFP.



 



The number of proposals received after solicitation of bids varies



widely among the agencies and by the specific subjects of the RFPS. 



Some receive only three or four proposals for each RFP, others may



receive as many as 30 proposals.  The size and cost of the



contemplated survey are often major factors in determining the



number of proposals offered: generally, the bigger the size and



cost, the greater the number of proposals.



 



                                48



 



 



 



 



 



Evaluating Proposals



 



Agencies are governed by Federal procurement laws and regulations



in evaluating proposals.  The following results are not a



description of the legal procedures but rather an encapsulation of



the processes agencies generally go through.



 



Agencies evaluate first the technical aspects of proposals and then



the business aspects.  Selection committees are formed to evaluate



and choose the best proposal.  The technical review committee



usually consists of three to eight people and includes persons with



backgrounds in statistical surveys and contracting.  Actual costs



are usually not known by the committee.  Some agencies allow



personnel from outside the agency to participate in these



committees, particularly when special expertise is needed.  The



business evaluation is usually conducted by contract and



procurement personnel.  This process consists of a review for



appropriateness and consistency of financial resources and project



organization.



 



Following the initial evaluation of proposals, offerors may be



asked to revise their proposals based on results of the initial



evaluation.  A final evaluation is then performed.  In the final



evaluation, rating sheets are completed for each technically



acceptable proposal.  The contracting officer determines the



official competitive range.  The finally selected proposal must



always be acceptable on technical criteria but need not be the



least expensive.



 



From issuance of the RFP to awarding of the contract requires at



least 60 to 90 days, often longer.  The final selection process can



take at least 30 days.  All agencies suggested that the length of



time for the entire process is too long.  They suggested



eliminating certain procedures to expedite selection of the



contractor.



 



Offerors not selected may ask for a debriefing, at which time



unsuccessful offerors can learn about the deficiencies in their



proposals.  Few of the agency respondents had experienced any



protests.



 



 



Monitoring and Other Considerations



 



A project officer, who is usually a person with academic training



and job experience in surveys, manages the survey for the life of



the contract.  Some agencies report problems with turnover in



project officers.



 



Agencies generally cited some problems in the execution of survey



contracts.  Some contractors have not performed well; others have



needed more time to complete the job; some difficulties have been



encountered with response levels and deciding when to stop seeking



survey responses.  A few agencies require progress reports and some



send staff for site visits.  For the most part, however, feedback



concerning contractor performance is obtained informally.



 



Most of the agencies stated that they perceived no ethical problems



in contracting.  Of those reporting ethical problems, one had a



staff member who had previously worked for a company specializing



in the survey subject.  Another expressed the fear that technical



personnel may sometimes tend to reveal too much information arising



from the confidential evaluation process.



 



                                49



 



 



 



 



 



CONTRACTOR RESPONSES



 



Eleven interviews were conducted with contractors, two of which



were pretests.  The interviewees were eight commercial statistical



firms and three nonprofit institutions.  Most were engaged in a



broad range of statistical work.



 



Of the eleven contractors that were interviewed, five specified the



professional staff available at their organizations.  Most of their



staffs included statisticians, computer systems analysts,



programmers, sociologists, and support staff.  Psychologists,



economists, and market researchers were also available full time on



some staffs.  One of the contractors stated that key personnel are



maintained in-house but extra staff are hired for contracts.



 



 



Sources of Information on RFPs



 



Almost all contractors consulted the Commerce Business for RFPS but



all have other sources of information as well such as personal and



professional contacts.  Most considered sources other than CBD to



be their most productive, particularly the personal and



professional ones.  Several contractors expressed reservations



about relying solely on announcements in CBD, saying that they



rarely win contracts they first heard about in CBD.



 



 



Level of Specificity in RFPs



 



Contractors had much to say about the level of specificity in RFPS. 



They noted that "the level varies widely among agencies.  Some



believed that contracting agencies make unrealistic specifications;



others said ambiguity or overspecification in RFPs reflect a lack



of sophistication on the part of the RFP writer.  A few stated they



were very comfortable with either very general or very specific



RFPS but the ones in between proved most troublesome.



 



Different levels of specificity were believed appropriate for



different agencies.  Agencies with a good technical staff could



write a detailed RFP (for example, including specifications of



sampling reliability, specific questions to be asked, and required



response rate); those without such staff could write only very



general RFPS.  Levels of specificity were also related to the kinds



of studies.  Some thought evaluation studies needed very general



requirements; studies where a specific set of data is sought should



have stricter requirements.



 



A frequent complaint was that the RFPs did not specify the level of



effort desired by the agency.  As to specific items to be included,



there was a general desire to have the RFP specify the population,



the key variables, and to a lesser extent the response rate.  Other



factors should be left to the offeror, particularly sampling



design, data collection method, data processing procedures, quality



control, and analysis of data.



 



 



Improvements to RFPs



 



Many contractors suggested that RFPs should be written more



clearly.  Several wanted a clear statement of goals and the amount



of money available.  Others suggested that the level of effort



desired should be specified.  A few suggested that outside experts



be hired to write RFPS.  A few mentioned that time constraints were



a problem; the time limitation on preparing proposals often hurts



small contractors especially.



 



Other suggestions made for improving RFPs included making RFPs more



consistent among agencies and giving the agency and the contractor



more flexibility to make



 



                                50



 



 



 



 



 



changes during the course of the contract.  A few contractors



stated they were reluctant to bid on fixed-price contracts.



 



A number of suggestions and comments were made regarding whether



the sponsoring agency or the offerors should provide various



technical aspects of a survey.  The majority of the contractors



preferred that the RFP state the target population and the key



variables to be studied.  Several contractors also suggested that



the level of effort (anticipated budget) be included in the RFP. 



Most contractors preferred that the offerors specify the particular



questions to be asked, sampling frame, sampling reliability, data



collection method, data processing and quality control procedures,



and analysis of the data.



 



 



Problems with Inadequate RFPs



 



Most contractors stated they have had problems with inadequate



RFPS.  One major problem was unrealistic expectations, learning



that the agency actually wants much more than was implied in the



RFP.  Problems were thought to arise from hasty writing or



authorship of the RFP by someone who did not understand the



technical issues.  Instances were pointed out where precision



requirements were incorrectly specified.  This can lead to



substantial problems in comparing proposals.  Other problems have



been lack of clarity and lack of understanding of confidentiality



issues.  In one instance, a contractor was forced to hire a lawyer



to educate agencies on issues of confidentiality.



 



 



Bases for Decisions to Submit Proposals



 



The two most important criteria determining whether contractors



submitted proposals were expertise in the subject field and



perceived chance of winning the contract.  Other factors were



expected profitability, size of contract, availability of staff,



and the amount of competition, including whether the firm had a



head start or inside track.  The nonprofit institutions gave



"interest in the topic" as a major reason to bid on a contract. 



More than half of the contractors did not respond to a question



concerning the types of professional staff they had available;



others who did respond said they had a full staff or could easily



get the services of any staff they needed.



 



 



Reactions to Bidders, Conferences



 



Almost all of the contractors stated that they always attended



Bidders' Conferences.  They did so mainly to assess the competition



and to make sure the competition did not get any advantages over



them.  A few felt the conferences were occasionally useful in



clearing up ambiguities, but one noted that offerors are often



reluctant to ask the real questions lest they help competitors. 



Others stated that Bidders' Conferences were most useful for large



technical studies.



 



 



Preparing Proposals



 



No strategy was consistently applied by contractors in preparing



proposals.  Replies on this topic tended to be vague, emphasizing



case-by-case basis or outdoing the competition.  Almost half said



they tried to submit a technically superior proposal.



 



Cost estimates for preparing proposals ranged from a low of $1,000



to high estimates of $10,000 and even one of $30,000.  Several



mentioned that their dollar figures did not take into account time



spent by staff.  A few gave costs in terms of percentage of the



final contract, such as 5 percent of expected dollar amount.



 



Costs of preparing proposals could be reduced by improving RFPs by



making them more explicit about capabilities and qualifications. 



Some contractors suggested



 



                                51



 



 



 



 



 



limiting the length of proposals or eliminating unnecessary and



redundant demands from RFPS.  One suggestion was for a two-stage



bidding process.  The first stage would be open to all offers, who



could submit a general description of their proposal and a list of



qualifications.  The agency would then select the most promising of



these for a second stage and invite detailed proposals.  This would



save money and reduce the number of detailed proposals to be



reviewed.  Other suggestions were for more qualified project



officers and for greater use of sole-source contracts when there is



clearly a front running firm.



 



 



Agency Selection Procedures



 



Most contractors believed that agency selection procedures did not



generally result in selection of the best proposal.  The primary



reason given was that, the winning proposal is not selected



primarily on technical grounds; too much emphasis is put on cost. 



Contractors attributed this to inexperienced or incompetent agency



personnel.  Contractors felt that lowest offerors were often not



well qualified but that taking the lowest offeror was a safe course



for agencies to follow.



 



The selection of the winning proposal was not believed by the



contractors to revolve around technical issues because agencies



judge proposals on the reputation of the firm and its staff.  They



felt insufficient time is allowed for adequate technical judgment



and that agency staff have poor memories, repeatedly selecting



contractors who have done poorly in the past.



 



There was some difference of opinion among the contractors between



those who believe a proposal should be judged on technical grounds



solely and those who believe more emphasis should be placed on the



firm's ability to perform.  A few of the contractors felt the best



proposal was not generally selected because of the alleged practice



of "wiring" contracts, where the agency knows beforehand which firm



will get the contract.



 



 



Improvement in Selection Procedures



 



Many suggestions focused on getting more technically qualified



people to evaluate the proposals within the agencies.  Some stated



that the selection procedures will be improved when the agencies



have competent people to write and evaluate the proposals. 



Suggestions were made for more peer review or for specification of



the methodology of selection.



 



 



Debriefings



 



Most contractors indicated that they had asked for a debriefing on



a contract not awarded to them.  Most said the debriefing was



useful, especially if they could compare their proposal to the



winning bid.  One contractor said the debriefings were good for



indicating what areas of the proposal were weak, what the agency



was really looking for and how the procurement process works. 



Another contractor noted that agencies are occasionally defensive



at debriefings, especially when minority firms are involved.



 



 



Ethical Considerations



 



Several contractors stated that, while the selection system is



generally clean and fair, some contract rigging or "wiring" exists. 



A number mentioned problems of confidentiality.  A few were



concerned with whether their proposals were kept confidential



during the evaluation process; they believed there were occasions



when they had not been awarded the contract, but that their ideas



had found their way into the winning proposal.



 



                                52



 



 



 



 



 



Another confidentiality problem concerns survey respondents' rights



to privacy.  Contractors expressed concern about being able to



maintain confidentiality of data collected, some contractors noted



that agencies lack experience in knowing how to use such data and



require education by contractors.



 



Another issue was sole-source contracts.  Some contractors believed



government regulations should be changed so that an agency can



award follow-on sole-source contracts to firms with which it is



well satisfied.



 



Other ethical issues raised were the assumption that small firms



cannot finance a large job and a bias against g(a) firms.  An



unethical practice on the part of the agencies was asking the



offeror to propose alternatives to the RFP specifications then



rewriting the RFP to use these alternatives.



 



 



Other Considerations



 



Among the other comments made by contractors were complaints



concerning the lengthy OMB clearance process as well as other long



delays.  Some of them felt the length of time it takes to get



questionnaires cleared was an ethical problem.  Others complained



of too much irrelevant information being required on proposals.



some contractors felt the government should consolidate the



procurement regulations of the various agencies.  Others complained



that year-end contracts were not really competitive because time



constraints precluded careful evaluation, that references on



proposals were not checked, and that many contractors defined



person-years differently which led to problems in comparing costs.



 



                                53



 



 



 



 



 



 



 



 



     Appendix II.   OFFICE OF MANAGEMENT AND BUDGET REQUIREMENTs:



                    Information Collection Budget and Clearance



 



INTRODUCTION



 



Almost all Federal agencies are required under the Paperwork



Reduction Act of 1980 (P.L. 96-511)1 to obtain approval from the



Office of Management and Budget (OMB) each time they propose to



collect or sponsor, under a contract or other agreement, the



collection of identical information from more than nine



respondents.  In addition to main data collection, activities



related to development or testing of data collection plans are also



subject to OMB review and approval if identical information is



sought from more than nine respondents in either a formal or



informal manner.  The document to be submitted foe OMB approval and



the time of submission will depend in part upon the agency



practice.  The development and shepherding through the applicable



channels of the request for approval is one of the most important



and demanding responsibilities of the project officer on a



contract.  Due to variations in agency requirements and policies



pertaining to OMB clearance, it is especially important to



determine them at the earliest possible stage in development of a



project.



 



Documents that require approval under the Paperwork Reduction Act



include the following:



 



-    report forms, application forms, questionnaires, or interview



     guides for in person or telephone surveys.



 



-    orders, regulations or other directives that include



     requirements for respondents to provide information or



     maintain records to be used or made available for use in the



     collection of information.



 



-    requests for answers to identical questions which are



     addressed to respondents by telephone, telegraph, form



     letters, information circulars and other devices.



 



-    any supplementary documents involved in these reporting or



     recordkeeping requirements, such as instructions or covering



     letters or introductory statements.



 



A number of standards have been developed and promulgated relating



to surveys, including standards for statistical surveys,



publication of statistics, and definitions for certain items; these



must be adhered to as they apply to a particular survey.  Early in



the development of a survey, the project officer should discuss the



nature of and plans for the survey with the agency reports



clearance officer.  This will provide an opportunity to become



familiar with the most recent directives pertaining to OMB



clearance and to acquire an understanding of the process as it will



apply to the particular survey.  A list of agency clearance



officers is available from the Office of



___________________________



 



1This legislation supersedes and extends the Federal Reports Act



of 1942.



 



                                55



 



 



 



 



 



Information and Regulatory Affairs at OMB if there is any



difficulty in identifying an appropriate agency official with whom



to consult.



 



 



INFORMATION COLLECTION BUDGET



 



A project officer should also be certain that provision for a



particular project is made in the agency's Information Collection



Budget (ICB).  The ICB is an annual, comprehensive submission from



each agency to the Director of the Office of Management and Budget



describing each existing and new information collection project



proposed to be implemented or continued in the succeeding fiscal



year.2 In the case of a project that has not been included in the



agency ICB, it will be necessary to determine the implications for



project timing and scope.



 



 



CLEARANCE



 



Types of Clearance



 



When the Request for Proposal (RFP) prescribes the data collection



plan (that is, the information to be collected and the method of



collection), OMB permits submission of the plan for OMB approval



prior to issuance of the RFP or signing the contract.  Agency



contracting policies vary with respect to insistence on receipt of



OMB clearance prior to actual signing of a contract.  But for large



data collection projects, it is wise to obtain OMB approval before



the contract is signed.  Contracts that include data collection



activities should contain a special provision in the RFP and in the



final negotiated contract that no data may be collected without



receipt of OMB approval.



 



If data collection is required by or implicit in the statement of



work for the contract, the project might be submitted for



preliminary (or concept) approval from OMB prior to signing of the



contract.  This can obviate some problems later, after work on the



project has begun, and it should facilitate approval of subsequent



OMB clearances for the Project.



 



A survey that has a pretest designed to help make decisions about



final questionnaire content or survey methodology will usually



require separate approvals for the pretest and the final data



collection plan.  Thus, up to three separate clearances may be



required for a survey and it is not unusual for a survey to require



two separate clearances.



 



 



Timetable for Clearance



 



Clearance is a lengthy process and ample time should be allowed for



it.  Several agency and contractor representatives interviewed by



the Subcommittee expressed



___________________________



 



2Upon completion of its review of each department's ICB request,



OMB transmits a "Passback" which gives a total information



collection allowance of respondent burden-hours for the department



for the fiscal year.  This passback may disallow specific projects. 



It may also contain specific suggestions for elimination or



reduction of respondent burden.  In providing this passback detail,



OMB is not committing itself to providing Paperwork Reduction Act



approval for any of the items contained in the ICB request;



Paperwork Reduction Act decisions will be based upon the merits of



each individual request for approval.



 



                                56



 



 



 



 



 



their frustrations about the time required by the OMB approval



process and mentioned additional expenses resulting from delays in



the OMB clearance process.  The agency clearance officer can



provide information about how long it can be expected to take for a



particular survey.



 



In general, at least 2 weeks should be allowed for each office that



must handle and review the clearance request within the agency. 



Under the Paperwork Reduction Act, the Director of OMB is given 60



days after receipt of a proposed information request to notify the



agency involved of the decision to approve or disapprove the



request and is to make such decisions publicly available.  However,



the Director may determine that a request submitted for review



cannot be reviewed within 60 days, in which case after notice to



the agency involved the Director may extend the review period for



an additional 30 days.  If the Director does not notify the agency



of an extension, denial or approval within 60 days (or if the



review period has been extended for an additional 30 days and



notice of action is not given within the time of the extension) a



control number is to be assigned without further delay, approval



may be inferred, and the agency may collect the information for a



period of not more than 1 year.



 



A prudent project officer will allow for the possibility of lengthy



delays in obtaining OMB approval for a project.  Delays do occur



from time to time and when a contractor is involved there can be



significant problems that arise at such a time if the project



officer has not had a contingency plan.  Such a contingency plan



should be built into the contract.  One way to handle this problem



is to write a phased contract, in which certain activities are



clearly identified to be done prior to receipt of OMB clearance and



others are to be done only after receipt of OMB clearance.  Another



method, which can be used with fixed-price contracts, is to include



provisions that: (1) no additional sums are authorized because of



delays in clearance, and (2) the duration of the contract is day-



for-day extended for delays encountered in clearance.  The program



contracts office can provide information on how each agency handles



this type of provision.



 



 



Clearance Submissions



 



There are formal specifications for the information to be provided



in support of a request for approval of data collection plans.  The



format for the supporting statement, as specified by OMB, must be



followed in writing a clearance request.  Clarity and brevity



should be relied upon in preparation of the clearance supporting



statement, with lengthy detailed description and discussion



reserved for inclusion in attachments.



 



It is the responsibility of the project officer to prepare the OMB



clearance request, submit it for approval, and answer any questions



that are raised by reviewers.  Once a contract has been let, the



project officer may wish to have the contractor assist in



preparation of the description of survey procedures to be used; in



any case, the project officer bears full responsibility for



reviewing any material prepared by the contractor for completeness,



correctness, and adequacy.  Often it is helpful to have the reports



clearance officer review the clearance package in draft prior to



formal submission.



 



At an early-stage in the project, there should be contact with



relevant statistical or other programmatic offices with subject



matter related programs and experience.  This,will usually ensure



proper coordination with other projects in the subject area and



will usually facilitate clearance.  Such contacts. should be



documented in the clearance submission.



 



                                57



 



 



 



 



 



In developing a timetable for the project, a realistic period



should be allowed for each stage in preparation for and conduct of



data collection activities (including follow-up of nonrespondents



and recontact of respondents for any reason).  Some slippage of



schedules inevitably occurs and should be allowed for in



determination of a requested expiration date.  If it becomes clear



that data collection cannot be completed by the expiration date for



OMB approval, a request for an extension should be submitted to OMB



at least 60 days before the existing approval is due to expire.



 



Once the project officer has submitted the clearance request



through channels to the appropriate clearance officer, he or she



should make every effort to remain informed about its progress



through the review process.  This will enable the project officer



to give the contractor current and correct information about the



status of the review and will hopefully keep the contractor from



trying independently to make contacts in the clearance chain -



something the contractor should never do.  Project officers, too,



should make all inquiries about projects in the review process



through proper channels.



 



 



AFTER RECEIPT OF APPROVAL



 



Once a survey has received OMB approval, the approval number and



the expiration date for the approval must appear in specified



locations on the approved documents.  Finally, a number of copies



of the final printed form(s), transmittal letter, instructions, and



any other document sent to each respondent must be provided for the



official clearance files.



 



Before a material revision or change is made in an approved



information collection, a formal request for revision must be



submitted to OMB for approval Changes in an information collection



requiring approval include the following: any increase in the kind



or amount of information sought; any increase in coverage; any



increase in the timing or frequency of reporting; any change in the



sample design or collection method; or a change in the purpose for



which the information is collected or required to be maintained.



 



                                58



 



 



 



 



 



                     Reports Available in the



              Statistical Policy Working Paper Series



 



1.   Report on Statistics for Allocation of Funds GPO Stock Number



     003-005-00178-6, price $2.40



 



2.   Report on Statistical Disclosure and Disclosure-Avoidance



     Techniques GPO Stock Number 003-005-00177-8, price $2.50



 



3.   An Error Profile: Employment as Measured by the Current



     Population Survey GPO Stock Number 003-005-00182-4, price



     $2.75



 



4.   Glossary of Nonsampling Error Terms: An Illustration of a



     Semantic Problem in Statistics (A limited number of copies are



     available from OMB.)



 



5.   Report on Exact and Statistical Matching Techniques GPO Stock



     Number 003-005-00186-7, price $3.50



 



6.   Report on Statistical Uses of Administrative Records GPO Stock



     Number 003-005-00185-9, price $5.00



 



7.   An Interagency Review of Time-Series Revision Policies (A



     limited number of copies are available from OMB.)



 



8.   Statistical Interagency Agreements (A limited number of copies



     are available from OMB.)



 



9.   Contracting for Surveys (A limited number of copies are



     available from OMB.)



 



 



Copies of, these working papers, as indicated, may be ordered from



the Superintendent of Documents, U.S. Government Printing Office,



Washington, D.C. 20402



 



 



 

(sw9.html)

ARROW UP

 


Page Last Modified: April 20, 2007 FCSM Home
Methodology Reports