Overview 
Survey Design Survey Quality Measures Trend Data Availability of Data
1. Overview (2010 survey cycle) 
a. Purpose
The 2010 Higher Education R&D (HERD) Survey, successor to the Survey of Research and Development Expenditures at Universities and Colleges, collected information from 742 research-performing academic institutions. The survey is the primary source of information on research and development expenditures at higher education institutions in the United States and outlying areas. The HERD survey's predecessor was conducted annually from FY 1972 through FY 2009 and collected information on R&D expenditures by academic field as well as by source of funds. The HERD survey continues to capture core information on R&D expenditures by source of funding and field to retain the historical time series. However, the survey has been revised and expanded to include more detail. The survey was revised to include R&D within non-science and engineering (S&E) fields in the totals reported throughout the survey. Previously, non-S&E R&D was reported but not included in the overall totals. The scope of R&D was also broadened to include expenditures on clinical trials and research training grants.
b. Respondents
The HERD survey is a voluntary establishment survey completed by designated contacts at U.S. universities and colleges. The majority of respondents for higher education institutions work in one of the following institutional offices: accounting, grants and contracts, controller, financial, institutional research, or sponsored programs.
c. Key variables
The HERD survey requests data from institutions on their R&D in the following categories. R&D is defined as creative work conducted systematically to increase the stock of knowledge (research) and to use this knowledge to devise new applications (development).
- R&D expenditures by field and source of funds (federal government, state and local government, business, nonprofit, institutional, and other)
- R&D expenditures funded from foreign sources
- R&D expenditures within medical schools
- Clinical trial R&D expenditures (phases I-III)
- R&D expenditures by character of work (basic research, applied research, and development)
- Total and federally funded R&D expenditures passed through to subrecipients or received as a subrecipient
- Federally funded R&D expenditures by field and federal agency
- Federally funded R&D expenditures funded by the American Recovery and Reinvestment Act
- R&D expenditures by cost categories (for example, salaries, software, equipment, indirect costs)
- Total and federally funded R&D equipment expenditures by field
- Headcount of R&D principal investigators and all other R&D personnel
- Headcount of postdocs working on R&D
- Academic institution
- Institutional characteristics (highest degree granted, historically black college or university [HBCU], public or private control)
- Geographic location (within the United States)
2. Survey Design
a. Target population and sample frame
The target population for the HERD survey includes nonprofit postsecondary institutions that have bachelor's or higher degree programs in any field and annually perform at least $150,000 in separately budgeted R&D in any field. Prior to FY 2010, the population was limited to institutions meeting the $150,000 expenditure threshold for R&D within science and engineering (S&E) fields and having bachelor's or higher degree programs in S&E fields only. Prior to FY 2004, in addition to 4-year and above institutions meeting the $150,000 threshold, all institutions that had doctoral programs in S&E and all HBCUs were included in the survey population, regardless of the level of R&D.
The frame for the FY 2010 HERD survey was the list of institutions considered in scope for the FY 2009 survey. This list was constructed through the screening of institutions identified as receiving federal funds by the NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, from institutions identified in the NSF-National Institutes of Health Survey of Graduate Students and Postdoctorates in Science and Engineering as having master's or doctoral programs in S&E fields, and institutions listed in the Higher Education Directory.
A new requirement for FY 2010 was that within a university system, each campus headed by a campus-level president or chancellor complete a separate survey rather than combining its response with other campuses in the university system. As a result of this step, the overall number of academic institutions in the population increased from 711 in FY 2009 to 742 in FY 2010.
b. Sample design
The FY 1997 survey was the last to be conducted as a sample survey. Since 1998 the survey has been a census of all known eligible universities and colleges.
c. Data collection techniques
The FY 2010 survey was conducted by ICF International under contract to the National Center for Science and Engineering Statistics (NCSES). Respondents could choose to print an Adobe Portable Document Format (PDF) questionnaire from the Web and submit a paper survey, upload an Excel version of the questionnaire, or use the Web-based data collection system to respond to the survey.
Respondents were sent survey information and a user ID and password via e-mail. The Web-based system contained instructions and built-in help designed to replicate the printed forms. Each institution's Web survey included comparable data from the 2 preceding years for arithmetical and trend check purposes. Large increases or decreases in expenditures compared with the preceding years were flagged, and respondents were asked to explain and reconcile these differences.
Data collection began in December 2010, with the official survey due date announced as 25 February 2011. Because the desired response rate had not been reached by the survey deadline, a series of reminder e-mail messages and phone calls were used to encourage response from all institutions between March and May of 2011, with priority placed on eliciting responses from the previous year's top 100 institutions in total R&D expenditures. The survey was closed for new submissions by the end of May 2011, and data verification and follow-up took place between June and October of 2011.
d. Estimation techniques
This survey is a census. Imputation was performed for nonresponding institutions in order to make population estimates (see section 3c for more information). Estimates are computed by summing reported and imputed values for institutions with the appropriate characteristics.
3. Survey Quality Measures
a. Sampling variability
Because the FY 2010 survey was a universe survey distributed to all institutions in the universe, there was no sampling error.
b. Coverage
Coverage of the large research institutions is excellent, because they are easily identified using the NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions. However, institutions with smaller amounts of R&D expenditures have been more difficult to find as they often do not receive federal funding for science and engineering R&D.
As part of the expanded FY 2010 HERD survey collection, a special effort was undertaken to screen 1,715 four-year and higher postsecondary institutions not currently included in the survey to identify additional institutions meeting the $150,000 threshold. These institutions were given detailed instructions regarding what to include as R&D expenditures and asked to respond to question 1 of the HERD survey, total R&D expenditures by source of funding. This effort achieved a response rate of 64.4% and identified 187 additional institutions over the $150,000. These institutions will be asked to respond to the full survey for FY 2011. The total R&D expenditures reported by these institutions was $249 million, or 0.42% of the $61.2 billion reported by the FY 2010 HERD survey respondents. More details on this effort will be included in the methodology report and technical notes for the FY 2010 survey.
c. Nonresponse
(1) Unit nonresponse—15 universities and colleges did not respond in FY 2010 out of a total of 742 eligible institutions, for a nonresponse rate of 2%.
Information for nonresponding institutions was imputed. Imputation was performed using prior years' figures derived from the data of respondent institutions with similar characteristics, including highest degree granted and type of institutional control (public or private). Imputed values account for less than 1% of the total R&D expenditures included in the survey. When an institution failed to report in a given year but reported in the next year, the data for the missing year were re-imputed to reflect the new information if it appeared likely that the new information would have a significant impact on the original estimate.
(2) Item nonresponse—The item nonresponse rates varied from 0% for Question 1, total R&D expenditures by source of funds, to 20.6% for Question 6, R&D expenditures by character of work (basic research, applied research, and development). The other questions with nonresponse rates over 10% were Questions 16, headcount of R&D personnel (13.3%) and 17, headcount of R&D postdocs (16.9%).
Missing information for nonresponse was imputed for all questions except ARRA-funded expenditures (Question 11), headcount of R&D personnel (Question 16) and headcount of postdocs (Question 17). Imputations were based on average increases, decreases, or proportions derived from data of respondent institutions with similar characteristics, including highest degree granted and type of institutional control (public or private), and total R&D expenditures.
d. Measurement
The most likely source of measurement error is institutional records containing categories different from those requested on the survey. For example, institutions are asked to report all R&D expenditures by field. The NSF-designed fields do not always translate to an institution's departmental structure, and adjustments must be made by the institution in order to complete the survey. A crosswalk between the NSF fields of R&D and the National Center for Education Statistics Classification of Instructional Programs (CIP) codes is provided with the survey in order to mitigate this source of measurement error.
Another example is NSF's category of institutionally financed organized research. The survey requests that the institutions report discretionary internal funds used for separately budgeted organized research. NSF discovered through debriefings conducted at the conclusion of the FY 2010 survey that there are varying definitions of what is considered institutionally financed organized research (included on HERD) versus departmental research (excluded on HERD). Some institutions classify all expenditures from separate accounts designated for research as organized research, whereas others only include internal R&D projects that are competitively awarded and have detailed budgets. A workshop is planned for the summer of 2012 in order to discuss these differences in definitional interpretations and determine if further revisions are needed for the survey instructions.
The reporting of unreimbursed indirect costs is another known source of measurement error. The survey requests that the total amount of indirect costs associated with a research grant or contract be calculated and reported, including costs that were not reimbursed by the external funding source. Calculating the unreimbursed indirect cost is done by multiplying the institution's negotiated indirect cost rate by the corresponding base and then subtracting the actual indirect cost recovery, preferably on a project by project basis. Previous site visits have determined that some institutions are not able to report the unreimbursed indirect cost amount.
4. Trend Data
Annual data are available for FY 1972–2010, with the exception of FY 1978. That year's survey covered a different population and used different questions than preceding or subsequent surveys and is therefore not comparable to other years. When the review for consistency between each year's data and submissions in prior years reveals discrepancies, it is sometimes necessary to modify prior years' data. This is especially likely to affect trends for certain institutions that fail to report every year, because current year data are used to impute prior year data. Individuals wishing to analyze trends other than those published in NCSES's most recent publication are encouraged to contact the project officer listed in section 5c for more information about comparability of data over time.
5. Availability of Data
a. Publications
The data from this survey are published annually, both in a summary InfoBrief and in full as detailed statistical tables in the series Higher Education Research and Development, or, for FY 2009 and earlier years, its predecessor series, Academic Research and Development Expenditures. Data are available by fiscal year on the NCSES website. Data for major data elements are available starting in 1972. Information from the survey is also included in Science and Engineering Indicators, National Patterns of Research and Development Resources, Science and Engineering State Profiles, and Academic Institutional Profiles.
b. Electronic access
All data from this survey are available on the NCSES website and in WebCASPAR. Selected aggregate data are provided in public use data files upon request.
c. Contact for more information
Additional information about this survey may be obtained by contacting:
Ronda Britt
Research and Development Statistics
Program
National Center for Science and Engineering Statistics
National Science Foundation
4201 Wilson Boulevard, Suite 965
Arlington, VA 22230
Phone: (703) 292-7765
E-mail: rbritt@nsf.gov
Last updated: March 29, 2012
