Background

The mission of the NIH is to support science in pursuit of knowledge about the biology and behavior of living systems and to apply that knowledge to extend healthy life and reduce the burdens of illness and disability.  As part of this mission, applications submitted to the NIH for grants or cooperative agreements to support biomedical and behavioral research are evaluated for scientific and technical merit through the NIH peer review system.  Program Project Grant (PPG) applications may be submitted for review three times per year and are associated with the October, January, and May National Heart, Lung, and Blood Advisory Council meetings.

In June 2007, the NIH initiated a formal, agency-wide effort to review in its entirety the NIH peer review system (http://enhancing-peer-review.nih.gov/). After careful deliberation and consideration of the recommendations resulting from this year-long effort, a number of key actions are being implemented in the NIH peer review system, and a change in the NIH scoring system is one of them. 
 
Until recently, each scored application was assigned a single, overall priority score that reflects the consideration of all review criteria (eg, significance, experimental approach, facilities). Individual reviewers  assigned scores on a 1.0 to 5.0 scale in 0.1 increments (e.g., 2.2), resulting in 41 possible rating discriminations for reviewers to make.  The reviewers’ individual scores were then averaged and multiplied by 100 to yield a single overall priority score for each scored application (e.g., 253).

Although this rating system has served the NIH and the research community well, several concerns led the NIH to consider a revised rating system for grant applications.  Making 41 discriminations is difficult for reviewers to do reliably, and scores increasingly have become compressed toward the positive end of the scale.  In addition, by averaging reviewer scores and multiplying by 100, the resulting priority score appears to have more precision than it actually has.  To address these concerns, the NIH considered scoring systems with fewer rating options to increase potential reliability and consistency and with sufficient range and appropriate anchors to encourage reviewers to use the full scale.  To increase transparency, the NIH also considered methods to communicate ratings from assigned reviewers even when the application is streamlined and not discussed at the review meeting.

 

New NIH Scoring System

The new scoring system is effective for all applications for research grants and cooperative agreements that are submitted for funding consideration for fiscal year 2010 (FY2010) and thereafter.  The first standing PPG application receipt date for FY2010 was January 25, 2009.  An important aspect of the implementation of the new scoring system is to use it in a consistent manner for applications considered in a given fiscal year.  The new scoring system utilizes a 9-point rating scale (1 = exceptional; 9 = poor).  Peer reviewers, whether they are part of the tailored review committee for the PPG or a member of the PPG Parent Committee, may only vote in single integers. The entire range of voting options for peer review is depicted at the end of this document.  In general, the new 9-point scoring system is meant to direct reviewers to concentrate on the overall impact of the proposed research rather than on the fine technical and methodological details of the study.  For additional information on the 9-point rating scale, please see NOT-OD-09-024 in the NIH Guide for Grants and Contracts.

 

Scoring the PPG

NHLBI review staff convened tailored review committees between March 31 and May 29 of 2009 to review the Program Project Grant applications received by the NHLBI for the January 25, 2009 receipt date.  All of these applications were then reviewed at the June 19, 2009 meeting of the NHLBI Program Project Review Committee.  The applications will undergo the final level of review at the National Heart, Lung, and Blood Advisory Council October 2009 meeting.  At each of these tailored review committee meetings, reviewer education regarding the new scoring system consisted of printed materials that described the new NIH scoring system and a pre-review teleconference or webinar (seminar over the internet) convened by the Scientific Review Officer (SRO) to explain the new scoring system prior to review of any of the applications.

In addition, the NHLBI Program Project Parent Committee had a comprehensive and rigorous orientation session prior to the beginning of the aforementioned Spring review cycle. The committee members also had a substantive discussion at their June meeting to ensure the consistent application of the new scoring system when voting the Overall Impact/Priority Score of the PPG applications.  The Committee concluded that the overall impact/priority score of the PPG should be clearly aligned with the descriptors of strengths and weaknesses that are provided NIH-wide for the new scoring system, as depicted in the table below.  Hence, the Parent Committee focused primarily on the text of the critiques prepared by the tailored review committees, not on the subproject priority scores. 

It is not unusual for the overall score accorded a PPG by the Parent Committee to reflect a different priority score than the arithmetic mean of the subproject scores voted by the tailored review committee.  The tailored review committee possesses specific and indepth knowledge of each of the subproject topics and votes a score as if the subproject were an individual, stand-alone research project grant application.  The Parent Committee maintains a broader view of each PPG as a whole and votes an overall impact/priority score that reflects the ability of the entire grant to operate as a complete program and to exert a sustained and powerful influence on the research field.