You Are Here: AHRQ Home > Research Funding Opportunities > Grants Process > Peer Review Process

Peer Review Process

Overview

In accordance to the Public Health Service Act and the federal regulations governing "Scientific Peer Review of Research Grant Applications and Research and Development Contract Proposals" (42 CFR Part 52h), applications submitted to the Agency for Healthcare Research and Quality (AHRQ) will be evaluated by AHRQ peer review process to ensure a fair, competent and objective assessment of their scientific and technical merit. The peer review of grant applications involves an assessment conducted by panels of experts established according to scientific disciplines or medical specialty areas.

A Scientific Review Officer (SRO) is the Designated Federal Official of the peer review group meeting. Her/his role is to make sure that each application receives a review that is thorough, competent and fair. AHRQ has established a set of principles or rules that must be followed to assure this goal, and it is the SRO's responsibility to ensure that these principles are observed. The SRO oversees, coordinates and administers the peer review process. The SRO performs initial administrative and technical review of applications to ensure completeness and accuracy. The SRO also nominates reviewers based on broad input from scientific community, AHRQ program staff, and experts who have served as peer reviewers.

Reviewers are selected to participate in the peer review groups based on a number of considerations, including:

Authorities in their scientific fields.
Dedication to high quality, fair, and even-handed reviews.
Demonstration of their abilities to work collegially in a group setting.
Experience in grant/contract review.

The SRO works with the peer review group chairperson to ensure that the peer review meeting is fair and yields high quality reviews. To accomplish this, the peer review meeting should include:

Balanced and adequate scientific discussion that supports the review group's recommendations.
Consistent, criteria-based scoring.
Adequate attention to protection of human subjects and inclusion of priority populations.

Following the peer review meeting, the SRO prepares summary statements for all applications. The summary statement contains:

Resume and summary of discussion.
Minimally-edited reviewers' critiques.
Final overall Impact score and percentile ranking.
Budget recommendations.
Administrative notes.

The summary statement is an official feedback to the applicant conveying the issues, critiques, and/or comments that were raised during the review of his/her application.

Streamlining/Triaging

Streamlining is a practice through which less competitive grant applications, as judged by the study section members, are not discussed at the review meeting. In order for an application to be streamlined, there must be unanimous agreement amongst study section members. These applications do not receive an overall impact score. For the "NOT DISCUSSED" applications, the summary statement integrates all of the written reviews from the assigned reviewers.

Understanding Your Priority Score

The AHRQ grant application scoring system uses a 9-point scale for the overall impact score and individual scores for five core criteria:

A score of 1 indicates an exceptionally strong application with essentially no weaknesses.
A score of 9 indicates an application with serious and substantive weaknesses with very few strengths.
A score of 5 is considered an average score.
Ratings are in whole numbers only (no decimal ratings) (go to Table 1).

Table 1.

Impact	Score	Descriptor	Additional Guidance on Strengths/Weaknesses
High	1	Exceptional	Exceptionally strong with no weaknesses
	2	Outstanding	Outstanding
	3	Excellent	Very strong with only some minor weaknesses
Medium	4	Very Good	Strong but with numerous minor weaknesses
	5	Good	Strong but with at least one moderate weakness
	6	Satisfactory	Some strengths but also some moderate weaknesses
Low	7	Fair	Some strengths but with at least one major weakness
	8	Marginal	A few strengths and a few major weaknesses
	9	Poor	Very few strengths and numerous major weaknesses
Non-numeric score options: DF = Deferred, AB = Abstention, CF = Conflict, NP = Not Present, ND = Not Discussed.
Minor Weakness: An easily addressable weakness that does not substantially lessen impact. Moderate Weakness: A weakness that lessens impact. Major Weakness: A weakness that severely limits impact.

Overall Impact

Reviewers will provide an overall impact score and critique to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following five core review criteria, and additional review criteria (as applicable for the project proposed).
Note that an application does not need to be strong in all categories to be judged likely to have major scientific impact and thus deserve a high impact/priority score.
For example, an investigator may propose to carry out important work that by its nature is not innovative but is essential to move a field forward.

Scored Review Criteria

Reviewers will consider each of the core review criteria in the determination of scientific and technical merit, and give a separate score for each core review criterion (using 1 to 9 scoring range).
Funding Opportunity Announcements list core review criteria.
For example, the five scored criteria for research grant applications are Significance, Investigator(s), Innovation, Approach, and Environment.

Final Overall Impact Score

The final overall impact score reflects the average impact scores provided by each individual members and multiplying the average by 10; the final overall impact/priority score is reported on the summary statement. Thus, the final overall impact/priority scores range from 10 (high impact) through 90 (low impact). Abstaining reviewers and those in conflict with the application and/or not present during the discussion do not provide a score.

Understanding Your Percentile

A percentile, based on the final overall impact score, is an application's rank relative to other applications that are reviewed by the same scientific review group at 3 consecutive meetings. AHRQ calculates an application's percentile based on the combined overall impact scores of applications being reviewed by the study section at which the application was reviewed and by the same study section at previous two meetings. Overall impact scores from the combined 3 consecutive review meetings are known as the base. Percentile ratings are calculated only for discussed applications. A percentile, which may range from 0.1 to 100.0; with lower numbers represent better scores, is a ranking used to normalize scores across peer review groups.

Review Criteria

Review Criteria and Considerations. Applications are evaluated for scientific and technical merit according to the Core Review Criteria, Additional Review Criteria, and Additional Review Considerations listed in Funding Opportunity Announcement (FOA).

Scored Review Criteria. Reviewers will consider each of the five review criteria below in the determination of scientific and technical merit, and give a separate score for each. An application does not need to be strong in all categories to be judged likely to have major scientific impact. For example, a project that by its nature is not innovative may be essential to advance a field.

There are 5 components that make up the core review criteria:

Significance and originality.
Investigator(s).
Innovation.
Approach.
Environment.

A description of each component is provided below.

Significance: Does this study address an important problem? If the aims of the application are achieved, how will scientific knowledge or clinical practice be advanced? What will be the effect of these studies on the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?
Investigators: Is the PD/PI and other key personnel appropriately trained and well suited to carry out this work? Is the work proposed appropriate to the experience level(s) of the principal investigator(s) and other researchers? Does the PD/PI and investigative team bring complementary and integrated expertise to the project (if applicable)?
Innovation: Is the project original and innovative? For example, does the project challenge existing paradigms or clinical practice; address an innovative hypothesis or critical barrier to progress in the field? Does the project develop or employ novel concepts, approaches or methodologies, tools, or technologies for this area?
Approach: Are the conceptual or clinical framework, design, methods, and analyses adequately developed, well-integrated, well-reasoned, and appropriate to the aims of the project? Does the applicant acknowledge potential problem areas and consider alternative tactics?
Environment: Do(es) the scientific environment(s) in which the work will be done contribute to the probability of success? Do the proposed studies benefit from unique features of the scientific environment(s), or subject populations, or employ useful collaborative arrangements? Is there evidence of institutional support?

Additional Review Criteria

As applicable for the project proposed, reviewers will consider the following additional items in the determination of scientific and technical merit, but will not give separate scores for these items.

Protection of Human Subjects from Research Risk: The involvement of human subjects and protections from research risk relating to their participation in the proposed research will be assessed See the "Human Subjects Sections" of the PHS398 Research Plan component of the SF424 (R&R).
Inclusion: Adequacy of plans to address the needs of both genders, racial and ethnic minorities (and subgroups). Adequacy of attention to AHRQ priority populations (see above discussion on Priority Populations in section IV.6 "Other Submission Requirements," and inclusion criteria included in section VIII of Required Federal Citations.)
Budget: Is the proposed budget reasonable and is the requested period of support appropriate in relation to the proposed research?
Privacy and Security Protections for Patients: The resources and processes to be used to address privacy and security issues in the development and implementation of the intervention will be assessed.
Degree of Responsiveness: How well does the application address the purpose and objectives of the FOA? Does the proposed project fit within an AHRQ Research Portfolio area? How responsive is the application to the special eligibility criteria, including the project requirements, noted in the FOA?
Resubmission Applications (formerly "revised/amended" applications): Are the responses to comments from the previous scientific review group adequate? Are the improvements in the resubmission application appropriate?
Importance and Impact (if applicable): What are the importance of the problem addressed and the potential impact of the solution of this problem on health or on health care delivery?
Data Sharing Plan (if applicable): Is the proposed data sharing plan or the rationale for not sharing research data reasonable?

Current as of August 2011

Internet Citation:

Peer Review Process. August 2011. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/fund/peerrev/peerproc.htm