NIH Extramural Nexus

As you may know, the recent “Enhancing Peer Review” self-study process at NIH led to, among other things, the introduction of bulleted critiques and overall impact scores. We received a lot of feedback on the changes, and a recurring theme was the desire for more information about the impact scores.

In light of this, we have begun asking reviewers to write a paragraph summarizing the factors that informed their overall impact score to supplement the bulleted critiques. This paragraph is included in the summary statements, and I hope you find it useful.

We also initiated a continuous review of the peer review process. You can see a report on the first round of surveys on the OER website.

I am constantly in awe at the speed at which things move in this electronic age (example: Watson on “Jeopardy!” last night), so I was delighted to see that yesterdays NIH Notice about a modification to the biosketch has already hit the blogosphere. In case you missed the announcement, applicants now can use their biosketch to explain how personal circumstances may have delayed their transition to an independent career or reduced their scientific productivity. This opportunity will provide peer reviewers, and others, additional information on which to base their assessment of the qualifications and productivity of the applicant. The change was implemented based entirely on comments we heard directly from our community—concerns that the existing biosketch could work against applicants when there were unexplained gaps. We listened to your concerns and developed a policy that we think is responsive and will benefit all applicants.

I have read or heard much about the dilemma of NIH applicants as they struggle to understand their chances of receiving NIH funding. As budgets flatten and tighten, this discussion has heated up. To declare that NIH success rates have hovered around 20% for the past five years does little to calm the storm of concern when we hear about shrinking percentiles and paylines. So how is it possible to have a success rate of 20% but a payline at the 7^th percentile? Let’s take a few moments to sort out what these things mean and think about how these numbers are derived and how they can differ.  

Impact Score

It all starts with the impact. This score is assigned by reviewers to indicate the scientific and technical merit of an application. Impact scores range between 1 and 9. A score of “1” indicates an exceptionally strong application and “9” indicates an application with substantial weakness. (I always wondered why at NIH low = good and high = bad but that predates me!) In assigning an impact score, reviewers consider each of five scored criteria: significance, investigator, innovation, approach, and environment, along with other factors like protection of human subjects and vertebrate animal care and welfare. Read more about scoring.

Percentile Rank

The percentile rank is based on a ranking of the impact scores assigned by a peer review committee. The percentile rank is normally calculated by ordering the impact score of a particular application against the impact scores of all applications reviewed in the current and the preceding two review rounds. An application that was ranked in the 5^th percentile is considered more meritorious than 95% of the applications reviewed by that committee. This kind of ranking permits comparison across committees that may have different scoring behaviors. It is important to note than not all research project grant applications (RPGs) are percentiled. For example, applications submitted in response to a request for applications (RFA) are usually not percentiled. In the absence of a percentile rank, the impact score is used as a direct indicator of the review committee’s assessment. Read more about percentiles.

Payline

Many NIH institutes calculate a percentile rank up to which nearly all R01 applications can be funded. For grant applications that do not receive percentile ranks, the payline may be expressed as an impact score. Institutes that choose to publish paylines in advance (see an example) calculate the payline based on expectations about the availability of funds, application loads, and the average cost of RPGs during the current fiscal year. Other institutes prefer to describe the process for selecting applications for funding (see an example) and then report on the number of applications funded within different percentile ranges at the end of the fiscal year (see an example). Because the NIH is currently operating on a continuing resolution and funding levels for the remainder of this fiscal year are uncertain, most of the NIH institutes have offered less detail this year than in the past.

But remember, even when an IC establishes a payline, applications outside of the payline can be paid under justified circumstances if these applications are a high priority for the particular institute or center. When these select-pay/out-of-order/priority pay/high priority relevance selections are made, it may result that other applications within in the payline are not paid because funds are no longer available to support them. 

Success Rates

The success rate calculation is always carried out after the close of the fiscal year, and it is based on the number of applications funded divided by the number of applications reviewed and expressed as a percent. To better reflect the funding of unique research applications, the number of applications is adjusted by removing revisions and correcting for projects where the resubmission (A1) is submitted in the same year as the original application (A0). Read more about success rates.

The Answer

Now we are equipped to answer our earlier question. How is it possible to have a success rate of 20% but a payline at the 7^th percentile? There are several real-life reasons why paylines (the ones that use percentiles) can be either higher or lower than success rates.

• Applications that are not percentiled are still factored into the success rate calculation. Thus, funding a number of awards that are not assigned percentiles will increase the success rate without changing the payline.
• The success rate for a particular fiscal year is a reflection of the funded applications and can include applications reviewed in the previous fiscal year; whereas, the payline encompasses only applications reviewed in that fiscal year. So awarding applications that were reviewed in the previous year will also increase the success rate.
• The average quality of the applications assigned to an institute will also affect its payline. If an institute happens to receive a set of applications with very good (low) percentile scores, its success rate will be higher than its payline, all else being equal. For example, in fiscal year 2010, the NIGMS R01 success rate was about 27% but the midpoint of the funding curve occurred close to the 21^st percentile.

Check out more reports on RPG success rates broken down by year (2001 to 2010) and IC.

Whew, you made it through. The difference between paylines, percentiles and success rates remains a confusing topic because of the compounding factors that rule out a simple linear relationship. You need to consider all the factors when assessing the potential for an individual application to be funded. Your best advisor on this issue, because of the differences in the ICs and programs, is your NIH program official. Give him or her call.

NIH has fellowship, training and career programs for the clinician scientist, and we have just added a new, exciting and unique opportunity for early stage clinician researchers. Scientists who successfully compete to become a Lasker Clinical Research Scholar will spend five to seven years in our intramural program. They will then be offered the opportunity to either stay at NIH as a senior clinical researcher or apply for up to five years of support to go back out into the extramural community. The program is open to clinician scientists from both the extramural and the intramural communities. Applicants must have an MD, MD/PhD, DO, DDS, DMD, RN/PhD, or equivalent clinical degree and must be no more than six years from completion of their core residency training program. The deadline is March 4, 2011 for a letter of intent and April 4, 2011 for the full application. Candidates must also obtain a DUNS number and register with the NIH eRA Commons prior to submitting an application.

More information, including the RFA, is at http://www.nih.gov/science/laskerscholar. Direct any questions about this program to Charles R. Dearolf, Ph.D., Assistant Director for Intramural Research, at LaskerScholar@nih.gov. 

I am delighted that NIH launched this program in conjunction with the Lasker Foundation. It represents a truly historic partnership between the NIH extramural and intramural programs. We’re expecting great things from our future Lasker Clinical Research Scholars!

I’d like to pass along some good news on the where-am-I-going-to-get-my-grant-funded-in-today’s-economy front. The National Health Council  just announced the availability of a database that will help link unfunded NIH applications with potential non-governmental funding sources.

We all know that you have more good ideas than NIH is able to fund. That makes this a very good time for the National Health Council to roll out HealthResearchFunding.org.  This database will serve as a kind of clearinghouse for unfunded research proposals.

The way this works is that PIs can choose to put basic information about their scored but unfunded NIH grant applications directly into the National Health Council’s HealthResearchFunding.org database. Potential funders will then be able to search the database to find projects they might be interested in funding.

This approach seems very efficient. Not only does it take advantage of the NIH peer review process, but if it works the way it is intended, it should reduce the amount of time and effort investigators and institutions put into looking for alternative funding sources. Time that can be better spent on research.

I am pleased that my staff had the opportunity to provide input into this effort, and I am looking forward to watching this idea roll out and evolve.