This week I want to address the field of health research in the lesbian, gay, bisexual, transgender and intersex communities (LGBTI communities). Earlier this month, NIH Director Francis Collins announced additional support of LGBTI health research by renewing and permanently establishing a trans-NIH LGBTI research coordinating committee under the leadership of two NIH institutes: the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) and the National Institute on Minority Health and Health Disparities (NIMHD). This committee was initially established to evaluate the NIH research portfolio and how it addressed health issues unique to LGBT individuals, as a follow-up to the NIH-commissioned Institute of Medicine (IOM) report published in 2011. See Francis Collins’s statement for more information about the IOM and NIH reports and plans to bolster LGBTI health research.

In February 2012, we released three specific funding opportunity announcements (FOAs) for R01, R03, and R21 awards in this research area. One misconception that I want to clarify is whether researchers can submit applications related to LGBTI health to other funding announcements. The answer is of course. While we issued these FOAs to draw specific attention to research needs in LGBTI health, NIH welcomes applications related to LGBTI health for any applicable FOA, and we encourage investigators to discuss their ideas with NIH staff in advance.

As scientists, we’re well acquainted with the importance of protecting our research from factors that could undermine objectivity. And as the nation’s largest single funder of biomedical research, it is vitally important that the public trusts the research NIH supports.

 To this end, NIH recently published a centralized document describing our policies pertinent to scientific research integrity. While NIH has extensive policies and procedures to help ensure the highest degree of scientific integrity, this new document –  NIH Policies and Procedures for Promoting Scientific Integrity –  for the first time consolidates this information from a diverse array of documents into a unified report that’s easily accessible to anyone interested in learning more about NIH research, even those new to working with or learning about NIH.

 An NIH working group, which included members of my staff in the Office of Extramural Research, developed this resource, and I’m pleased to see the outcome of their hard work. This document not only fulfills a White House directive for a public document on scientific integrity, but it is also supports transparency and public accountability that is at the core of NIH’s mission.

Time flies when you’re having fun! I started Rock Talk on January 19, 2011 (see the first two posts here and here) and when Nature asked me to write about my adventures in blogging to commemorate Rock Talk’s two year anniversary I gladly agreed… check out my article here.

As a followup to my recent blog post on fiscal year 2012 success rates, I’d like to post an update of an earlier blog post where I explained how paylines, percentiles and success rates relate to one another. It’s a long one, but should be helpful in understanding what we mean when we look at success rates.

 “Paylines, Percentiles and Success Rates” with updates added:

I have read or heard much about the dilemma of NIH applicants as they struggle to understand their chances of receiving NIH funding. As budgets flatten and tighten, this discussion has heated up. To declare that NIH success rates have hovered around 20% for the past five years does little to calm the storm of concern when we hear about shrinking percentiles and paylines. So how is it possible to have a success rate of 20% but a payline at the 7^th percentile? Let’s take a few moments to sort out what these things mean and think about how these numbers are derived and how they can differ.

IMPACT SCORE

It all starts with the impact. This score is assigned by reviewers to indicate the scientific and technical merit of an application. Impact scores range between 1 and 9. A score of “1” indicates an exceptionally strong application and “9” indicates an application with substantial weakness. (I always wondered why at NIH low = good and high = bad but that predates me!) In assigning an impact score, reviewers consider each of five scored criteria: significance, investigator, innovation, approach, and environment, along with other factors like protection of human subjects and vertebrate animal care and welfare. Read more about scoring.

PERCENTILE RANK

The percentile rank is based on a ranking of the impact scores assigned by a peer review committee. The percentile rank is normally calculated by ordering the impact score of a particular application against the impact scores of all applications reviewed in the current and the preceding two review rounds. An application that was ranked in the 5^th percentile is considered more meritorious than 95% of the applications reviewed by that committee. This kind of ranking permits comparison across committees that may have different scoring behaviors. It is important to note than not all research project grant applications (RPGs) are percentiled. For example, applications submitted in response to a request for applications (RFA) are usually not percentiled. In the absence of a percentile rank, the impact score is used as a direct indicator of the review committee’s assessment. Read more about percentiles.

PAYLINE

Many NIH institutes calculate a percentile rank up to which nearly all R01 applications can be funded. For grant applications that do not receive percentile ranks, the payline may be expressed as an impact score. Institutes that choose to publish paylines in advance (see an example) calculate the payline based on expectations about the availability of funds, application loads, and the average cost of RPGs during the current fiscal year. Other institutes prefer to describe the process for selecting applications for funding (see an example) and then report on the number of applications funded within different percentile ranges at the end of the fiscal year (see an example) Because the NIH is currently operating on a continuing resolution and funding levels for the remainder of this fiscal year are uncertain, most of the NIH institutes have offered less detail this year than in the past.

But remember, even when an IC establishes a payline, applications outside of the payline can be paid under justified circumstances if these applications are a high priority for the particular institute or center. When these select-pay/out-of-order/priority pay/high priority relevance selections are made, it may result that other applications within in the payline are not paid because funds are no longer available to support them.

SUCCESS RATES

The success rate calculation is always carried out after the close of the fiscal year, and it is based on the number of applications funded divided by the number of applications reviewed and expressed as a percent. To better reflect the funding of unique research applications, the number of applications is adjusted by removing revisions and correcting for projects where the resubmission (A1) is submitted in the same year as the original application (A0). Read more about success rates.

THE ANSWER

Now we are equipped to answer our earlier question. How is it possible to have a success rate of 20% but a payline at the 7^th percentile? There are several real-life reasons why paylines (the ones that use percentiles) can be either higher or lower than success rates.

• Applications that are not percentiled are still factored into the success rate calculation. Thus, funding a number of awards that are not assigned percentiles will increase the success rate without changing the payline.
• The success rate for a particular fiscal year is a reflection of the funded applications and can include applications reviewed in the previous fiscal year; whereas, the payline encompasses only applications reviewed in that fiscal year. So awarding applications that were reviewed in the previous year will also increase the success rate.
• The average quality of the applications assigned to an institute will also affect its payline. If an institute happens to receive a set of applications with very good (low) percentile scores, its success rate will be higher than its payline, all else being equal. For example, in fiscal year 2011, the NIGMS R01 success rate was about 24% but the midpoint of the funding curve occurred close to the 19^th percentile.

Check out more reports on RPG success rates broken down by year and IC at report.nih.gov – if you’re interested in other success rates, you can find them on our RePORT website as well.

Whew, you made it through. The difference between paylines, percentiles and success rates remains a confusing topic because of the compounding factors that rule out a simple linear relationship. You need to consider all the factors when assessing the potential for an individual application to be funded. Your best advisor on this issue, because of the differences in the ICs and programs, is your NIH program official. Give him or her call.

The numbers for fiscal year (FY) 2012 are in. Here are some facts about applications and awards in FY2012, compared to FY2011:

This 2012 data, and data from past years, can be found in the NIH Data Book. This is the first place to look for summary statistics on NIH awards — data and charts are exportable for easy incorporation into reports, presentations, or your own blog posts.

Looking back on these data, the first thought that comes to my mind is, “We made it.” Despite a flat budget and complex fiscal times, we maintained last year’s success rate and slightly increased the amount of award dollars that went to research project grants. We continue to strive to maintain a diverse portfolio of biomedical research, and keep this important work moving along quickly.

Even though the current fiscal year remains uncertain, we know that above all, it is critical to support your continued work on innovative science. Stay tuned for more data.