NIH Extramural Nexus

We recently received a letter from a group of extramural scientists expressing concerns about the sunsetting of the A2 applications. I thought that the entire NIH research community would be interested in reading our response….

Many thanks to you and your colleagues for your letter describing concerns about the policy sunsetting the A2 resubmission applications, and for your input on the partial or total restoration of A2 applications. The large number of signatories on the letter demonstrates the depth of concern on this issue. As you know, we at NIH have given the issue of the length of time it takes to fund meritorious science considerable thought. The policy to sunset A2 applications was developed with substantial deliberation and has been in place for over two years. It was published in the Federal Register on October 8, 2008, with implementation beginning on January 25, 2009, as part of NIH efforts to Enhance Peer Review.

One of the concerns expressed at the time the policy was being developed was that applications were piling up as A1s and A2s in a “queue” awaiting funding. Data were collected showing that grants funded on their first submission (A0) had dropped significantly during the previous five years (See the Feb. 2008 NIH Peer Review Self-Study page 33). Furthermore, virtually all applications that initially scored in the top 20% were eventually funded but only after “waiting” one or two years in the A1/A2 queue. The overwhelming recommendation was to find a way to fund the same number of applications containing the most meritorious science earlier. Various strategies were discussed, and after additional input and analysis, Dr. Elias Zerhouni, then Director of NIH, decided to sunset A2 applications.

Is the policy working? It certainly has achieved the intended goals: the number of applications funded as A0s is increasing and there is no queue piling up at the A1 level (Fig. 1). In other words, the proportion of funded A1 applications has remained constant while the proportion of funded A0s has risen dramatically. In the near future more grants will be funded as A0s and over time, this should translate into a decreasing likelihood of highly meritorious applications needing even one resubmission.

Figure 1: Distribution of each year’s new R01 awards by version

There is little doubt that some great science is not being funded because pay lines are decreasing, regardless of the number of permitted resubmissions. Restoring A2 applications will not change that picture and will increase the time and effort required for writing additional resubmissions.

The new A2 policy also appears to not have had a negative effect on new or early career investigators. As shown in figures 2 and 3, the new and early career investigators are doing relatively well, thanks also to NIH guidance to Institutes to fund similar percentages of R01 applications submitted from new and established investigators. Even with the new resubmission policy, the percentage of R01 funded applications from new investigators has increased to the present 30% of total competing R01 applications. Reinstating the A2 policy could particularly affect new and early career investigators who are now experiencing a shorter time to award (Fig. 3), thus allowing them to conduct independent research earlier in their careers.

Figure 2. Awards of new, unsolicited R01 grants to new NIH investigators.

Figure 3. Investigators’ time-to-award. New, unsolicited R01 applications, by fiscal year of original A0 application.

Another concern in your letter was the uncertainty in the scientific community about what constitutes a new application versus an extensively modified application. The policy is the same as the one that we have had since 1997 the only difference is previously, the policy applied to A3s and now it applies to A2s. But we agree with you that the determination about what constitutes a new application is not always easy and some applications are in “gray areas.” To enhance fairness, consistency, and transparency, we have instituted a formal appeal process which allows the applicant to provide a rationale for reconsidering the decision.

In addition, it is important to note that each Institute and Center uses scientific priority setting to inform their funding decisions and invest funds strategically to best meet their mission. In some cases, this means that applications are selected for funding that fall outside of the payline while others that fall within the payline may not be selected for funding. This flexibility allows NIH to fund highly meritorious and programmatically relevant A0 and A1 applications even if they fall outside the payline. Therefore, it is unlikely that permitting the review of additional revised applications will substantially change the science that NIH supports.

Thank you again for sharing your thoughts. We will continue to evaluate the outcomes of this and other peer review policies and have planned to include the issue of sunsetting the A2 application in the next round of peer review enhancement surveys. We will continue to adjust our policies to adapt to scientific progress and changes in fiscal realities.

Sincerely,

Sally J. Rockey, Ph.D. Deputy Director for Extramural Research

Lawrence A. Tabak, D.D.S., Ph.D. Principal Deputy Director

I wanted to let you know that we posted our 2010 OER Report yesterday. My office provides much of the NIH infrastructure that allows grants to happen. If you are interested in learning more, I suggest you take a look at the report and post a comment if you have any questions about what you read.

We’re always interested in data, and at NIH we use data to examine the impact of new policies. Among the changes NIH implemented under its Enhancing Peer Review initiative was the assignment of scores to each of five individual criteria for research grant applications: significance, investigator(s), innovation, approach, and environment. The purpose of these criterion scores is to provide additional information to the applicant, but today I am going to use them to show you how we can examine reviewer behavior.

This topic will be familiar to those readers following the NIGMS Feedback Loop. In his latest post on this topic, NIGMS Director Dr. Jeremy Berg presented an analysis conducted by my Division of Information Services. I’d like to elaborate a bit more on this and provide some additional analyses. 

Note that the data presented represents 54,727 research grant applications submitted for funding in fiscal year 2010. Of these, 32,546 applications were discussed and received final overall impact scores.

Below you see distributions of the criterion scores (averaged across reviewers) and the numerical impact scores. (50% of impact scores fall within the colored boxes with 25% below the median in dark blue and 25% above the median in light blue). These distributions reveal differences in the use of the rating scales by reviewers. It appears that reviewers use the full scoring range for all 5 criteria and the impact score (the lines below and above each colored box range from 1 to 9), but you will notice that for the approach criterion reviewers distribute their scores more widely than for the other criteria.

Also, the criterion scores are moderately correlated with each other and with the overall impact score.

As expected, when the criterion scores are subjected to a factor analysis, their inter-correlations result in two  factors—one on which significance, innovation, and approach load heavily—accounting for most of the covariance among criterion scores, and a second factor with a smaller portion of covariance involving investigator and environment. 

For applications receiving numerical impact scores (about 60% of the total), we used multiple regression to create a descriptive model to predict impact scores using the applications’ criterion scores, while attempting to control for ten different “institutional” factors (e.g., whether the application was new, a renewal, or a resubmission). In the model, scores for the approach criterion had the largest regression weight, followed by criterion scores for significance, innovation, investigator, and environment. The same pattern of results was observed across multiple rounds of peer review and institute funding decisions.

As Jeremy noted in one of his posts, it may be tempting to over interpret such complex data, so we continue to explore their implications and will watch to see if and how these patterns evolve over time. I’m sure we’ll be hearing from you on your interpretations of these data, which will be interesting and informative.

I’m packing my bags for Scottsdale, Arizona…well, not exactly yet. After this cold winter in the Washington, DC metro area, I’m really just mentally packing. Each year NIH puts on two Regional Seminars on Program Funding and Grants Administration. The first one will be in Arizona in April (27-29), and the second will be in Ft. Lauderdale, Florida in June (22-24).

One of the most rewarding aspects of my role at NIH is the opportunity to meet and exchange experiences with students, investigators and administrators new to working with NIH. To top it off, I get to be there with about 35 of my fellow NIH and HHS policy and grants management, program, and review officials to meet and share with more than 500 seminar attendees. 

I invite you to join me at one of this year’s seminars and attend one of my sessions, perhaps NIH and You: 2011, New and Early Stage Investigators, and/or a session called Rock Talk, which, like this blog, will be an informal forum for discussing the hottest topics affecting the NIH extramural research community.

If you think you’d like to join us, visit our seminar website for more information. Whether you are a biomedical or behavioral scientist early in your career, a grad student or postdoc interested in mapping your career with NIH, a grant writer, or administrator trying to gain more insight into NIH while making some valuable contacts, these seminars will provide you the resources on your journey. I hope to see you there.