I. Assessment Plan
Plans for the assessment were developed by an organizing committee of NIGMS staff who had not previously been involved with any aspects of the glue grant program. This group included representatives from all NIGMS program divisions, the Office of Scientific Review, and the Office of Program Analysis and Evaluation (Roster of Organizing Committee). This group independently, but in consultation with managers of the program and NIGMS leadership, developed the assessment questions to be answered and the plan for assessment, selected participants for the expert panel review and assembled materials for the reviewers to consider.
The assessment plan included six major components:
- Request for information on outcomes from the glue grant investigators
- Request for data on utilization of Nature Gateways
- Independent bibliometric analysis
- Request for community input
- Expert panel review
- Consideration by the NIGMS advisory council
Because work on many of the projects has not been fully completed, this assessment is considered an interim outcomes assessment. However, because it is necessary to make decisions about the future of the program at this time, a timely assessment of outcomes to date is essential.
II. Material Assembled for the Expert Panel Review
The table of contents of information assembled for the expert panel review is listed in the attached Panel Review Binder. This includes the agenda, the roster, and the questions to be answered by the panel, which consist of Global Assessment Questions that apply to the program as a whole and Individual Grant Assessment Questions with respect to each of the funded projects.
Background information included in the binder or presented at the meeting included:
Additional background data was provided on previous evaluations of the glue grant program:
Information from Glue Grant Investigators: Responses received from the principal investigators are considered privileged communications for the purpose of this evaluation and are not being made public. These included very thorough and thoughtful responses to the questions posed and exhaustive appendices of publications acknowledging the glue grant awards, including the verbatim language of the acknowledgement and an explanation of the specific contribution of glue grant funding to the published work. In view of the fact that the Alliance for Cellular Signaling had ended several years before this assessment, the final report to the NIGMS advisory council was used as the basis for the review of this project without additional input from the investigators.
Nature Gateway Data: Because usage and potential continued support of the Nature Gateways associated with each of the projects is of interest to NIGMS, this data was assembled courtesy of Stephanie Diment, Nature Publishing Co. Information on another Nature Gateway, the Structural Biology Knowledgebase, was provided for comparison. The expert panel did not comment much upon this information, although they did extensively review the availability of information through these Web sites. Analysis of this Web site usage data is being carried out by NIGMS staff for internal discussion but is not available for public release.
Independent Bibliometric Analysis: The organizing committee developed a preliminary set of questions for study by an independent contractor capable of analyzing publicly available data for potential impacts of the glue grant awards. This formed the basis for a proposal by Juliana Blome, chief of the NIGMS Office of Program Evaluation and Analysis, to the NIH Office of Program Evaluation, which provided funds for the study. A specific proposal was received from Discovery Logic, a division of Thomson Reuters, Inc., to complete the study. Because of delays in getting a contract in place, only limited information was presented at the expert panel meeting. Preliminary work was presented by Joshua Schnell and useful feedback was received from the panel to further guide the study. The final report of the study is included as part of this report.
Request for Community Input: The organizing committee, in consultation with the glue grant management scientific liaisons and program officials and NIGMS management leadership, developed a set of community input forms to solicit opinions on the glue grant program. These were approved in their final form by the Project Clearance Branch, Office of Management Assessment, OD, NIH, as a request for information (RFI). The NIGMS established a glue grant assessment Web site that provided information about the glue grant program in general, links to each of the previously funded program Web sites, and a link to the community input forms Web site on the NIGMS Intranet. The availability of this opportunity for community input was advertised initially through the NIGMS Feedback Loop blog on November 5, 2010. The closing date of December 15, 2010, was extended January 15, 2011, in a second NIGMS Feedback Loop blog post and also advertised through a notice (NOT-GM-114) in the NIH Guide for Grants and Contracts on December 8, 2010. The opportunity was echoed on the Nature Gateway sites linked to each of the glue grants and in newsletters of the American Society for Cell Biology (ASCB), the American Society for Biochemistry and Molecular Biology (ASBMB) and the American Society for Pharmacology and Experimental Therapeutics (ASPET). The response site was open from November 4, 2010 – January 15, 2011. A third-party contractor, Ripple Effect Communications, Inc., was engaged to analyze the responses to this community input activity. Their final analysis is the: NIGMS Outcomes Assessment Community Input Summary. This summary was presented to the Outcomes Assessement Expert Panel at their meeting on March 21-22, 2011.
Selection of Expert Panel Members: Panel members were selected to represent a cross section of the scientific community, avoiding scientists who had direct financial support from the glue grant awards. Panel members included representatives of the areas of science that had been supported by each of the glue grants (i.e., signaling, cell biology, glycobiology, lipids and inflammation) to provide informed input on the impact of the glue grant awards on these fields. The panel also included members chosen for their knowledge in the area of bioinformatics. Additional members were chosen for their breadth and general knowledge of trends in scientific research. Many were current or previous members of the NIGMS advisory council. The panel also included former NIGMS Director Dr. Marvin Cassman, who was able to provide perspective on the circumstances under which the glue grant program was initiated. Institutional affiliations were not considered a conflict. Collaboration with one of the glue grant researchers or having made use of resources produced by the glue grants were not considered conflicts. The Roster of Expert Panel Members is attached.