Chapter 3. Findings (continued)

3.4.3. Collaboration

Collaboration among the CERTs investigators and their CERT and non-CERT work was characterized by the terms synergy, inter-digitate, collaboration, networking, and leveraging. Developing collaborative relationships is a process; as one CERT investigator said "we've gone out to a number of resources within the university to create synergy... it takes time...we've made real strides." The CERTs investigators offered a few examples of projects in which they were collaborating with individuals from other CERTs (the exception more than the rule); a particular example that was often raised as illustrating CERTs collaboration was the Patient Safety grant. A Steering Committee member explains:

The CERTs agreed that, instead of competing with each other, the seven centers would collaborate, and all sign on to the HMO CERT's application, to be sure that all of the talent from all of the CERTs would be brought to bear on that program.

Therefore, the HMO CERT led the grant and involved the other CERTs centers as collaborators. Collaboration was in part driven by evolution toward the idea that "... a center could get more done collaborating and finding synergies than one could ever do alone." (SC) More specifically, "these centers have been doing research for a long time, but now the centers have a forum to discuss in advance the potential impact of their work with others in addition to the project officer of a federal agency."(SC).

The Risk Series was characterized as a key event by CERTs investigators, but others also recognized it as a venue that brought the CERTs together to collaborate if not literally with one another, to at least contribute to the topic of risk and frame their research toward that end. One CERT investigator said that one of the risk series "stimulated this project. So I had not even thought about this ... I attended those sessions ... and that prompted me to think about this project and it led to me doing this work as part of our CERTs renewal application." The investigator explained further, "This did not come up as a specific topic at that conference, but I went and heard all the discussions about risk communication, risk awareness, and risk behavior and I thought, "well I work in musculoskeletal diseases; nonsteroidals are really an important area." Therefore I applied the methods and the line of thinking." (UAB).

When respondents identified examples of collaboration they would refer to the scientific calls, which offered a forum to obtain feedback from other CERTs and identify opportunities for collaboration. As one investigator said:

I think this is an important and innovative process we have in the CERTs which is forcing this collaboration across groups that have common interests and use similar methodologies... it's a remarkable example of collaboration across universities.

The descriptions of the CERT collaborations have more to do with coming together to share ideas and offer feedback (and possibly resources) and less often to collaborate on research projects. However, the CERTs research centers were also groups of individuals or actual centers prior to becoming CERTs, "as great as they are, they were not chosen to be a network because of how well they work together." The CERTs certainly demonstrate intra-CERT collaboration, however inter-CERT collaboration has been limited. This is in part, as the UNC CERT stated, that because they are focused on pediatrics and the others CERTs are focused on primarily adult populations, there are few opportunities for collaboration other than on methodologies. However, there are opportunities that the investigators stated they were exploring with other CERTs. One investigator stated, "we are hoping to collaborate with them" while another investigator identified the crossover in one of the new CERT's work with the research they had done. Other respondents, including steering committee members, perceived that there was limited cross-CERT collaboration:

The hardest thing has been finding a theme that really allows these centers to work together around a particular concern or issue... most multicenter trials at NIH come with large amounts of money to support centers... it is hard to get people to work together when there's not a clear research question, hypothesis or priority program that everyone's working on and only a handful of support staff... it is hard when grants are small and there is not a clear direction around what it is that you want done through a coordinated effort.

Although the CERTs have had some collaboration given the academic environments in which they are based, as many respondents acknowledged, collaboration is not the primary culture of academia. One respondent contends:

The initiative that CERT investigators have put toward collaborative effort... it is atypical of what happens in academic spheres... the competitive nature of academia normally precludes such collaboration. Scientific calls sharing results, discussing meaning, is actually very risky. Having guiding principles and really making this a true collaborative agreement the way it was from the program's initiation. ... there was so much pressure on AHRQ, especially in the political sphere to become much more directive oriented... don't answer any other questions... don't think outside the scope ( e.g. Decide Network)...

Additionally, a couple of the steering committee members identified the collaboration as a struggle for the CERTs to identify common therapeutic areas and topics to truly collaborate on. One of the federal agency representatives also stated that collaboration also requires greater funding or can be a greater financial burden than not collaborating with one another.

3.4.4. Public-Private Partnerships

We developed the idea of a public-private partnership activity, which we were able to do through cooperative agreements, to determine whether we could also obtain outside funding and to provide leeway to the centers to find partners so that they could fulfill the requirements of the legislation. (AHRQ)

The CERTs program was constructed to support and facilitate partnerships, primarily to leverage funding. The CERTs program espouses a public-private partnership as a key value, stating, "For our results to apply to the 'real world,' the research must reflect a collaboration of groups with different perspectives and resources: patients, health care providers, government, academia, delivery systems, payers, purchasers, and manufacturers of medical products."⁵⁴ The CERTs have two different levels of engagements with partners. One is project partners who work with CERTs on research and education projects and the other is PATHs partners who partner with the program, attend the annual meeting, and essentially represent a sector involving no more than one person per organization.⁵⁵

Though the CERTs program was structured to create and depend on partnerships, the program devised rules of engagement given their goal to be an unbiased, national resource of researchers conducting research that otherwise would not be conducted (e.g. by the pharmaceutical industry).

Through the national CERTs network we've set up principles that are very rigorous to allow us to discern whether projects that we want to consider CERTs projects satisfy a set of criteria...the private-public partnership committee reviews all the proposed CERTs projects and determines whether or not they satisfy those principles. (UAB)

We compiled from lists of partners provided on CERT Web sites and upon request of staff at each CERT the partnerships between each CERT and its partners and the CERTs program as a whole. The data help to provide an indicator of the quantity, scope, and diversity of CERTs' partners. The partnership levels can serve as indicators of a CERTs involvement and collaboration in both the research and practice communities as well as indicate the potential for practical application of their research. The prominence of certain partner types also serves as an indicator of a CERT's primary research focus.

The Penn CERT had the highest number of partners (47) between 2002 and 2005 (Exhibit 11). Across the CERTs, the medical products industry (including pharmaceutical companies) was the most common partner, followed closely by federal organizations. The Duke and UNC CERTs displayed similar partnership levels. A UNC CERT investigator touted, "we developed a most interesting range of public-private partnerships... we understood what AHRQ wanted to see in terms of crafting genuine public private partnerships." The Duke CERT held twenty-nine different partnerships across the evaluation period, also most frequently partnering with members of the medical product industry. The UNC CERT entered into twenty-eight partnerships, most frequently either with health care delivery organizations or "other" organizations.⁵⁶ The UAB CERT entered into twenty-one partnerships between 2002 and 2005, and an Alabama CERT investigator said that, "partnering is the thing that we believe we've done the best" (UAB). Alabama's partners were most commonly from federal or other organizations.³⁹ The Arizona, Vanderbilt, and HMO Research Network CERTs each has fewer than ten different partnerships between 2002 and 2005. Both Vanderbilt and the HMO Research Network partnered most frequently with federal organizations. The UAB CERTs most frequent partners were from federal organizations or others. Additionally, the CERTs collaborated with partners representing local, regional, and national organizations and government agencies, non-profit organizations, and private companies.

Some CERTs investigators characterized the nature of their relationships with partners. The CERTs worked with partners to obtain data, have access to patients, to disseminate findings, to impact the partner's policies, and as collaborators on the research. For example, CERTs collaborated with professional societies to disseminate findings. In another example, a CERT worked with a professional society to administer a survey about clinical guidelines to ultimately improve the society's guidelines. One CERT investigator characterized her experience as positive and gratifying to work with a professional society to increase dissemination. Other CERTs listed partners as the sites of data collection. Another investigator included access to a professional society partner as one of the impressive resources that the CERT makes available to a researcher. The HMO CERT, in contrast to the other CERTs, is structured as a network or built-in set of partnerships, and the Vanderbilt CERT has a longstanding history of partnering with the state's medical assistance program (TennCare).

I'm impressed with the collaboration around not only what we were funded for but with how it has expanded to involve all kinds of other organizations, industry, and government in ways that I have not seen in most other projects which are funded by the federal government... I don't know of any other government-funded project that has gotten this kind of cross collaboration. (UAB)

The CERTs have had partnerships for many reasons including:

• Data (e.g. pharmacy data, outcomes data, VA data, MA data).
• Databases (i.e. membership database—cardiologists; MA, VA).
• Dissemination.
• Evaluation (e.g. risk management program for a CV medication).
• Data Collection sites (e.g. hospital, academic health centers).
• Leverage funding to use a partner to evaluate or conduct research locally or regionally to move to the national level (leveraging funding of CERTs work).
• Educational intervention (with partner).
• Co-investigators or true collaborators, scientific collaboration.
• Administration of a grants program for a disease organization (MD Arthritis Foundation).
• Access to patients.
• Provided venue for conducting research, community clinics, methadone clinics, pain center; other centers (academic).
• Other medical school or health care professional school.
• Blood glucose monitors provided by the partner for research.
• Private research organization, collegial partnerships and collaboration on projects (Partner—provided part of salary (leverage funding), provided access to data, and capacity to study some of these questions in the veteran setting.)
• Publications.

"CERTs has facilitated and leveraged our ability to partner in a dramatic way." (UAB)

The CERTs has program partners or PATHs partners. "The Partnerships to Advance Therapeutics program aims to facilitate opportunities for public and private organizations to collaborate on research and educational projects to optimize the use of therapeutics."⁵⁷ We characterize the different CERTs PATHs (Partnerships to Advance Therapeutics) partners that were involved with the CERTs from 2002 through 2005 in Exhibit 12.

Exhibit 12 CERTs Funding Sources

	Penn	HMO	Arizona	UAB	UNC	Duke	Vanderbilt
AHRQ CERTS	12.8%	68.0%	60.0%	25.9%	83.6%	74.4%	40.0%
AHRQ (other)	22.3%	5.0%	NA	NA	NA	3.9%	NA
NIH	42.8%	NA	NA	46.7%	NA	NA	20.0%
Government (other)	12.3%	15.0%	25.0% (FDA)	12.8%	0.3%	0.5%	20.0%
Private (Foundation/ non-profit)	1.9%	NA	15.0%	NA	5.7%	NA	NA
Private (pharmaceutical industry)	6.5%	10.0%	NA	0.8%	NA	18.7%	20.0%
Private (other)	NA	NA	NA	NA	NA	0.3%	NA
Academic (host health system)	1.5%	NA	NA	NA	3.3%	1.9%	NA
Academic (host university)	NA	2.0%	NA	1.1%	5.0%	NA	NA
Academic (other)	NA	NA	NA	0.7%	1.6%	0.3%	NA
Other63 (unspecified)	NA	NA	NA	11.9%	NA	NA	NA

The number of PATHs partners remained fairly steady between 2002 and 2005, rising from thirty-one partnerships in 2002 to forty-one partnerships in 2003 before dropping to thirty-five partnerships in 2005. In each year there was a diverse array of PATH partner types. Professional societies were the most common PATH partners across all four years followed by the Medical Products Industry and partners categorized as "other."⁴⁰ The "public-private partnership model encourages a responsiveness to what questions need to be answered." (CC). While the CERTs program was designed to encourage public-private partnerships primarily to leverage funding, the partnerships have also helped to further dissemination, identify research collaborators, provide unusual data sources, and further impact policies.

Return to Contents

3.5. CERTs Program Strengths & Successes: Results of the Appreciative Inquiry Exercise

As described above, while the purpose of this evaluation is to analyze the impact of AHRQ's Pharmaceutical Outcomes Portfolio and to determine if the program is moving toward its goals, the "Appreciative Inquiry" focuses only on those aspects of the program that have promise for the future. In addition, this technique can help encourage favorable organizational change among Portfolio stakeholders. The methodology was designed to answer these research questions: (1) What do various stakeholders view as the most successful processes and outcomes of the CERTs? and (2) How can this information be used to maximize, leverage, or build upon success in the future?

3.5.1. Strengths

Current Strengths of the Program

The current strengths of AHRQ's Pharmaceutical Outcomes Portfolio as perceived by the various participants can be categorized into the following five areas:

• Collaboration.
• Cross-Disciplinary Composition of CERTs Researchers.
• Flexibility of Program and Researchers.
• Role of Steering Committee.
• Role of Coordinating Center.

Collaboration

In terms of collaboration, Participants cited the following examples of collaboration as being among the greatest strengths of the CERTs:

• Collaboration among different CERTs.
• Collaboration between CERTs and the Steering Committee and/or Coordinating Center.
• Collaboration between CERTs researchers and outside partners, both public and private.

"Through collaboration on the CMS proposal, we were able to achieve spontaneous collaboration with political forces." Group Readout.⁵⁸

Key stakeholders and AI participants highlighted the following outcomes of such collaboration:

• Overall greater productivity within the CERTs and within the Portfolio at large.
• Additional funds, e.g., funds obtained for the Risk Series.
• Equitable distribution of AHRQ's supplemental funds.
• Greater leverage of external partnerships.
• Number and "prestige" of articles published.

Cross-Disciplinary Composition of Portfolio

In addition to collaboration, participants felt that one of the greatest strengths of the Portfolio was the fact that it brought together people from a diverse set of backgrounds, each with their own unique yet complementary perspective. In particular, AI participants pointed to the cross-disciplinary nature of the Steering Committee and the value that diversity brings to the overall program in terms of new ideas, partnerships, resources, and possibilities for future research.

Flexibility of Program and Its Members

Participants felt the flexibility of the Portfolio with respect to its overall design and among its researchers was also one of the program's greatest strengths. For example, participants highlighted the flexibility and freedom to be creative:

• Developing new initiatives.
• Identifying and developing new partnerships.
• Responding quickly to new proposals (e.g., CMS proposal).
• Addressing new and emerging issues.

Role of Coordinating Center

Participants felt that the successes of the program could also be attributed to the highly effective role of the Coordinating Center as thought leaders and as honest brokers between all of the different stakeholders and between the different stakeholders and outside partners.

Role of Steering Committee

Finally, the role of the Steering Committee was singled out as one of the strengths of the program for the following reasons:

• The extensive leadership and involvement of committee members.
• Access to other scientists or researchers, potential partners, etc. via the Steering Committee members.
• Scheduled networking events and opportunities to share with one another.
• Opportunity via the Steering Committee and its contacts to publicize and disseminate the success of the program and specific research outcomes.

"The Steering Committee provides an informal and formal network that helps amplify what the work does."—Group Readout.

Life Sustaining Forces

In addition to asking AI participants and stakeholder respondents to describe the Portfolio's greatest strengths, they were also asked to articulate the underlying forces or elements that help drive and sustain the CERTs program. Asking such a question helps to uncover some of the less tangible yet equally important program forces that may be enabling and/or fueling the program, such as shared values, commitments, or vision.

John Eisenberg's vision and the text of initial CERTs' RFA

Several respondents noted that it was the initial vision of John M. Eisenberg, M.D, the former director of AHRQ, as articulated and expanded upon in the CERTs' RFA that not only originally engaged them, but continues to inspire them in their work.

Excerpts from the Original CERTs' RFA, January 27, 1999 Purpose The Agency for Health Care Policy and Research (AHCPR) invites applications from non-profit organizations to establish Centers for Education and Research on Therapeutics (CERTs). CERTs is a three-year program that will support demonstration Centers. These Centers will evaluate, develop options and methods, and conduct and perform pilot studies. These studies will consist of state-of-the-art clinical, health services, or laboratory research to increase awareness of the benefits, risks and effectiveness of new uses, existing uses, or combined uses of therapeutics. This demonstration program seeks new and more effective ways to develop, translate and disseminate objective information on therapeutics to health care providers and other decision makers to improve practice. In addition, CERTs may selectively develop protocols and possibly undertake pilot studies on the comparative cost effectiveness and safety of medical products. This will be accomplished with data on appropriate therapeutic usage and outcomes; and the identification and prevention of medical errors and adverse effects. The long-term goal of the program will be to improve the quality of care while reducing costs. Research Centers Work carried out by each Center is to be multidisciplinary and must address various health care providers, settings, and geographic areas. Multidisciplinary research may involve scientists in medicine, pharmacology, epidemiology, engineering, pharmacy, nursing, human behavior, statistics, economics, organizational behavior and related fields. The long-term goal of the program will be to improve the quality of care while reducing costs. In addition: The Center will have demonstrated expertise in dissemination and translation of research on therapeutics into practice. The Center has demonstrable evidence of a sophisticated understanding of health care systems and current quality improvement strategies. The Center should have experience in working with health system leaders to translate research into practice, with potential for developing partnerships between the research centers and health care systems to enhance opportunities for broad scale implementation. The Center should have experience in leading multi-center research teams.

"This is exactly what needs to be done and this is exactly the mission that I have in my career."—CERTs PI.

"It was the compellingness of the vision that attracted everyone to the issue in the first place."—CERTs PI.

AI participants explained it was not only the fact that they were inspired by this vision, but that this vision was a strong one from the beginning (i.e., clearly articulated and practical), which has enabled the program to evolve and grow.

Commitment to Practical Science

Many Participants also cited their shared commitment to practical science as being one of the driving forces for the success of the CERTs program. Similarly, others described their commitment to applied science as a "shared altruistic goal" that unites them in their work and desire to see the program succeed.

"This is a practical science, only a few steps away from applying it to practice and changing behavior."—AHRQ member.

Proof of Concept

Finally, others commented on the fact that the program has grown, produced tangible outputs, and is successfully addressing all of its objectives as being one of the self-sustaining forces that has not only validated the concept of a network of research centers collaborating and leveraging public-private partnerships, but also provided inspiration to continue to improve upon it.

"CERTs provide an excellent opportunity for longitudinal and coordinated thinking. It is a real proof of concept."—Group Readout.

"We produce research that matters."—Group Readout.

Furthermore, participants noted that the government has affirmed the program and its efforts.

"HHS sees the CERTs as a strategic asset."—CERT PI.

3.5.2. Partnership Successes

Participants had very little trouble citing the successes of the CERTs so far. They included the following:

Partnership Successes

• CERT to CERT partnerships.
• CERT partnerships with federal agencies.
• CERT partnerships with the Department of Health and Human Services (HHS).
• Other more specific examples:
  • Patient Safety.
  • Risk Series.
  • CBER /UAB/Duke—TNF Project, Arthritis Foundation.
  • TMR—FDA/Duke/STS.
  • Direct to point interventions—"BB".
  • UNC/AAP—Rickets + ADHD.
  • CERTs/CDC/FDA—Adverse Reactions.
  • CAQH/NCQA/Duke.
  • AHIP—HMO.

Successes in the Dissemination of Information

• North Carolina Rickets—The strong dissemination effort helped to resolve the rickets issue in North Carolina. Local innovation informed larger entities of the problem and the Steering Committee collaborated with these forces to represent the interests of the North Carolina communities. Results include new policies and guidelines, vitamin D coverage and most importantly reduction in rickets prevalence in North Carolina.

Successes in Setting or Influencing the National Agenda

• The Risk Management effort which changed the way the FDA and other large programs think about risk management.

Return to Contents
Proceed to Next Section