Chapter 3. Findings (continued)

3.7.5. Recommendations for Improvement

Four investigators made unrelated recommendations for improving the progress reporting process. They recommend:

• The provision of information to AHRQ on an efficient, continual basis without burdening CERT investigators. This investigator believes that it would be helpful for AHRQ to be able to access the information included in progress reports real-time, not only through yearly reports.
• Progress reports be further used as vehicles of dissemination.
• Fewer and better coordinated progress reports.
• CERTS submit their progress reports in June or July, as opposed to May.

Coordinating Center staff members provided numerous recommendations for improvement. Two respondents emphasized the efficiency and promise of the CERTs Information Technology Transfer (CIT) project. One of these staff member highlighted that the CITs interface is better geared towards tracking results, which would be more helpful for both AHRQ and the Coordinating Center. This staff member specified that the Coordinating Center desires information on why a project was done, results, significance of those results, recommended actions, and instructions as to who should carry those actions out. This staff member also indicated that the CITs interface is compatible with many types of research and could potentially be used beyond the CERTs project. This sentiment was echoed by the second staff member. This staff member also highlighted the fluid nature of the CIT database and the hope that the CIT project will serve to make information more uniform and allow real time information to be pulled as needed.

One Coordinating Center staff member suggested that AHRQ provide feedback on how progress reports are used. An additional staff member requested that AHRQ make the progress reporting process as easy as possible for the CERTs, while another believed the progress reporting process should be made more fluid.

3.7.6. AHRQ/Coordinating Center/CERT Communication & Dissemination

Five investigators commented on the level of communication with AHRQ, both generally and as this communication relates to the dissemination of progress reports. Two CERT investigators praised the Coordinating Center for continuously updating AHRQ regarding CERT activities, so that AHRQ may appropriately respond, as well as ensuring that information provided by the CERTs is properly disseminated. This view contrasted with that of one CERT investigator who described communication with the Coordinating Center as difficult at times and praised one of that CERT's staff members for ensuring progress reports are disseminated and reviewed. An additional investigator from that CERT described their relationship with AHRQ as excellent and praised AHRQ for being supportive of changing research directions. This investigator contrasted this flexible attitude with significantly more rigid NIH grants.

One CERT investigator mentioned AHRQ's need to improve communications with the CERTs and attributed this to AHRQ's desire for non-interference from the CERTs. Some Coordinating Center staff members also highlighted the need to improve communications with AHRQ. One staff member indicated that pushing the Coordinating Center annual report through AHRQ is a very difficult. Another staff member mentioned AHRQ's concern that it is not kept adequately up-to-date on what is happening within the CERTs. This staff member perceives the once a year provision of information as appropriate but also states that this structure makes it difficult to ensure that every person needing the information provided by progress reporting receives it at the time it is needed. An additional Coordinating Center staff member stated how difficult communications with AHRQ are.

3.7.7. Portfolio Grantee Progress Reports

Administrative Burden

Only one of the four portfolio grantees interviewed commented on the administrative burden of the progress reporting process, describing it as reasonable. One grantee recommended more detailed reports that might result in a more onerous process but would be better suited for providing AHRQ needed information.

AHRQ Utility

Two out of the four Portfolio grantees interviewed questioned the utility of the progress reports to AHRQ. One highlighted the lack of clarity as to where the reports end up and who makes use of them. The second grantee states that progress reports are a low priority for them as a direct result of never receiving comments for feedback from AHRQ on the information submitted. This grantee recommends a more detailed and specifically aimed report that, despite increasing the administrative burden, would be more suited to AHRQ's needs.

Return to Contents

3.8. Progress towards meeting Agency and DHHS objectives from 2002-2005

The Pharmaceutical Outcomes Portfolio contributes to AHRQ's mission to improve the quality, safety, efficiency, and effectiveness of health care for all Americans and the mission of the U.S. Department of Health and Human Services' (DHHS) mission to protect and improve the health and well being of the American public. The previous findings provide detail on a significant portion of the outputs, outcomes, and impacts of the portfolio on research, which provides an overview of how well the Portfolio is doing towards meeting AHRQ and DHHS objectives. In this section, the relevant Portfolio objectives and the CERTs aims are provided in tandem with AHRQ and DHHS objectives. Additionally, the OMB PART goals, as an assessment-rating tool, are key goals against which the Portfolio and ultimately the Agency are measured. The accomplishments of the Portfolio that related to the OMB PART goals are also described in this section. Lastly, a description of the CERTs program, as a key entity funded by AHRQ as part of the Pharmaceutical Outcomes Portfolio and the program's processes and responsiveness to the Agency and DHHS priorities are described.

3.8.1. How do Portfolio objectives map to AHRQ and DHHS priorities?

Below is a description of how the objectives of the Pharmaceutical Outcomes Portfolio and the CERTs mission align with AHRQ DHHS priorities. The CERTs program mission is to conduct research and provide education that will advance the optimal use of drugs, medical devices, and biological products, which explicitly works to fulfill each of the five Pharmaceutical Outcomes Portfolio goals. The CERTs program mission coincides with the Portfolio goals, subsequently fulfilling the same DHHS priorities. The Portfolio's five program goals⁶⁴ are provided as well as the relevant AHRQ and DHHS priorities to which they relate.

1. Understanding benefits and risks. Expand our knowledge about the benefits and risks and outcomes of pharmacological therapies so that better decisions can be made about how and when to appropriately use pharmaceuticals to improve health.

• HHS Goal 2: Enhance the ability of the Nation's health care system to effectively respond to bioterrorism and other public health care challenges.
  • Objective 2.2: Improve the safety of food, drugs, biological products, and medical devices.
• HHS Goal 3: Increase the percentage of the Nation's children and adults who have access to health care services, and expand consumer choices.
  • Objective 3.4: Eliminate racial and ethnic health disparities.
• Healthy People 2010: #17 Medical Product Safety.

2. Advancing optimal use in clinical practice. Identify opportunities and strategies to increase the likelihood that patients will receive the right treatment at the right time from their health care providers across all practice settings.

• HHS Goal 5: Improve the quality of health care services.
  • Objective 5.1 Reduce medical errors.
  • Objective 5.2 Increase the appropriate use of effective health care services by medical providers.
  • Objective 5.3 Increase consumer and patient use of health care quality information.

3. Helping consumers derive maximum benefit. Identify and evaluate strategies for communicating the information that consumers need to make decisions about the appropriate use of therapeutics, in consultation with their health care providers.

• HHS Goal 3: Increase the percentage of the Nation's children and adults who have access to health care services, and expand consumer choices.
• HHS Goal 5: Improve the quality of health care services.
  • Objective 5.2: Increase the appropriate use of effective health care services by medical providers.
  • Objective 5.3: Increase consumer and patient use of health care quality information.

4. Informing policies. Provide government agencies, managed care organizations, employers and other decision-makers with scientific evidence to inform their decisions and evaluate the policy implications of their decisions.

This program goal is implicitly related to all of the content-related goals and objectives and the translation of those into policies, when applicable.

5. Supporting the extension of education and research. Support multi-disciplinary efforts to educate health care providers, researchers and students about how to evaluate the optimal use of therapeutics and apply scientific evidence to practice.

• HHS Goal 4: Enhance the capacity and productivity of the Nation's health science research enterprise.
  • Objective 4.1: Advance the understanding of basic biomedical and behavioral science and how to prevent, diagnose, and treat disease and disability.
  • Objective 4.3: Strengthen and diversify the pool of qualified health and behavioral science researchers.
  • Objective 4.4: Improve the coordination, communication, and application of health research results.

The Pharmaceutical Outcomes Portfolio program goals 1-3 all directly address AHRQ's mission to improve the quality, safety, efficiency and effectiveness of health care for all Americans. Program goal #4 helps to translate the fulfillment of 1-3 into relevant policies, and program goal 5 illustrates the capacity building component to sustain and further advance AHRQ's mission.

3.8.2. Progress toward meeting DHHS objectives

The Pharmaceutical Outcomes Portfolio appears to contribute to the United States Department of Health & Human Services (DHHS) Goals and Objectives (FY 2004-2009)⁶⁵ in a substantial way. A brief description of the contributions of the Portfolio has made to the applicable DHHS goals and objectives is provided below:

HHS Goal 2: Enhance the ability of the Nation's health care system to effectively respond to bioterrorism and other public health care challenges

We did not identify within the Portfolio a focus on bioterrorism. We identified one CERT PI serving on a committee regarding anthrax. However, within this goal is an objective that is a primary focus of the CERTs.

Objective 2.2: Improve the safety of food, drugs, biological products, and medical devices.

The CERTs program has as its explicit purpose "to conduct research and provide education that advances the optimal use of therapeutics (i.e. drugs, medical devices, and biological products)," ⁶⁶ which includes improving the safety of these products (go to 3.3 for further detail). For example, the CERTs have explored many issues on safety of drug use in populations like pregnant women (i.e. ACE inhibitors in pregnant women) and pediatrics (i.e. pediatric devices). The CERTs have conducted research on medication issues causing QT prolongation and Torsades de Pointes and increased the knowledge of practitioners, and appears to have contributed to the knowledge that led to the withdrawal of medications from the market because of this effect. Lastly, the CERTs program held a series of think-tank conferences to discuss with policymakers and other key stakeholders risk assessment, management and communication strategies.

HHS Goal 3: Increase the percentage of the Nation's children and adults who have access to health care services, and expand consumer choices

The Portfolio has not directly affected individuals' access to health care services, however there are examples of the Portfolio research contributing to understanding health care for ethnic minorities.

Objective 3.4: Eliminate racial and ethnic health disparities

We have not found that the Portfolio has made a substantial body of awards that are focused on health care disparities per se, although we did identify a few examples. The UNC CERT worked with colleagues to identify a disease that was considered essentially eradicated, specifically vitamin D deficiency rickets in African-American breast-fed infants. This study not only uncovered rickets, but impacted policy, by working with their colleagues and connections at the NC WIC program office to present on the findings and work with them to include vitamin D supplements on the WIC program. (go to 3.3.5 for further description). Additionally, the Arizona CERT has worked with a community clinic that serves a largely Hispanic population. The UNC CERT in a research project on antibiotic resistance identified ethnic patterns specifically "Hispanic children have higher resistance to certain species." Additionally, CERTs investigators shared examples in which their studies included multi-lingual materials. Lastly, Portfolio grants involving interventions, both produced materials for the different languages spoken in the intervention areas. In summary, there was demonstrated responsiveness when racial or ethnic groups are part of the research population, are identified as having special needs or issues or, are at greater risk.

HHS Goal 4: Enhance the capacity and productivity of the Nation's health science research enterprise

Our evaluation suggests that the Portfolio has enhanced the capacity and productivity of the Nation's research enterprise. The hundreds of projects, publications, presentations, and dissemination efforts have furthered the health science research agenda. Each of the applicable objectives is described below.

Objective 4.1: Advance the understanding of basic biomedical and behavioral science and how to prevent, diagnose, and treat disease and disability

The CERTs research to some extent has contributed to the advancement of this objective, however less so than other research agendas, as the CERTs research primarily involves health services research and pharmacoepidemiology. There are examples, particularly in the CERTs with their multidisciplinary focus to have multi-method and approaches to research. There are also a few examples of investigators linking health services research with basic biomedical research (e.g. grantee's antimicrobial resistance intervention involved health services data and biological data from nasopharyngeal cultures).

Objective 4.3: Strengthen and diversify the pool of qualified health and behavioral science researchers

Our evaluation has found that the CERTs program has demonstrated a commitment to strengthening the pool of health researchers. However, the data collection did not include determining the diversity of those trained or involved with the CERTs.

The CERTs program in particular has demonstrated an intentional focus to train and prepare future researchers within the CERTs context, particularly in health services research and pharmacoepidemiology. Not only does the CERTs program have as an aim to strengthen the pool of qualified researchers in therapeutics, but the program demonstrated this and investigators attested to it. Additionally, some junior faculty attributed the CERTs with helping them advance their careers and certainly further their exposure in therapeutics research. (go to Educational Outcomes for additional information).

Objective 4.4: Improve the coordination, communication, and application of health research results

The CERTs again have as an aim and has demonstrated a focus on how findings translate to practice with their efforts in changing guidelines and striving to disseminate their research beyond academic publications. The public-private partnership mechanism of the CERTs fosters relationships that also contribute to how the CERTs are able to communicate (e.g. professional societies) and apply health research results (e.g. change guidelines). Additionally, the Coordinating Center's continued developed of CIT, a mechanism that will centralize and facilitate collection, organization and dissemination of CERTs work will likely be a great product/output of the program and ultimately improve the coordination and communication of research findings. The Portfolio grants included intervention studies that included strong dissemination efforts and the continuation of those efforts.

HHS Goal 5: Improve the quality of health care services

The Portfolio has demonstrated an effort to improve the quality of health care services, specifically for the following objectives.

Objective 5.1: Reduce medical errors

The Portfolio has included research that explores medication errors. The CERTs have conducted research on medication errors in various settings and a number of topics specific to medications (go to Impact Level 1 and 2 for further detail).

Objective 5.2: Increase the appropriate use of effective health care services by medical providers

The Portfolio research includes examples of research that explore effective health care services and the uptake by medical providers.

Objective 5.5: Accelerate the development and use of an electronic health information infrastructure

The CERTs has conducted research on the use of electronic health information, particularly for communicating risk like point of care technologies, risk communications, and computerized physician order entry (CPOE).

Healthy People 2010: #17 Medical Product Safety
The Portfolio has also demonstrated a commitment to at least one of the goals of the DHHS Healthy People 2010, #17 Medical Product Safety. The CERTs have made significant contributions in the area of medical product safety (go to Outcomes and Impacts chapter 3.3). This is evidenced in the research the CERTs have conducted as part of the Prescribing Safety Program. Additionally, the CERTs' Risk Series aimed:

1. To explore current and future methods of managing the risks of FDA-approved therapeutic products to ensure maximum benefit and safety for patients.
2. To develop a research agenda to monitor the effectiveness of these risk-management approaches and their aggregate effects on patients and the healthcare system. These aims support and facilitate objective #17 of Healthy People 2010, "To ensure the safe and effective use of medical products," particularly in monitoring drugs and adverse events and providing useful data about the safe use of drugs.⁶⁷

The Portfolio as a whole and the CERTs program in particular have contributed to DHHS goals and objectives in the areas that would be expected given the focus on pharmaceutical outcomes and therapeutics. However, the Portfolio has contributed to some goals more than others. The Portfolio, specifically the CERTs program, has focused on developing the health care research workforce and understanding drug safety, whereas the Portfolio does not appear to have had as much of an emphasis on health care disparities. This is not surprising for a program that is charged with a breadth of research that encompasses all therapeutics, including drugs, biologics, and medical products.

Despite trying to fulfill the goals and objectives of DHHS and AHRQ, the CERTs have demonstrated responsiveness to the shifting priorities, as a CERT CC member describes:

The role per the RFA has not changed, but what we focus on at various periods of time can change. People's expectations (particularly at AHRQ) can change regarding how they perform their functions. During 2001 period, FDA was very focused on arrhythmia. So we focused on that and presented to FDA for public hearings. When CMS, with the Medicare drug benefit had a need, we applied for a contract that we thought we could add value and sent unsolicited proposals as well.

3.8.3. OMB PART Goals

AHRQ's Pharmaceutical Outcomes Portfolio is expected to contribute to meet specific Office of Management and Budget (OMB) the performance assessments using the program assessment-rating tool (PART). Below is a description of each of those goals and examples of the extent to which the Portfolio is working towards those goals and contributing to their achievement.

1. To reduce congestive heart failure re-admission rates during the first six months after initial admission by approximately 2% per year through 2014.
2. Decrease the inappropriate use of antibiotics in children by approximately 2.5% per year through 2014.
3. Reduce hospitalizations for upper gastrointestinal bleeding by 2% per year through 2014.
4. As an efficiency goal, improve the appropriate use of therapies to treat peptic ulcer disease by 20% by 2010.

The Duke CERT has conducted research and informed policies related to heart failure and which are important for the fulfillment of the first PART goal. The Duke CERT is conducting and conducted research on the use of evidence-based therapies and beta-blockers in heart failure, developed a registry of heart failure inpatients at Duke, studied the cost-effectiveness of treating heart failure patients with beta-blockers, and is working on an outpatient heart failure program for quality improvement. Additionally, in response to an AHRQ interest in methodological approaches to facilitate comparative effectiveness analysis the Duke CERT proposed to use inverse probability-weighted estimators to compare outcomes (survival and rehospitalization for heart failure (HF) after discharge). Besides the Duke CERT contributing to the fulfillment of the PART goals, a Duke investigator explains the impetus:

Duke has gone into doing heart failure research they had not planned to do until seeing the OMB goals and realizing they had investigators at their disposal.

This statement illustrates the responsiveness of the CERT to the OMB PART goals and ultimately AHRQ and DHHS' missions. Other CERTs conducted research that contributed to the other PART goals which is described below, however others did not explicitly state deciding to do particular research because of its fulfillment of an OMB PART goal. The Portfolio research has also contributed to the second OMB PART goal to decrease the inappropriate use of antibiotics in children to some extent, as described below. The Penn CERT has been the primary CERT that has contributed to this goal because of their thematic focus on anti-infectives.

The Penn CERT conducted research on the use of antibiotics in the treatment of acne and the impact on resistance. Another Penn study's results may be used to devise modifications in antibiotic use that may lead to the prevention of candidemia in critically ill children. A Penn investigator is developing a computerized intervention that, "using a simple computer game, a modification of 'Space Invaders,' will evaluate the usability and initial efficacy of this intervention in teaching children ages 6 to 16 when it is appropriate to request and/or use antibiotics." Another Penn study "compared predicted incremental overall antibiotic use and broad spectrum antibiotic use between strategies with looser guidelines and strategies using AAP criteria with and without tympanometry."

The HMO CERT conducted a retrospective cohort study to determine the rates of antibiotic use. An HMO CERT investigator has proposed studying changes in the incidence of serious bacterial illnesses as a result of decreasing antibiotic use in primary care settings; in the context of introducing pneumococcal conjugate vaccine. The HMO CERT is conducting a project on health plan member/physician education for judicious antibiotic use in children, on which they are collaborating with the HMO Research Network CERTs health plans, Council for Affordable Quality Healthcare (CAQH) and the Association of American Health Plans (AAHP) to reduce unnecessary prescribing of antibiotics through health plans nationally. The HMO CERT is conducting a study aimed at reducing the use of antibiotics in children.

The UNC CERT, with its focus on pediatrics, has contributed to the understanding of antibiotic use in children with projects assessing pediatric treatment guidelines that could encourage better adoption of practice guidelines (e.g. reduce overuse of antibiotics). Also, UNC is conducting a study to determine the impact of antibiotic prescribing at initial visit on the probability and frequency of acute otitis media (AOM)-related return visits among North Carolina (NC) Medicaid patients.

A Penn investigator describes the context related to the second OMB PART goal:

It's hard to know exactly what should be credited here, because certainly there's been more public awareness. Certainly the CDC in particular has invested in some public education and professional education campaigns... there have been many local activities of varying strength, and it's the kind of work that gets published. It gets us in the public domain and people thinking about it. ...I expect we're not at the targeted goal overall, but the data would suggest that in general, particularly in pediatrics, overall use of antibiotics has clearly gone down in these targeted conditions. I think that's been a little bit less true in adult and particularly in geriatric settings and less true in emergency care settings, that's one reason we focused there in our latest study, but the overall news is still generally good... but there has been an increased use of broader spectrum antibiotics, with unintended negative consequences [where] people are perhaps prescribing less but when they do prescribe they prescribe even more broader spectrum antibiotics than they really need to. The net effect, in terms of the public health goal, which is to reduce resistant infections and make infections more treatable, is very hard to gauge. ... in terms of a number of issues things are probably better today than they were five or ten years ago. I'm not sure that we've achieved the goal, and I don't know that our improvement is going to continue... but I would say there is good news...

Besides the projects focused on the appropriate use of antibiotics in children, the Portfolio has included research on the appropriate use of antibiotics in adults and research on antimicrobial resistance, including interventions to improve antibiotic use and resistance in communities. An AHRQ representative characterizes the CERTs' contribution to the OMB PART goals in the following:

There are a number of the CERTs that have pilots that could certainly be useful if you can disseminate them to the rest of the country as a way of approaching the national goals.

The Portfolio has primarily demonstrated a commitment to meeting the first two PART goals in terms of the research content and goal two more than goal one. This may be in large part due to the nature of the CERTs research centers and their thematic focus, specifically that the Penn CERT focuses on anti-infectives, the UNC CERT focuses on pediatrics, and the Duke CERT focuses on cardiovascular therapeutics. The CERTs investigators, key leaders on the Steering Committee, senior investigators, and PIs all stated a commitment to the PART goals, regardless of how challenging they are to achieve. An AHRQ representative describes the change in goals:

I think the goals changed a few years back... I think that AHRQ is moving towards a different role and so its programs needed to also change the role that they had. AHRQ is becoming more involved in education, dissemination, and implementation, and less of a research agency. It has fewer resources to fund much in the way of research so the CERTs had to adjust to that. ...I think there's still some transition going on, but that changing role is something that was imposed on the CERTs as opposed to it being internal, so there are challenges with that, it's almost a different set of skill sets. So we'll go from just being research on therapeutics, to research and education in therapeutics.

Return to Contents
Proceed to Next Section