Chapter 4. Summary and Conclusions

The four objectives guiding this evaluation were:

• Measure and assess to what extent the Agency contributed new knowledge as a result of its funding for children's health research (extramural and intramural) and disseminated and/or translated effectively its findings to meet AHRQ's strategic objectives of improving the safety, quality, effectiveness and efficiency of health care as well as wider DHHS strategic objectives.
• Measure and assess to what extent AHRQ's children's health care activities, i.e., its research findings, meetings, conference support, products, tools, etc., improved clinical practice and health care outcomes and influenced heath care policies over the past fifteen years.
• Measure and assess AHRQ's financial and staff support for children's health research as well as Agency internal handling of children's health grants, contracts and intramural activities research with/among other AHRQ programs, portfolios and activities and other DHHS and federal agency efforts.
• Measure and assess to what extent the Agency succeeded in involving children's health care stakeholders and/or create partnerships to fund and disseminate key child health activities.

Summary

Primary Objectives 1 and 3

We addressed Objectives 1 and 3 together by examining the number of activities, funding for those activities, and number of publications over time and according to strategic goals and portfolios of work.  In addition, we interviewed key stakeholders about their perspectives regarding support for children's health at AHRQ.

We found that over the past 15 years, there have been significant shifts in the number of child health activities funded, the amount of funding (both absolute and as a percentage of the AHRQ budget), and the strategic goals or portfolios of work funded.

Through most of the 1990's, AHRQ maintained a steady level of external activities related to children's health.  Starting in 1999, both the number of activities and the total funding directed towards children's health projects rose dramatically before dropping off again in 2003.  This corresponds to a relative increase in the proportion of the total AHRQ budget that went toward external activities related to children's health from 9 percent prior to 1999 to 28 percent during the 1999 to 2002 time period.  Since 2003, there has been a noticeable downshift in both the number of external activities and the funding for them. 

In examining the types of external activities, we found a shift toward research on health information technology and patient safety.  The proportion of activities addressing the AHRQ Strategic Goals in these areas increased from none to nearly one-half of the overall portfolio.  Similarly, activities that corresponded to the AHRQ Children's Health Strategic Goal on creating tools and nourishing talent for children's health services research went from 15 to 44 percent of the total.  Looking at AHRQ's external activities through the lens of the Portfolios of Research, we found that the activities addressing the health information technology and patient safety missions increased from 0 to 39 percent of the entire portfolio over time.  Each of these trends reflects general shifts in priorities for AHRQ as a whole.

The number of publications tracked the level of funding for strategic goals and portfolios of work.  Exceptions were that the percent of publications in the health information technology and patient safety portfolios were less than the percent of funding for those portfolios, but this is probably due to the fact that these activities have been funded relatively recently and so have likely not produced their full complement of publications.

Two recurrent themes regarding AHRQ support for children's health activities emerged from the key informant interviews.  The first was the issue of whether children's health should be considered (and funded) separately from other activities.  Several interviewees made a cogent case for theoretical, practical, and ethical reasons to consider children separately and made specific suggestions for how to accomplish this.  The countervailing sentiment was that, in an Agency with a relatively small budget, setting aside specific funding for children's health activities was likely to result in a children's health budget insufficient to accomplish any of the Agency's children's health strategic goals. Nevertheless, there are specific issues that are inherently child-only, such as research on SCHIP, for which it is appropriate to argue for set aside funding.

The second theme, related to the first, was the perception that children's health activities at AHRQ were dependent almost entirely on the efforts of individual Agency staff.  Interviewees noted that there were virtually no formal structures or policies at the Agency to ensure that children's health was adequately addressed. Interviewees praised Denise Dougherty, as Senior Advisor on Child Health, and Lisa Simpson, for their unflagging efforts to raise the profile of child health, and noted that compared to other special populations, child health has a more organized and effective constituency within the Agency.  However, they also noted that the Office of Senior Advisor has neither authority nor funds to ensure inclusion of this special population and relied almost entirely on personal persuasion.  They also noted that, with Lisa Simpson's departure, advocacy for child health at the level of the Office of the Director had diminished markedly. Moreover, within Agency Offices and Centers, it was up to individual staff to ensure that children's health was included in activities.  Interviewees lamented that there did not seem to be an institutional-level, coherent voice for child health at the Agency and wondered aloud what would happen to children's health research at the national level if AHRQ did not continue, in some important way, to lead.

Overall, the Agency has supported a broad range of child health activities and produced a substantial body of new knowledge in children's health.  There was a marked increase in activities and funding in 1999, followed by declines after 2002 and corresponding shifts, in external research grants, away from investigator-initiated grants to targeted solicitations increasingly focused on health information technology and patient safety.  These changes brought into stark relief questions regarding the place of children's health within AHRQ's overall mission and the corresponding lack of structural support (authority or resources) devoted to children's health.  Many key stakeholders worried aloud that children's health research is critically endangered and wondered whether AHRQ would continue to play a major leadership role in children's health on the national level.

Primary Objective 2

We addressed Objective 2, measuring and assessing the impact of AHRQ child health activities, through bibliometric analyses and case studies. In the Stryer analysis, we found that the majority of publications resulting from AHRQ-funded child health activities were descriptive findings. This is consistent with AHRQ's first children's health strategic goal—to contribute new knowledge about child health services. A substantial minority of publications evaluated or informed policy or practice. Relatively few publications evaluated interventions to improve outcomes. This highlights the difficulty of publishing studies that answer the Porter Question. Categories of external activities that had relatively higher Stryer scores included 'Using data to make informed choices,' 'Translating new knowledge into practice,' 'Health information technology portfolio,' and 'Prevention portfolio.' The average child health publication was cited 6.6 times and appeared in journals of impact scores average about 3.6. Unfortunately, we do not have comparative data on other AHRQ or non-AHRQ research programs, which are necessary to put these analyses in context.

The qualitative case studies and key informant interviews were meant to illuminate the processes by which AHRQ has an impact on child health activities. Both case studies—SCHIP/CHIRI™ and moving evidence to practice in ADHD and asthma—illustrated several lessons on maximizing the impact of AHRQ activities.

First, impact is maximized when structures are in place to encourage cooperation and communication among researchers and a variety of stakeholders.  The SCHIP/CHIRI™ case exemplified this lesson: The RFP was structured to require researchers to work with policy makers; AHRQ partnered with other funders and was able to benefit from the Packard Foundation's additional funding and also from its focus on dissemination and its ability to underwrite activities that foster collaboration (for example, dinner meetings); and the individual projects were required to set aside funds to support initiative-wide meetings and publications.  In the case of ADHD and asthma, one of the main impacts of AHRQ involvement was the ability to draw on the resources of CERTs to synthesize evidence, of PBRNs to change practice, and of tools to improve outcomes.

A second lesson from the case studies was that, in many of the child health activities funded by AHRQ, impact (disseminating relevant information to policy-makers in a timely manner, changing and documenting change in practice and outcomes) all too often relied on the individual efforts of PIs and AHRQ staff rather than being programmed into the activities.  With some notable exceptions (for example, CHIRI™), activities are seen as focused on generating products for academic journals.  Additionally, the AHRQ infrastructure was seen as being oriented towards the 'front end' of research grants—soliciting, reviewing, selecting, and funding—rather than the 'back end'—disseminating timely and relevant information to policy makers, documenting impact on clinical practices or outcomes, tracking and compiling the ways that AHRQ products are used by various stakeholders.

A third lesson from the case studies, and related to the second, was the tension inherent in serving the needs of multiple stakeholders.  In the case of CHIRI™, tension led to a creative solution that enhanced the impact of this set of activities. In other cases, the tension was not addressed as successfully.  Many interviewees alluded to the tension between the role of the Agency in funding policy-relevant research versus directly informing the policy debate.  Child health activities at AHRQ were seen as being more focused on the former, rather than the latter. Another tension is between the needs of academicians to generate peer-reviewed publications on the one hand, and the needs of policy-makers, clinicians, and families for timely, actionable information on the other.  Clinicians and family-oriented organizations saw AHRQ's child health activities as being more focused on the needs of academics rather than on their needs.  Another tension is the one between a broad spectrum of needs in children's health and a very limited budget.  Specific, focused investments designed to generate spectacular answers to the Porter Question were seen as key to the Agency's continued viability.

Overall, AHRQ-funded child health activities have had a substantial impact on the research community, as the bibliometric analysis demonstrates.  Impact on policy and clinical care was maximized when structures were in place to encourage collaboration among stakeholders and when dissemination and implementation were planned, not hoped for.

Primary Objective 4

The key informant interviews suggest themes regarding AHRQ's partnering with the children's health services research community, with other HHS agencies, with the policy community, and with the clinical/improvement community.

Interviewees perceived AHRQ as having been successful in nurturing a growing children's health services research community, through NRSA fellowships, conferences, and training grants.  However, they also noted that this same community is in danger of dissolving or moving to other areas of inquiry as funding in this area continues to be scarce.

In terms of partnering with other HHS entities, key stakeholders perceived limited success.  Interviewees at other HHS entities professed little experience or interest in partnering with AHRQ, despite substantial respect for the AHRQ personnel that they had interacted with.  The partnerships with other HHS entities tended to be around specific initiatives or committees and were seen as initiated or maintained at the program or project-officer level.  Interviewees could not cite examples of high-level inter-agency collaboration in children's health.

In terms of partnering with policy entities, CHIRI™ was a good example of successful partnering, but interviewees noted that there is little interest in children's health policy at the federal level.  Several interviewees suggested that AHRQ ought to further collaborate with state-level policy makers and with CMS around SCHIP and Medicaid.

Interviewees reported mixed success in partnering with the clinical and patient/family communities.  They suggested that far more could be done to partner with providers through professional societies such as the AAP and through organizations involved in improving children's health care.

Overall, AHRQ's partnering with other entities has been mixed, but promising opportunities exist.

Return to Contents

Conclusions and Recommendations

Although the focus of our analyses was on child health activities, many of the comments we received related to AHRQ as a whole, and it seemed that at least some of the child health specific issues that we identified may apply to other parts of AHRQ as well.  So while our conclusions and recommendations address child health specifically, some might have more general application.

Pursuant to Primary Objective 1, our evaluation shows that the Agency has contributed a substantial body of new knowledge as a result of its funding for children's health research (extramural and intramural) and has disseminated this new knowledge effectively in the peer reviewed literature.

It is difficult in any field to trace a direct line between research activities and improved health care activities or clinical practice, or between research and an influence on health policy. Nevertheless, pursuant to Primary Objective 2, our bibliometric analysis, case studies, and key stakeholder interviews suggest that AHRQ's children's health care activities have played an important role, along with activities of other child health stakeholders, in improving clinical practice and health care outcomes and in influencing specific heath care policies.  We note, however, that all of these analyses required a substantial effort on our part to identify the AHRQ research and projects that were related to child health, as well as to identify their outputs and effects.

AHRQ's objectives and budget have changed over the years, both overall and for children's health activities.  A tightening budget and a shift away from investigator-initiated external research grants have highlighted questions regarding the place of children's health within AHRQ's overall mission.  Pursuant to Primary Objective 3, our interviews and case studies show that individual Agency staff have performed heroically in assuring continued support for child health activities.  But the interviews and case studies also suggest that there is a lack of structural support (authority or resources) devoted to children's health and that this has limited AHRQ's financial and staff support for children's health research.

Pursuant to Primary Objective 4, the Agency has had mixed success in involving children's health care stakeholders and/or creating partnerships to fund and disseminate key child health activities.  AHRQ has been successful in helping to create a community of children's health services researchers and in disseminating policy information to state-level policy makers.  There has been some success in engaging with the practice community through entities such as the AAP, but more is needed.  There has been limited success in partnering with other HHS entities and with federal-level policy makers.

Building on this analysis, we offer the following suggestions for the Agency's consideration.

Through their children's health activities, AHRQ has successfully created and disseminated knowledge and engaged the pediatric academic community, but has been less successful in creating and documenting improvement and in engaging the children's health quality, practitioner, and patient/family community.  Answering "the Porter Question" (What effect is the research having on people?) for children will require AHRQ's child health activities to focus more effectively on identifying and pursuing opportunities to apply existing evidence, to implement change, and to document improved processes and outcomes.  It will also require the Agency to work more closely with clinical organizations such as the AAP and with quality improvement organizations in order to better leverage the strengths of each.  Experience with the CHIRI™ program as well as with AHRQ's work on asthma and ADHD show that this is possible, at least in well focused areas.  The CHIRI™ program required (and facilitated) collaboration between researchers and policymakers and in that way could be a model for collaboration.  However, it was organized around an ongoing federal project slated for possible reauthorization.  In the absence of a similar national quality improvement project, it would be necessary to pursue a more piecemeal approach.  We note that, with the restriction of funding for investigator-initiated grants and increasing prescriptiveness from DHHS regarding what funds can be used for, fewer opportunities exist for pediatric quality improvement research.

The fate of children's health at AHRQ seems to rest in the hands of a few extremely dedicated individuals who are forced to rely on their powers of personal persuasion.  In order to institutionalize child health as an Agency priority, certain structural and procedural changes are required.  A more formal and rigorous monitoring of Agency funding commitment by priority population should be instituted, with regular presentations to Agency leadership and identification of under-funded areas.  Endowing the Senior Advisor for Children's Health with institutional authority (for example, participating in AHRQ staff employee performance reviews) or with small amounts of discretionary funds to pursue promising opportunities to answer the Porter Question would substantially enhance effectiveness.  AHRQ should also examine how to increase the effective use of the contracts mechanism to pursue focused efforts to answer the Porter Question.

Given AHRQ's limited budget, partnering/leveraging other HHS entities is critical.  The initiative for partnering must come from the highest levels within the Agency, rather than relying on program-officer level contacts.  AHRQ should aggressively pursue collaboration with CMS around SCHIP and Medicaid.  We note that AHRQ staff are currently engaged in collaborative efforts with CMS around child health issues and that the Agency's authorizing language must guide collaborations.

Finally, we note that all of these analyses required a substantial effort on our part to identify the AHRQ research and projects that were related to child health, as well as to identify their outputs and effects.  AHRQ should consider building a system (or modifying existing systems) to reliably track the inputs, outputs, and impacts of its child health activities so that this does not have to be done on an ad hoc basis as for this study.  Such an infrastructure would provide a coherent and consistent picture of children's health activities at AHRQ, which would be a useful tool for both management and evaluation purposes.  In addition, given shifts in the amount and direction of funding, it is increasingly important to devote adequate resources to ensuring continuity in institutional memory and in institutional message. 

Return to Contents

Bibliography

Forrest, C. B., L. Simpson, et al. (1997). "Child health services research. Challenges and opportunities." JAMA 277(22):1787-93.

Gidwani, P., E. Sobo, et al. (2003). Laying the foundation: Identifying major issues in applied child health services research. Child health services research: applications, innovations, and insights. E. Sobo and P. Kurtin. San Francisco, Jossey-Bass: 25-66.

Roessner, D. (2002). Outcome measurement in the United States: State of the art. American Association for the Advancement of Science. Boston, MA.

Stryer, D., S. Tunis, et al. (2000). "The outcomes of outcomes and effectiveness research: impacts and lessons from the first decade." Health Serv Res 35(5 Pt 1):977-93.

Return to Contents
Proceed to Next Section