Evaluation Manual: Case Studies

Introduction to Program Evaluation for Public Health Program

Evaluating Appropriate Antibiotic Use Programs

Released April 2006

Print-friendly Version of the Case Study Section (71 KB, 17 pages)

The following case studies illustrate the use of CDC’s framework in evaluating appropriate antibiotic use programs. These case studies provide concrete examples of the steps followed, problems encountered, and solutions found in planning and carrying out an evaluation and interpreting and sharing its results. Furthermore, the case studies illustrate the usefulness of the logic model as a tool for both program planning and program evaluation. Finally, the case studies reinforce the importance of building evaluation into program plans from the beginning.

The cases described are hypothetical, designed to include some typical intervention activities and demonstrate some of the common issues raised during program evaluation. These cases are not intended to serve as models or blueprints for program design or evaluation. Instead, it is our hope that the challenges raised by these case studies will help inspire solutions in the field.

Case Study: Clinic-Based Education for Patients and Providers

Background

A state health department received funding from the Centers for Disease Control and Prevention (CDC) to design and implement an appropriate antibiotic use campaign. This state is not part of CDC’s Active Bacterial Core surveillance (ABCs), but the health department does have a sentinel surveillance system through which it collects and analyzes resistance data from several hospitals in the state. Epidemiologists, clinicians, and health educators within the health department identified antibiotic resistance and inappropriate antibiotic use as important problems in this state. Even though resistance rates decreased both nationally and within this state over the past several years, health department officials knew that the persistence of inappropriate antibiotic use could contribute to future increases in resistance rates. Doctors within the health department also reported that antibiotic prescribing rates remained high for viral upper respiratory conditions.

With CDC funding, the health department decided to hire a coordinator to organize a coalition of interested parties to develop and implement an intervention to promote more appropriate antibiotic use. Because CDC funds were limited, the group knew it would be important to evaluate this effort to ensure that resources were used wisely and to later advocate for continued funding.

Step 1: Engage Stakeholders.

Once the coordinator was hired, she began forming the coalition by identifying stakeholders to provide input on the development of the campaign and its evaluation. The program coordinator, who was housed within the communicable diseases branch of the health department, facilitated these early coalition meetings and invited health department staff with related interests and experience. Staff epidemiologists helped document the need for the program with surveillance data on local antibiotic resistance patterns. Other groups within the health department had considerable experience with community interventions and evaluations and thus were included in this effort. Public health nurses and health educators from maternal and child health talked about their experiences developing educational materials and campaigns for clinic use and described materials and approaches that had been effective.

Additional stakeholders for the evaluation of this program included physicians and nurse practitioners within the health department because program staff knew that any activities implemented within the health department clinics would need the support of the providers working in these clinics. As these clinicians joined the health department staff at coalition meetings and other planning meetings, they shared their ideas about what types of materials they would use in the health department clinics. They helped develop evaluation questions, and they helped program staff plan for the dissemination of evaluation results, particularly among healthcare providers. Physicians from the community were also identified and included through professional organizations (e.g., state medical associations), teaching hospitals, health plans, and independent practice groups. Program staff made efforts to include local physicians who were leaders in the community because they would be able to help set norms favoring appropriate antibiotic prescribing.

The coalition also made efforts to engage patients or consumers – those who would be affected or served by the program – because they knew that by talking with patients, they would better understand the factors influencing antibiotic use, and they could also promote a sense of ownership for the intervention and its evaluation. The coalition considered forming a consumer advisory group but felt they did not have the time or resources. Instead they chose to informally talk with patients in health department clinic waiting rooms, and they used these discussions to elicit information about knowledge and behaviors surrounding antibiotic use. Through these talks, the program coordinator identified a few particularly interested and outspoken patients and invited them to attend regular coalition meetings and continue to be involved in planning and evaluation efforts.

Step 2: Describe the Program.

The stakeholder engagement proved influential in development of both the program and the evaluation. Coalition members initially had very different ideas about how to reduce inappropriate use of antibiotics. Providers cited patient demand as the primary reason they prescribed antibiotics when they might not be needed. Therefore, they felt that educating patients would lead to less demand and reduced antibiotic use. Consumers, however, were convinced that doctors and other providers were responsible for over-prescribing, citing short office visits and complicated explanations about their diagnosis and treatment plans. The program coordinator was able to guide the coalition through a planning process that helped members identify multiple factors influencing antibiotic prescribing and antibiotic use. Program staff knew from formative research that knowledge, attitudes, and behaviors of both consumers and providers contribute to inappropriate antibiotic use. They also knew from evaluations of other programs that efforts targeting both consumers and providers have proven to be most effective. In the end, the coalition decided to develop educational materials for both patients and providers and chose to distribute these materials through the health department’s clinics in hopes of reaching a broad and diverse population. Because of the many references to lack of communication between patients and providers, they wanted to use the new materials to try to improve patient-provider interactions relating to antibiotic prescribing and use.

The coalition agreed that the overall goal of the program was to reduce inappropriate antibiotic use and decrease the spread of antibiotic resistance. Thanks to the extensive stakeholder engagement, the coalition also had a clear sense of the multiple paths they would need to follow. The resulting objectives and activities to meet this goal are summarized in the following table.

Table 1: Objectives and Activities

Objectives	Activities
Providers' Objectives: Increase adherence to appropriate antibiotic use guidelines. Increase provider knowledge and awareness of appropriate antibiotic use messages. Change social norms among providers to favor appropriate prescribing.	Providers' Activities: Develop provider educational materials and distribute in health department clinics. Provide community-based professional education for providers.
Patients' Objectives: Decrease patient demand for antibiotics. Increase patient knowledge and awareness of appropriate antibiotic use messages.	Patients' Activities: Develop patient education materials and distribute in health department clinics.
Patient-provider Communication Objectives: Improve patient-provider communication.	Patient-provider Communication Activities: Provide community-based professional education for providers. Develop educational materials for patients and providers and distribute in health department clinics.

For the patient education component, the coalition planned to develop and distribute health education materials (brochures, fact sheets, and posters) at the health department clinics. Educational materials were also developed for providers (detailing sheets modeled after those used by pharmaceutical companies), and these were mailed to physicians and nurse practitioners working at the health department clinics. Respected doctors in the community gave lectures at educational events for providers on topics such as appropriate antibiotic prescribing and tips for improving doctor-patient communication. Continuing medical education credits were provided for those attending.

These proposed program activities and their intended outcomes have been diagrammed in the following logic model. This model is a visual depiction of the activities and objectives listed above and shows the connection between specific activities and objectives (Exhibit 1).

Exhibit 1: Logic Model: Clinic-Based Education for Patients and Providers

Click to enlarge

Step 3: Focus the Evaluation Design.

During the stakeholder engagement (Step 1), the coalition had found that different stakeholders had very different priority outcomes for the project and therefore had very different ideas on where to focus the evaluation of the project. The group knew from early discussions with patients that they wanted to know if physicians would spend more time explaining whether or not they needed antibiotics and what they could do to feel better. The coalition had heard from doctors and nurse practitioners that they wanted to know if patients would ask for antibiotics less often and follow their prescriptions when they did receive antibiotics. Health department staff wanted to know if the intervention had any effect on antibiotic prescribing.

Since this evaluation was planned during the first year of the project implementation, the program staff decided that the overall purpose of the evaluation was to improve the program materials and strategies to increase the likelihood of reaching the program’s intended outcomes. In particular, staff wanted to know if patients read and understood the materials because they wanted to make changes to the materials if needed. Similarly, staff wanted to know if providers found the provider materials useful. They planned to make changes to the content and/or delivery of these materials if needed. Staff also wanted to know what providers thought of the community-based education and whether participation in these activities had any effect on communication skills or social norms.

Even at this early stage, program staff and other stakeholders expected to see some outcomes achieved. As mentioned earlier, stakeholders were interested in different sets of outcomes. Providers hoped to see decreased demand for antibiotics and increased adherence to prescribed antibiotics. Patients hoped for improved communication with their providers. Health department staff thought that the intervention would help to improve communication between patients and providers, resulting in decreased patient demand and greater patient satisfaction. Stakeholders decided to measure patient-provider communication because both patients and health department staff had explicitly named communication as an outcome of interest. In addition, improved communication was expected to contribute to decreased patient demand for antibiotics, which was the primary outcome of interest for providers.

Program staff chose not to measure antibiotic prescribing rates in the first year. Even though there was a high level of interest in this long-term outcome among the health department staff and clinic providers, the coalition decided to focus their outcome evaluation efforts on patient-provider communication (a short-term outcome) since resources for this evaluation were limited. In addition, because the overall purpose of the evaluation was to improve program efforts, program staff also included several process measures to document the implementation of program activities and measures to evaluate satisfaction with the new materials.

Based on the discussion above, the following evaluation questions were developed:

• Were patient and provider educational materials developed and distributed as planned?
• Was the community-based education for providers developed and implemented as planned?
• How satisfied were patients and providers with the materials and community-based education?
• Did patient-provider communication improve as a result of the intervention?

Step 4: Gather Credible Evidence.

The coalition decided to use both qualitative and quantitative data to best understand the implementation and effects of the program. The table below summarizes their data collection plan (Table 2). To answer the first two evaluation questions, which focused on the implementation of the program, staff collected and reviewed program logs, registration forms, and sign-in sheets.

Program staff wanted to hear from a large number of patients to assess satisfaction levels with the new educational materials, so they designed a short questionnaire to be completed by patients after their visits. They also interviewed a small sample of patients following their visits to get more in-depth and qualitative information. Program staff developed a similar questionnaire for providers to assess their satisfaction with provider materials and lectures. Staff also interviewed a small sample of providers to add qualitative data about how providers used the materials.

Improving patient-provider communication was one of the project’s objectives. At first, program staff had hoped to use a pretest-posttest design to survey patients before and after introducing the educational materials. As the intervention planning progressed, staff realized they did not have time to develop the questionnaire and survey patients before the planned launch of the intervention. More importantly, they had no way of predicting when clinic patients would return to the clinic, so they would not be able to easily collect pre-intervention and post-intervention questionnaires from the same patients to then compare results and measure improvements in communication. Instead, program staff used a posttest-only design and included questions on the patient questionnaires and interviews to assess their understanding of their providers’ explanations of diagnosis and treatment.

Although program staff were not able to use a pretest-posttest design with patients, they were able to do so with providers. Staff talked about conducting observations of patient-provider interactions to measure provider communication skills, but they did not have the funding or staffing to do this. Instead, providers were surveyed and interviewed both before and after the introduction of the new materials to assess their ability to explain antibiotic use to their patients.

Table 2: Evaluation Questions, Indicators and Data Sources

Evaluation Question:	Indicators:	Data Sources:
Were patient and provider educational materials developed and distributed as planned?	Number of materials developed. Number of materials distributed.	Program logs
Was the community-based education for providers developed and implemented as planned?	Number of educational events held. Number of providers attending events by medical specialty.	Registration forms. Sign-in sheets.
How satisfied were patients and providers with the materials and community-based education?	Percentage of patients who report satisfaction with materials. Percentage of providers who report satisfaction with materials. Numbers of materials distributed by providers to patients. Percentage of providers who report satisfaction with community-based education.	Patient questionnaires. Patient interviews. Provider questionnaires. Provider interviews.
Did patient-provider communication improve as a result of the intervention?	Percentage of patients who state that they understand providers' explanation of diagnosis and treatment. Percentage of providers who state they are able to explain antibiotic use to patients.	Patient questionnaires. Patient interviews. Provider questionnaires. Provider interviews.

Step 5: Justify the Conclusions.

Research staff at the health department analyzed data from the patient and provider questionnaires and interviews. Providers reported high levels of satisfaction with the materials but much lower levels of satisfaction with the community-based education. Providers also reported high levels of use of the new materials. Qualitative data from the provider interviews showed that providers were very satisfied with the materials and felt better able to talk about antibiotic use with their patients after the intervention than they did before. Many of the providers said that the new materials were good “tools” for patient education and helped them focus their discussions with patients around antibiotic use.

Patients reported much lower levels of satisfaction with the materials than did the providers. Qualitative data from the interviews helped staff understand this dissatisfaction. Many patients felt the materials were overly complicated and did not provide clear explanations of what to do to relieve their symptoms when antibiotics were not necessary. Patients, like providers, positively rated patient-provider communication, with the majority of patients saying they had received clear explanations of their diagnosis and treatment from their providers. A significant proportion of patients said that although they understood their providers, they left their visits with some unanswered questions.

Program staff were not entirely sure how to interpret these results, and stakeholders met to discuss the relative importance of the various findings. Stakeholders were pleased with the findings from the questionnaires and interviews showing that both patients and providers reported fairly high levels of communication and that providers reported improved communication following the introduction of the new materials. They were also pleased that providers reported high levels of satisfaction with program materials. However, they were surprised to find that patients did not understand the materials, especially since most said that they understood their providers’ explanations. While the intervention appeared successful as judged by the outcome of patient-provider communication, the group wanted to address the apparent lack of patient satisfaction with the materials. Stakeholders also felt that the project could be improved since patients reported leaving visits with unanswered questions.

Step 6: Ensure Use and Share Lessons.

Based on the discussion among stakeholders regarding the interpretation of data and conclusions about program success, the following recommendations were developed:

Coalition members recommended that patient materials be revised in the future with significant input from patients in order to improve comprehension and satisfaction. Coalition members decided to discontinue the community-based education due to the relative lack of satisfaction.

Some of the coalition members raised the idea that nurses, medical assistants, and other clinic staff could be an untapped resource for health education. Patients had complained of not having enough time with providers, and providers thought that these other clinic staff could help provide the additional education that patients wanted. Health department staff decided to distribute the provider materials to other clinic staff, hold a training to introduce them to the project and gain their support, and look for opportunities for these staff to take on roles in educating patients on appropriate antibiotic use.

Given the high levels of satisfaction with provider materials and high ratings for communication between patients and providers, the health department designated funds to continue the project with the changes noted above. Stakeholders decided to collect more data on the intended outcomes of the project (i.e., knowledge and awareness of appropriate antibiotic use messages and antibiotic prescribing) during the second year of implementation.

Case Study: Media Campaign

Background

Researchers and administrators at a large health plan monitored antibiotic prescribing for several years and found increases in the number of antibiotics prescribed, as well as dramatic increases in health plan expenditures for antibiotics. They knew from national studies that many antibiotics are prescribed unnecessarily for upper respiratory infections and wanted to reduce inappropriate prescribing to both improve patient care and cut health plan costs. The researchers and administrators brought in physicians and other providers at the health plan to discuss the problem of antibiotic overuse and to develop possible approaches. The providers all cited patients’ lack of knowledge about proper antibiotic use and high expectations for antibiotics as the main factors contributing to over-prescribing. As a result, the health plan decided to develop a media campaign to try to change public knowledge and awareness about antibiotic use. Since their data showed particularly high antibiotic utilization among young children, they decided to focus on parents of this population.

A coalition composed of health plan staff, medical professional groups, healthcare providers, public health department staff, and university researchers was formed to develop the media campaign. The coalition hired a project coordinator with funds from the Centers for Disease Control and Prevention (CDC) and began planning the media campaign. The group decided to use the radio public service announcement (PSA) and the print poster developed for CDC’s national campaign, Get Smart: Know When Antibiotics Work, since the timing of their campaign coincided with CDC’s national media launch. During the first year, the coalition solicited and received funds from several of its member organizations and used these funds to print posters and distribute them in community settings (i.e., community pediatric clinics, libraries), and to promote placement of the radio PSAs. The campaign was launched in two communities with high levels of membership in the participating health plan. Staff collected data to document the implementation of the campaign (e.g., number of ads placed) but did not measure any outcomes of the campaign. At the end of the campaign’s first year, the coalition decided to implement the media campaign in two new communities and to develop and implement a more comprehensive evaluation plan.

Step 1: Engage Stakeholders.

The project coordinator convened a series of meetings to begin planning the evaluation of the expanded media campaign. A diverse group of stakeholders was identified, including coalition members (described above) and representatives from the local radio stations that had run the PSAs during the first year of the campaign. The coalition also wanted to include members of the target audience, so they made presentations about the campaign at school and community functions in the new target communities and recruited parents of young children to participate in the planning of the evaluation.

Stakeholders had very different views of what should be measured by this second evaluation. Health educators shared their experiences evaluating other media campaigns and recommended an assessment of exposure to campaign messages. Radio station representatives were also interested in exposure and offered to collect measures of the reach and frequency of message exposure. The coalition members who had contributed funds towards the media campaign felt it was more important to show some “results” this year, stating that they needed to report back to their organizations and show that their money had been put to good use in order to keep their organizations engaged. These organizations were willing to accept a process evaluation in the first year, but they felt increased pressure to show results during the second year.

The project coordinator and other coalition members wanted to use the evaluation to document adherence to their implementation plans and to find out if consumers were motivated by the campaign to seek additional information about appropriate antibiotic use. The coalition had developed a website at the end of the first year, and the group now decided to update and improve the website and to include their Web address on all media pieces for the second year of the campaign. Finally, the parents involved contributed yet another perspective to the evaluation. They said they often felt overwhelmed by the quantity of ads and messages in the media, and they wanted to know if parents even noticed or paid attention to the campaign PSAs and posters.

Step 2: Describe the Program.

As a result of stakeholder discussions and the need to show “results,” the coalition realized they needed to make the goals and objectives of the media campaign more explicit and discuss what would constitute success for the project. The coalition had never formally stated goals or objectives for the project, but they now saw the evaluation of the second year of the campaign as a great opportunity to engage the coalition in this process.

The coalition agreed that the long-term goal of the program was to decrease the spread of antibiotic resistance. The project coordinator then helped the coalition draft a logic model to show the relationship between program activities (the media campaign) and desired outcomes leading up to the program’s long-term goal. Stakeholders identified intermediate milestones between the implementation of the project activities and this long-term goal, including:

• increased consumer knowledge and awareness of appropriate antibiotic use messages,
• increased number of consumers seeking information (i.e., from the program website),
• decreased patient demand for antibiotics, and reduced inappropriate antibiotic use.

The following logic model visually depicts the activities and intended outcomes of the project and the hypothesized relationships among them (Exhibit 1).

Exhibit 1: Logic Model: Media Campaign

Click to enlarge

Drafting the logic model helped the group gain clarity regarding its goals and objectives. In the end, three of the outcomes from the logic model were adopted as the program objectives, as summarized in the following table (Table 1).

Table 1: Objectives and Activities

Objectives	Activities
Increase knowledge and awareness of appropriate antibiotic use messages. Increase number of consumers seeking information on antibiotic use. Decrease patient demand for antibiotics.	Distribute posters and radio PSAs. Improve and update program website.

Step 3: Focus the Evaluation Design.

As noted, during the stakeholder engagement in Step 1, the coalition found that different stakeholders had differing views as to how to focus the evaluation of this program. The project coordinator and some of the other health department staff wanted to focus on implementation of the campaign and exposure to media messages, while some of the coalition members (particularly those who had contributed funding) wanted to measure results or outcomes. Parents were most interested in whether or not consumers saw the posters or heard the radio PSAs.

As part of the process evaluation conducted during year one of the campaign, coalition members had documented the implementation of the campaign and collected data on the number of media pieces placed, the timing and location of these placements, and the estimated number of audience impressions (or viewings) for each type of media used. Stakeholders decided to continue to collect this process data to make sure that the media campaign activities were implemented as planned; however, they decided it was important to add outcome measures this year since the evaluation would be used to justify continued funding from their partners.

The coalition met to select primary outcomes of interest. Some of the physicians from participating health plans wanted to measure antibiotic use and suggested using the number of prescriptions as an indicator. Marketing experts on the coalition convinced the group that, based on much research and evaluation, it was unrealistic to expect significant behavior change as a result of a media campaign alone. Instead they suggested looking at shorter-term outcomes such as changes in knowledge and awareness that are likely to precede, yet eventually contribute to, the desired behavior change. Those representing the funding organizations lobbied hard to look beyond knowledge and awareness to some actual “results,” believing that they should see behavior change at this point in the campaign if it were working. In the end, the group chose to focus on both short-term and intermediate outcomes, including knowledge and awareness, information-seeking, and measures of patient demand for antibiotics. They believed that all of these factors could be influenced by a media campaign and that changes in patient demand would be seen as tangible results by the coalition members.

Based on the discussion above, the following evaluation questions were developed:

• Was the media campaign implemented as planned?
• Did consumer knowledge and awareness of appropriate antibiotic use messages increase?
• Did consumers who were exposed to the media campaign seek information about antibiotic use?
• Did patient demand for antibiotics decrease?

Step 4: Gather Credible Evidence.

The first of the group’s evaluation questions is a process question and looks at the implementation of program activities. To determine whether the campaign was implemented as planned, the program coordinator replicated the process evaluation employed in the first year. The coordinator collected samples of all the media materials in use and reviewed program logs documenting poster distribution. Coalition members from local radio stations facilitated the media tracking and provided figures for the numbers of PSAs aired and the estimated number of people who heard the ads in both of the new target communities.

The remaining evaluation questions look at outcomes, changes in things other than the program and its staff. The coalition decided that a survey was the best way to measure changes in knowledge and awareness of appropriate antibiotic use. They did not have funds to implement a survey of their own, but fortunately, one of the participating health plans agreed to add a few questions to an existing state-wide consumer telephone survey that was being used to assess community need for new pediatric clinics. This survey was being conducted with the same target population as the media campaign – parents of young children. A few questions were drafted on knowledge and awareness of appropriate antibiotic use messages, and these questions were added to the survey. Some parents of young children who had participated in early coalition meetings as stakeholders reviewed the questions to ensure comprehension.

Next, coalition members proposed using the number of hits to the program website as a measure of consumers seeking additional information about antibiotic use. Finally, the group discussed a variety of ways to measure patient demand for antibiotics. Coalition members wanted to objectively measure patient demand, but they did not have the resources to observe or record patient-provider encounters. They also considered surveying providers to find out whether or not their patients asked them for antibiotics, but they did not have funds to develop and implement this type of survey. Instead they decided to include a question on the consumer telephone survey asking parents if they had requested antibiotics for their children, and they used this self-reported data as the measure of patient demand.

Because of the prominence of the issue of antibiotic resistance and the ongoing CDC national media campaign, the coalition knew it would be difficult to isolate the unique contributions of their efforts to change consumer knowledge and demand. They considered adding a simple question to the survey to determine consumers’ source of information on the topic but decided they would need a stronger case to keep the funding organizations engaged. Fortunately, the health plan survey was being administered in multiple communities, and the project staff was able to choose two communities with similar demographics to serve as control communities. The health plan survey was conducted in the two new campaign communities and two control communities both before and after the introduction of media messages into the new campaign communities. Within each community, phone calls were made to random households with young children. Half of the surveys were conducted before the media campaign began, and the other half were conducted afterwards with different respondents. Because of this design, changes in knowledge, awareness, and demand were measured at the population level, rather than the individual level. The table below summarizes their data collection plan (Table 2).

Table 2: Evaluation Questions, Indicators and Data Sources

Evaluation Question:	Indicators:	Data Sources:
Was the media campaign implemented as planned?	Number of posters and radio PSAs placed. Estimated number of people who see or hear ads.	Media materials, program logs, media tracking
Did consumer knowledge and awareness of appropriate antibiotic use messages increase?	Percentage of consumers who report seeing posters and hearing radio PSAs. Percentage of consumers who believe antibiotics are not useful for colds and flu. Percentage of consumers who are aware of the threat of antibiotic resistance.	Consumer telephone survey
Did consumers who were exposed to the media campaign seek information about antibiotic use?	Number of program website hits.	Website tracking
Did demand for antibiotics decrease?	Percentage of consumers who reported that their child had cold or flu symptoms and who also reported asking their provider for an antibiotic.	Consumer telephone survey

Step 5: Justify the Conclusions.

The data were collected and analyzed according to plans. The data from the process evaluation showed that all media materials were developed as planned but that ad placement varied dramatically between communities. One of the campaign communities documented two to three times the number of posters and ads placed and number of audience impressions as compared to the other campaign community.

Stakeholders met with the staff responsible for ad placement to better understand the differences in the number of ads placed and posters distributed in the various communities. Staff in both campaign communities had followed a protocol for contacting radio stations and seeking placement of the radio PSAs. The two campaign communities had received the same amount of funds for poster placement from the coalition and had distributed roughly the same number of posters with this funding. However, in one of the campaign communities, project staff had developed partnerships that resulted in increased public exposure to the campaign. Staff in this community had met with local hospitals and clinics and had succeeded in placing the radio PSAs on telephone recordings for callers on hold. In addition, the hospitals had provided in-kind donations of printing services and had distributed additional posters throughout their provider networks. Furthermore, these staff had worked closely with parents at community schools and had played the radio PSA and distributed campaign posters at school meetings within this community.

Program staff analyzed results from the consumer telephone survey and found that knowledge and awareness of appropriate antibiotic use messages increased in the community with high levels of message exposure as compared with the low-exposure community and the control communities. However, self-reported demand for antibiotics did not change significantly in any of the communities.

Website tracking showed a significant increase in the number of hits to the program website following the introduction of media messages, but the tracking software used did not allow the breakdown of totals by community. Stakeholders assumed the website hits came from the community with greater exposure to the campaign, but they were not able to document this. The group concluded that while providing a website as a resource for people seeking more information was an important component of the media campaign, the number of people seeking further information was not a useful evaluation indicator in this case given the limitations of their software.

Stakeholders were not surprised that demand for antibiotics did not change in the target community, which showed little or no improvement in knowledge and awareness of appropriate antibiotic use. With the exception of the marketing experts, stakeholders did, however, expect to see changes in demand as a result of increases in knowledge and awareness and were surprised to see no change in demand within the campaign community that documented increased knowledge and awareness. Coalition members concluded that the media campaign alone was not enough to lead to decreases in patient demand. Marketing experts on the coalition spoke of the need for a “supportive environment” to reinforce and supplement the knowledge and attitude changes in order for this behavior change to occur. The group hypothesized that in this case, healthcare providers did not provide the supportive environment needed to result in decreased patient demand. Public knowledge and awareness had increased in one of the campaign communities, while provider knowledge and skills in this same community remained relatively unchanged. Even though healthcare providers had been exposed to campaign messages as part of the general media campaign, the campaign did not include specific activities or tools for providers. Stakeholders concluded that to achieve behavior change (i.e., decreased patient demand or decreased inappropriate antibiotic use), they would need to target healthcare providers in order to improve providers’ knowledge and communication skills.

Following this stakeholder discussion, the project coordinator revised the logic model to include another pathway depicting provider education activities. The project coordinator proposed developing prescribing guidelines and patient education materials to distribute to healthcare providers. This activity was expected to contribute to increased provider knowledge and awareness of appropriate antibiotic use and to increased skills to communicate about appropriate antibiotic use with their patients. The following logic model shows the relationship between provider education and the hypothesized outcomes of these and other program activities (Exhibit 2).

Exhibit 2: Revised Logic Model: Media Campaign

Click to enlarge

Step 6: Ensure Use and Share Lessons.

Based on the discussion among stakeholders regarding the interpretation of data and conclusions about program success, the following recommendations were developed:

Stakeholders recommended continuing the media campaign and expanding the program to include a provider education component consisting of (at a minimum) the distribution of prescribing guidelines and patient education materials as tools to help improve patient-provider communication. Health plan partners suggested additional vehicles for provider communication and education within the health plan system, including internal newsletters and continuing medical education events. The stakeholder group also stressed the importance of including an evaluation component for any new provider education activities.

Because additional PSA placement and poster distribution had resulted in greater exposure and increased knowledge and awareness of campaign messages in two of the target communities, stakeholders were reminded of the importance of partnerships and recommended pursuing partnerships to expand the reach of future campaign efforts. Finally, stakeholders recommended that staff responsible for media placement in the various communities maintain frequent contact to share successful strategies and techniques.