A Guide to Developing a TB Program Evaluation Plan - Webinar Transcript

This is an archived document. The links and content are no longer being updated.

November 14, 2005
2:00 p.m. CST

(Extremely poor audio made transcription very difficult)

Coordinator ...Maureen Wilce, ma’am you may begin.

(Audio cut-outs)

Maureen

Hello thanks, everybody for waiting with us this afternoon. I apologize for the little bit of delay here in getting started. I’d like to thank everyone’s interest in joining us with our first Webinar on the evaluation and how to conduct evaluations. As you know, the Evaluation Working Group has been working on strategies for evaluation capacity building for a couple of years now. One of the things that we as a group decided, that we need to offer some trainings to help people understand how to better conduct evaluations.

The Introduction to Program Evaluation Manual has been released by ... CDCs within the last month and a half. This seemed like an excellent time to begin our presentations on conducting evaluations. We’ve got – our presentation today will be by Ann Powers from Battell. Ann is a Senior Research Scientist and she works with Battell in Arlington, Virginia. She’s been conducting evaluations for more than ten years for a variety of federal clients, including CDC, USAID and The National Institute of Health, as well as for state, governments and foundations. Her PHD is in social psychology from the University of Missouri. She has expertise in mixed method evaluations and working with multiple sites. So with that, I’d like to let her begin to talk to you about the first part of Program Evaluation. Ann?

(Poor audio, background noise)

Ann

Okay thank you, Maureen. Thanks, everybody for coming and I’d like to apologize – technical things going on here. So thanks very much for joining us. What I’m kicking off today is the first of a couple of sessions that we’re going to be doing. Each one of the sessions will probably last about generally two hours. What we’re trying to do is to give you an overview of Program Evaluation, especially coming on the heels of this new document that CDC has released in the last month and a half.

...So ... for people who are new to the field for Program Evaluation. For those of you who are very ..., who’ve heard some of these talks before; I hope you’ll find something new in it. What we’ll do – I’m going to do a presentation on talking about a new adoption of the Program Evaluation and also the first part of the CDC Framework. Then we’ll have time at the end for questions and if there is time still, there’s a small example that I want to share with everybody. We’ll see how the timing goes, especially with coming in a little bit late. Actually now your microphones and the folks on the phones are closed. So I would hope that if you have questions while I’m talking, you’ll write them down and we can address them when the mikes open.

(Poor audio, background noise)

Just to go over a little bit about the learning objectives for the next couple of sessions. What we want to do with the first session and that’s today, is to help you become familiar with the six sets of the Evaluation Framework. There are a lot of other evaluation frameworks and they’re very good ones. We’ve chosen to use the CDC one because it’s flexible, it’s been well used out in the field and there’s also – it can be used with any type of service program. There’s also – it’s very straight-forward and easy to learn and a lot of resources to help you in terms of trying to do your own evaluations.

Including – and ... to CDC in general, I just wanted to mention that the TB Group has an Evaluation Working Group that’s been doing a number of products. Certainly these sessions are part of that and a number of things that have come out and will be coming out, that I’m going to assist you. So all of them are built around this framework and that’s why I think useful for this ... particular application.

So today what we’re going to do is talk about the framework, the steps of the framework and also talk more in-depth about ... which you’ve been getting in stakeholders. Then there’s some sessions coming up. The next session is Thursday at 11:00 I believe and Tom....

Maureen 1:00.

(Overlapping voices, poor audio, background noise)

Ann

1:00 I’m sorry. Tom Tackle will be doing that and that’s talking about step two and three of this training work and in more detail and how we develop a lot of this model and how to use a ... model and sort of very important. They’re definitely important to public health programs. The last two sessions that we had on this ... this was a more a tentative plan originally that we would do two more session, talking about steps four and five in the third session and then step six in the fourth session.

At the moment we’re thinking those are tentative – Maureen is hearing that maybe there’s some interest in some different – in some other topics or in additional topics. So we’re going to call those tentative right now and if you have ideas or suggestions about some other things that you’d like to hear or like to learn about, just let Maureen know. You can e-mail her or call her and let her know about some ideas that you have about for addition sessions.

So why don’t we start with what we’re calling session one here. Again these are your learning objectives. To become familiar with the six steps of the CDC Evaluation Framework and then also to learn how to identify and engage stakeholders, which is step one. What I want to talk about today and help you learn is the basics of what Program Evaluation is and what is isn’t. We’ll talk about that first. So we’ll talk about the differenced between evaluation and research and the differences between evaluation surveillance and evaluation and monitoring.

(Audio cut outs, background noise)

I want to talk about why evaluation is important and what can be evaluated. Also what it means to be – the utilization focus evaluations, which are really one of the core pieces of the CDC Framework. I want to talk about the steps ... and conducting evaluations as outlined in the framework and also the standards for conducting program evaluations. How you can use the standards that are part of the framework to help you design and conduct your evaluation and then how to identify and engage stakeholders as I said.

(Very poor audio)

So let’s start first of all with what is evaluation? In the three perspectives here, two well-known people and third very well-known person. The unknown person who writes a lot I gather. Michael ..., to him evaluation is the systematic investigation of the merit, worth or significance of an object. Like Carol said, it’s the systematic assessment of the operation in our outcomes of a program, a policy compared to a set of explicit or inclusive standards as a means of contributing to the improvement of the program or policy. Then the third, unknown, so ... evaluate ... systematic way to determine the value of a program, program components or activities.

In all three of these definitions you do see the word systematic and what that really means that evaluation is planned, it’s disciplined in its objectives. One of the activities that many of you have been doing this fall is writing an evaluation plan and that’s part of systematizing evaluations. Another word that you see is the word value, which is not a word that you see with research commonly but because evaluation involves judgments and there are values involved. Something becomes – you know you make a judgment of value.

Evaluation is a selectable discipline and it does mean different things to different people. The definition that I think is a very good one and that is part of the new CDC introduction to Program Evaluation for Public Health is the systematic collection of information about the activity, characteristics and outcomes of programs who make judgments about the program in truth, program effectiveness and/or informed decisions about program development. So you can see right there, it’s a very broad concept and useful in many ways.

(Audio cut-outs, background noise)

So let’s talk a little bit about what evaluation isn’t before we talk about what it is. ...This ... contrasts evaluation and research. They are familiar in some ways in that they both employ methods that are systematic in their investigations but there are a lot of fundamental differences between the two. Research is generally conducted to produce knowledge that is generalized across different groups or different programs or different settings. Valuations are conducted to generate findings that are intended for the use by specific programs at a particular time.... ....is being conducted. With research, the questions and the hypotheses are typically researcher-driven and evaluations on the other hand, the questions for evaluations are usually derived from the program or more importantly from the stakeholders.

(Poor audio, background noise)

Evaluation, as I mentioned, is value-based, so it involves judgments about quality. Research is something that we try not to do. It’s usually conducted without judgment or bias to the best of your abilities in any rate. So therefore for that reason, research is normally done in a really controlled setting as much as possible to prevent the ... from coming into the research. Evaluations, because they’re intended to be used for program improvements, evaluation is conducted in a real world setting and it has to take into account the environment and the context in what you’ve based the....

Roles are much clearer with research. The researcher is – sets out to be objective and not part of the situation but ... sometimes experiences role conflicts that could make him be part of the program that they’re evaluating. Sometimes we also – there’s a difference in publication because research has a – seeks to generalize knowledge, you often will publish those findings. With evaluations, we’re finding they’re less often published. There not never published but they are less often published.

(Loud background noise)

Okay this is a really quote I think what really sums up evaluation. Research speaks to prove, evaluation speaks to improve. I know they use similar methods; similar data announced at sometimes is the real underlying point of doing these two activities is very different. Evaluation is going to involve looking closely at the operation of a program or a program initiative, which is often the case at TB. And gaining some understanding through this examination and then making recommendations to improve the program.

(Audio cut-outs)

I think it’s useful to really keep in mind the word improves when we’re talking about evaluation because that is our – what we seek to do in the ... and it’s not to criticize or to blame or even to eliminate some part of the program, it’s solely for improvement. So if the goal of evaluation is to improve a program then no evaluation is good unless the findings are used to make a difference. This is a very important point and I’ll talk a little bit more about it as we move through the framework and you’ll hear it through all of the sessions.

Sometimes after an evaluation is done, the reports in the papers that are produced from it are – a report is written and then it goes and sits on a shelf and they aren’t used. So you cant’ use the findings to improve the delivery of a program or to improve outcomes and that’s really something that we’re thriving to avoid in this particular case. So you want to begin evaluation with the intention that you’re going to use the results and as part of your planning to plan and strategize ways that you can use those results. You can involve your stakeholders and you can really use them to improve your programs and you’ll see that in the framework, the last two steps or the last steps about dissemination and utilization.

So let’s talk about Program Evaluation in comparison to surveillance in.... These are frequently used terms and they’re quite often used activities and surveillance typically refers to the tracking of a disease or a risk behavior. The TIMS System is a surveillance system but evaluation is different in surveillance in that – evaluation is very specific. Program Evaluation is specific to one program during a certain time period, where surveillance tends to be a more global activity.

Monitoring is also used for tracking changes in programs and it is simply a maze of tracking and recording and there is no evaluation done in it. There’s no assessment about the quality of anything in monitoring. It’s an information gathering system. What’s important though, is that I think to mention about both of them is that they can both be used as part of Program Evaluation. I mean monitoring and surveillance data are important pieces. They can be important pieces of your evaluation so we don’t want to not use them but we want to use them to help us understand other things.

(Audio cut-outs)

So this ... directives what can be evaluated? That’s most everything I think. It’s all types of programs and all types of program initiatives, which are some of the things that we’re doing in TB. The range of activities includes services and interventions to pieces of organizational structure, training and educational systems, administrative systems, laboratory systems. It’s very broad and evaluation can be adapted to all different kinds of settings.

(Poor audio, background noise)

It is I think important – an important message for the TB program because many of these things are things that the programs are undertaking, particularly initiatives that may be new or may be in the planning stage right now. That’s likely the kind of thing that’s going to be investigated and going to be evaluated as part of a TB program, rather than a whole – the entire programming.... For example be interested in an outbreak investigation rather than the entire program. This is just an example of some of the range of things that you could evaluate if you chose to.

(Audio cut-outs, background noise)

Now this is sort of moving onto when to conduct evaluations and how evaluation changes with this phase of the program that you happen to be in – involved in. Now traditionally some of the evaluations have been conducted toward the end of programs after the programs are over with very mature programs. It’s not necessarily at that point in order – you’re not able to make any changes to make improvements to a program, but to be most effective in the way that we want to see evaluation and.... To do it at all phases for the life of the program to result in improvement.

In the conception phase, which is when you’re planning a new program, evaluation can be used to assess motivation for starting the program, to define the need for the program and to inform decision makers about what a new program should look like and also to define its objectives and expectations for the program. Also to help you clarify some important factors that are really essential in terms of building the program and lying down the foundation for it.

When you’re developing the program, the next phase on, you can use evaluation to monitor your progress and also to do any course corrections you might need or redefine program activities or expectations. Again, find any gaps that you might have. If you move into a stable or a mature program, again you can use evaluation findings to monitor the program and you can also begin at this point to use it to provide feedback to management.

You can also use – begin to think about showing program successes at this point as well as the areas for improvement. As the program becomes more stable, the initiative becomes more stable and mature, you can begin to assess outcomes at this point. Then finally at the end of the program, evaluation will help you assess the value of the program, as well as document any lessons learned, that you might want to use in the future.

(Background noise, audio cut-outs)

You can make an assessment of how the – if you’re doing an intervention, how that was adapted into the particular situation and it’s a change that’s sustainable at that point. You can look for unexpected outcomes after a program has.... It’s also a time to assess and evaluate whether the intervention could be replicated beyond this point and what factors would – are necessary to be able to do that. With some of the – when we’re talking the TB program, many of them I know are stable or mature but you also may have an initiative or a new program or a new effort that’s in the planning stage or in the developing stage.

So this is kind of not – any one program could be in various stages and it’s not an all or nothing kind of thing. So you may have a very mature program that’s starting a new initiative and you can kind of mix and match those things and put those together. But the point of the slide is that the stage of the program development is going to influence the reason for conducting Program Evaluation.

So why evaluate programs? The first reason is to gain insight and this is to evaluate a program to judge its merit or gain insight about the program and its operations. This can include providing information concerning the practicality of the new approach or developing program and to find out where the program is and where it’s heading. Evaluation will also help you assess where we are in the program development and how to identify information that you need to, to move onto the next step of the program development. It also will help you with decision making and also in terms of managing resources and services more affectively.

So for example, an evaluation of testing and treatment for LTDI might show how many people in high-risk populations and how many people in low-risk populations are being tested. So you can use that to and form decisions about whether it is useful for example to review the testing in a low-risk population.

(Very poor audio)

The second reason is to improve practice. We can evaluate the program for the purpose of making improvement or changes in practices. It can improve or enhance what we choose to do and this is an important thing to do when you’re in the middle of the implementation of a program, to tell you how the – if there are changes that need to be made or focus that needs to be changed. It can also help you understand why you achieve these ... or perhaps not the success you did if you didn’t meet your objectives and it’ll information on stuff that you can strategize and plan and implement changes so that you can improve the effectiveness of your program.

To assess the effects and this is – when you’re conducting an evaluation, to see how well objectives and goals were met or are being met. It can show how effective an intervention is at inducing the intended changes over time that it was set up to do. It helps to demonstrate the value of efforts. It’s very systematic in documenting the contribution towards your goals. It’s useful in terms of effecting – the fact is that it can count decision makers at all levels, understand the benefits and the consequences of the program and at very important times, at critical times, evaluation can help you advocate for your program.

(Bad audio, background noise)

The last reason to evaluate is to build capacity, to increase funding in ... scales and strengthen accountability. ...To help you in terms of ... capacity about making self-directed changes, such as developing skills or building infrastructure. So this is the CDC Evaluation Framework. Many of you have probably have seen this diagram or some version of it.

What I want to start moving into now is to really talk about the pieces of – to giving you an overview of the pieces of the evaluation and the framework. ...Is set up to be a practical tool and it is generally non-prescriptive and it’s shown as – it’s in a circle and this really represents the circular nature of the evaluation process. So it shows how interdependent the steps are. Also, although it cannot entirely illustrate it, that there really can be conducted in a non-linear order and then you may be conducting the steps more than one step at a time. So it has this kind of circle to it as a way of demonstrating that it is not necessarily, although we call them steps – step one, step, two, step three, step four.

Although the steps can be conducted in a different order or they can be done simultaneously, there is a reason why the steps are sequenced in the way that they are. That’s primarily because the earlier steps provide a foundation for the subsequent steps. So engaging stakeholders is an important piece to help you describe your programs. Once you develop a program description, then you can start to think about focusing and evaluation design.

(Poor audio, background noise)

All of the steps are important in – or however they’re being done or how they ... are doing at one time. Each one has an important piece to add to the whole circle. The circle also represents that the framework is iterative and that program evaluation is iterative. The information or the findings from one evaluation or from an evaluation, is used to be fed back into the program to make improvements. So this ... ensures lessons learned to step six begins the cycle again building the new cycle again.

At the core of the framework are the evaluation standards. This is usability – I’m sorry utility, usability, propriety and accuracy and these are four standards that are intended to maximize the quality and effectiveness of the evaluation efforts. Each one of these standards is applicable to each one of the steps in the framework and I’ll talk about those in a second.

So here are the six steps. I’ll go over each one briefly but hopefully in future presentations we’ll cover them in more depth. Step one is the NDH stakeholders and stakeholders are those that have a vested interest in your program and are potentially affected by the programs and its evaluations. It’s important to note that stakeholders can be both supporters and skeptics.

(Bad audio, background noise)

Step two is describing the programs. This is the step in the evaluation that allows us to gain a better understanding of the program and its activities. The meanings, the context and the ... copulation for the program or the.... Then major program components like resources, activities, outputs and outcomes in the relationship with each other are defined in step two.

Step three is a very crucial step. Does it help us determine the focus of the whole evaluation? When you’re planning an evaluation – you know something that you have many worthwhile questions that come in or identify all your stakeholders that you consider for evaluation. It’s easy to want to think about evaluation or evaluating everything and collecting a lot of information but it all bounds on limited resources and times and staff and money. So step three is the point at which you think carefully about what it is you want to evaluate and set some priorities.

(Very loud background noise)

Step four is gathering kind of credible evidence. This is where we’re working into data collection ... and this is where you’ll identify the indicators that you’re going to use. Then once you have indicators and you develop message for the indicators and then analyze the data.

Step five is justifying conclusions and this is analysis and interpretations. The interpretation part of this phase is really critical and this is where you judge a finding against your findings against established program benchmarks. The important step here for example is step five, to involve the stakeholders that you’ve identified and engaged in step one, to help you interpret the findings from your evaluation.

(Audio cut-outs, poor audio)

Then step six is to ... you – ensure use and to share lessons learned. This is as we’ve said all ready, a very important part of your evaluation of planning for and including a way to use the data to approve the program. The purpose of this slide is really to think about and reinforce the ... the circular nature of the framework. Really it illustrates how the steps are interdependent. So one of the basic components of this framework is using the results and that’s part of step six. It’s important to create a market for your evaluation findings and that’s part of step one, engaging the stakeholders.

Then finally part of creating a market and being responsive to that market, is focusing an evaluation on the important questions and answering the questions that your market or your stakeholders would like to know. This continues on to show this iterative nature of the framework and this is really illustrating how we – although we refer to step three as the focus, step one and step two are both important factors in focusing. Step one talks about identifying your stakeholders, finding out who cares about the program and finding out what they really – what they want to know. How they define the program and how they define success for your program.

Step two helps to focus the evaluation by establishing the monitor of the markers that we view public health outcomes that your program is interested in, in changing or creating. Step two – this will be something that Tom will talk about on Thursday so I won’t spend a lot of time on that.

I want to spend a little bit of time in talking about the standards for effective evaluation. These are the things in the middle of the circle. Good evaluations are conducted under the guidance of standards for program evaluation that have been set forth by the joint committee of standards for educational evaluations.

The four standards, as I mentioned, are utility, feasibility, propriety and accuracy. What the standards do is that they serve to guide your decision making process of each step in the framework, so that you can ensure that your evaluations stay focused and balanced all the way through.

(Background noise, poor audio)

On the next few slides I want to talk about the standards and really the over ... purpose – or one of the over ... purposes for using the standard, is to help you design the best evaluation for your program and for your situation. Like research, there is really no wrong and right evaluation. Instead, what you’re trying to do when you’re designing an evaluation is to make choices that are going to ... to your particular situation or your particular program. When you apply the four standards, utility, feasibility, propriety and accuracy, if you apply each one of them during the steps of the evaluations framework, what’s going to result is the most worthwhile evaluation for your situation, which is really your objective here.

What’s on this slide is that under each standard there’s sort of a general overarching question that you might ask yourself when you’re designing the evaluation. To just show you how that when you apply them, you end up with an evaluation that’s useful, feasible, proper and accurate.

So to talk about each one of the standards individually: the first one is utility and utility insures that the evaluation serves the information needs of the intended users of the evaluation and also the stakeholders. As we said, the evaluation does no good unless it’s used to improve a program. So knowing who needs the information and how the information will be used helps you to tailor the evaluation and focus the evaluation to address a need.

So you need to take the time to find out what people need on – and also in terms of what it is that they do and how they contribute to the program, how they’re interpreting their productivity to be improved. The whole point of this is really to ensure that the evaluation results are used. So when we’re thinking about utility, this requires asking questions like, who needs the evaluation results, what do they need, will the evaluation provide relevance and otherwise useful information in a timely manner for the users?

(Poor audio, difficult to understand)

Also the best way to do this is really to talk to your stakeholders about this. This is one of those – it’s a standard that, while you can plan, it’s very important to really have your stakeholders address the feasibility. Feasibility is another standard to really keep in mind when you’re planning an evaluation. This is standard is to ensure that the evaluation conducted is realistic, prudent, diplomatic and frugal because resources and ... in public health programs are also limited.

Although as I mentioned, a lot of this – you know you’d like to everything and there aren’t enough resources to do that. So feasibility helps you to prioritize the need of your evaluation, the need for evaluation. So the kinds of – the questions that you want to ask is, are the activity’s plans realistic, given the time, resources and expertise that we have available to us?

The next standard is propriety and this ensures that those of us involved in the evaluation and either evaluators or members in the evaluation team, behave legally, ethically and with regard to the welfare of those involved or affected by the program and by the evaluation. An evaluation can involve handling personal and sensitive information and this may be very true especially with a TB program or TB initiative because the evaluation may involve the review of medical records and charts and the handling of information that patients and providers may regard as highly sensitive. So these are the – so it requires that we address this standard.

(Poor audio, background noise)

Although with Program Evaluation, generally is exempt from institutional review board review. It is still important to really consider that the data that we’re using for the evaluation is handled well and that we protect the rights of privacy of the people involved. So a couple of the questions that will help ensure that the evaluation is conducted with propriety is, does the valuation or the methods employed in conducting the evaluation protect the rights of individuals and the welfare that are involve? Does the evaluation engage those directly affected by the program as well as those affected by the potential changes in the program? That’s program participants and patients ... the community and again this is speaking back to step one of engaging the stakeholders.

(Audio cut-outs, loud background noise)

Have we included people in the evaluation process and given them a voice to the process into making decisions about their own health and all the....

(Loud background noise)

Finally the last one is accuracy and accuracy ensures that the evaluation conducted reveals and conveys reliable and valued information. It’s important to have accurate information because it’s the foundation of any ... decision ... and if you’re going to use the results to improve a program it’s very important that the results be accurate. So it’s important when you’re planning you evaluation to identify ways or methods – methods that will help us yield the most valid and reliable results. This is a subject I think for part of a future presentation, that there are some ways that we can indeed ensure that.

So I’m going to move into the next part of the presentation, which is engaging stakeholders, which is step one. I’ve mentioned it a couple of times but now we have an overview of the whole framework and how the things fit together. What I want to talk about is how to first of all identify the stakeholders that are essential to the program and to the program evaluation. Then figure out who your key stakeholders are and how to engage them in the evaluation.

I just want to repeat a definition of the stakeholder, so we’re all on the same page. Stakeholders are people and are organizations that are affected in the programs, are interested in the results of the evaluation and/or have a stake in what will be done with the results of the evaluations.

(Loud background noise)

This is a ... that the ... are able to absolutely critical evaluation because other than ... are people, people, people and people. So why do we engage stakeholders? This is just for the staff ... in the framework, so that’s certainly an indication of the importance. There are a lot of reasons why the stakeholders are important to your evaluation and I’m going to give you a list here. First, stakeholders are the people who will implement whatever changes or improvements the evaluation recommends. Engaging them early on in the process will increase their buy-in and thus risk ... increase the chance that the recommendations from the evaluation are used.

Stakeholders are also important in providing a reality check when you’re doing an evaluation and they’ll keep the evaluation grounded throughout the process. Sometimes stakeholders can also generate the need of political pressure or support or even sometimes the resources that you need for evaluation. So they’re very important in that respect. In addition,

(Very loud background noise)

stakeholders know the program, in many cases better than anyone and they can provide important insights as ... at your evaluation. So they’ll increase the credibility of the evaluation findings once you collect them. They can assist you with the dissemination of the findings later on. They can advocate for the program and also they may be your funders. So your stakeholder may be able to provide future funding.

(Loud background noise)

So who are the stakeholders? They are persons generally interested in the program or affected by the program and there are three broad categories that we typically will get stakeholders ... from. Those that are involved in program operations, those that are served or affected by the program and they will tend to do this ... the evaluation findings. Then once we have identified who the stakeholders are, then we want to talk about what they’re – think about what they’re interests or perspective in the evaluation happens to be.

(Very poor audio, loud background noise)

Do they support the program or are they skeptical about the program or antagonistic for the program? One way you can do this, is kind of systematically in – as I said if we have a little time I might be able to talk more about this – is to create a list, for example set it up into a table with the three categories of stakeholders. Those involved, those served or affected by and ... the evaluation and then list-out the persons or the type of persons who are part of each group. You’re going to know at that point while you’re doing this that stakeholders tend to fall in more than one group. These aren’t usually....

Then once you’ve generated the list, think about what their interest in the program is or what their interest in the valuation is. Then also think about how you might go about involving them in the evaluation, at which stage in the evaluation they could part of. This is of course one of those iterated tasks where you can begin with a list but it’s important to share the list with your evaluation team and with some of your stakeholders to make sure that you at least initially capture the universe of stakeholders.

So here are some examples: persons that are involved in the program operations can be staffed, managers and other partners who are directly involved in the day-to-day activities of getting tasks done that are part of program operations. This can include public health nurses and TB clinic clerks, as well as members of your own program staff. Those that are affected by the program include clients or patients, their families and friends as well as providers and community groups. So this can include TB and LTDI patients, managers and coordinators at school, prisons, shelters and other congregate segments. Also it may include representatives of populations that are just proportionately affected by TB.

(Poor audio, difficult to understand)

The third category, the intended users of the evaluation is going to include policy makers, managers, administrators, advocates and funders. So of course that involves CDC, public health managers and administrators and also it may involve advocacy groups too are also to the ... evaluations. Then the final census on this ... is urging you to include both supporters and skeptics. Both have important points of view and having skeptics involved in your evaluation is going to increase the credibility of the findings.

The one caveat that I would say about having skeptics involved is really to be able to consider their point of view and their motivation and why they are skeptical. For example, it may be that your skeptics are interested in redirecting program thoughts. You know, their resources are limited and they’re anxious in redirecting the funds. So it’s important to keep that sort of view point in mind. However, it is very useful to have your skeptics involved in the evaluation because having them involved will ultimately strengthen the evaluation, increase its credibility and result in more improvement to your programs.
So this slide talks about which stakeholders matter most and as you can see by earlier slides, there are many, many potential stakeholders. The universe of stakeholders is very large and what we want to illustrate by this slide is, how do you go about identifying the key stakeholders in your evaluations? That from this universe, how do you get down to a group that gives you a wide enough viewpoint but is a workable number? So the longer you ... is the universe of the evaluation stakeholders. This is everybody that you and your evaluation team have identified under the three categories of stakeholders. Then you use this list to then narrow down the stakeholders.

The smaller boxes below is the illustration of the key stakeholders and this is the stuff that of the universe of the stakeholders. They can include people or persons that will enhance the credibility of the evaluation, such as I was just mentioning; your skeptics; your program staff; the day-to-day people that implement changes that will result from the evaluation; advocates for the recommended changes for the program and they often maybe go to fund, authorize or expand the program. This is just really to think about that these folks are your key stakeholders and the ones that are most important to find a way to engage in the process.

So your stakeholders are important and the next sort of step is to think about how and when to engage stakeholders and this slide offers some suggestions. Although it’s not exhaustive list, usually whoever is leading or coordinating the evaluation has some responsibility for identifying and engaging the stakeholders. Stakeholders as you’ll see from this list, it’s important that in step one you identify them and engage them but that they are engaged throughout the process. It doesn’t at the same time mean that all stakeholders are involved in all stages.

(Background noise)

A very small subset may be involved in the evaluation from planning to use but it may also be that certain stakeholders, key stakeholders step in at certain times of the evaluation and then step out. So it’s a ... selectable group of key stakeholders but it’s important that you remember that key stakeholders should be involved at every level. So some of these suggestions in terms of how to involve your stakeholders, the first two are part of step two of the framework. That’s going to use your stakeholders to help you describe your program activities, the context and priorities of the program. Then also working with your stakeholders to define the problem that you may be confronting in your program or the initiatives that you want to investigate.

The next three involve steps three and four of the evaluation framework. That includes selecting evaluation questions and.... Stakeholders can also serve as data sources. They may be someone who’s part of an interview or in the case of ... it may be some part of the start review. They will define what constitutes the proof of success for your program. Interpreting findings can be parallel to step five of the evaluation framework.

Then step six, which is dissemination use are the last two items. That stakeholders can help you disseminate the information to their sphere of influence and in many cases they can help you implement the results, whether it’s a change in program operation or a change in funding or need for political support.

So I wanted to stop and talk and take some questions if we have some.
(Loud background noise)
Maureen MCI operator? Are you – operator?

Operator Thank you, one moment. I have no questions at this time.

(Loud background noise)

Maureen

I’m going jump in and ask a question, Ann....

Ann

Okay.

(Almost inaudible because of poor audio)

Maureen I’d like you to just quickly address how you would think about the standards in regards to ... step one ... stakeholders. The standards to how....?

(Loud background noise)

Ann ...

Around here and we’ll take all the standards in order. The standards of usability utilities – feasibility for ... accuracy – I would use them in terms of working with my stakeholders, wanting to know what they want to know. I guess I would address them initially, so with the utility standards, I would find out what they think the problems are, how they would like to use the evaluation and where we want to focus our efforts. The second one feasibility, I would talk to stakeholders, especially those who for example might have to – or be implementing changes or how do the day-to-day operations and then find out, could they have implement....? Can they implement the evaluation and really talk to them about if they are able to implement the evaluation and on what kind of changes they would be able to implement?

Propriety certainly is part of talking to stakeholders because you want to be sure that you’re doing something that is responsible and ethical and engaging your stakeholders at this point will help you identify any further problems. Sometimes it’s difficult from certain view points to understand what ethical and responsible problems are and I’ve learned that. I sit on an institutional review board and I often find there are program people and the community reps have an entirely different point of view than I do as a PI. So that’s a very important part of engaging your stakeholders.

(Loud background noise)

Of course accuracy, again you want to talk to them and you want to engage them as part of both talking to them initially and then helping you to think about interpreting your findings. Being able to understand the context of your findings, so that any sort of recommendations you make are really grounded in the most accurate information that you have. Does that answer the question? ...Oh you read the questions.... How can I get a copy of the ... for the presentation? That’s going to phase-up correct, Maureen?

(Loud background noise, audio cut-outs)

Maureen

Right after this presentation, give us about 48 hours or so and then you can go into login.... The logins at the bottom of the e-mail that says how to watch the replay, you should be able to get the slides from there. If there are any problems ... also ... e-mail me or ... and we can definitely get....

(Very poor audio)

Ann

...Assessment and evaluation, I might ask you to.... In my mind, I think evaluation still has ... one step beyond assessment.... It talks about making – well assessment may involve making a judgment and evaluation makes a judgment but in many ways it’s – the purpose is different. I mean evaluation is meant to improve or meant to change or to make recommendations. In my mind, assessment stops one step short of evaluation....

(Very bad audio)

Maureen

I agree with you on that and when we you were going over the reasons, because I think evaluation, you talked about was a ... to gain insight and a lot of times it’s.... ...Of different types and many people use the words that are changeably ... at all, which we ... about our language here at all because – so I think you can think about ... being – looking at the big picture but ... questions, evaluations can answer....

Ann

Yes and I think evaluation – it’s intended to be a tool for improvements.

(Audio cuts-out)

Maureen

And assessment can be as well but they can be just – it’s that they tend to be more focused.

Operator I do have a question from the phone lines, would you like to take that?

W

Yes.

Operator

One moment, Samantha Chatogie you may ask your question.

Samantha

Sure I was wondering how this would relate to the triple A – you know the three As, the cycle to assess, analyze and action. Is there any relationship to the framework that you just described?

(Bad audio, also cuts-out, overlapping voices)

Ann

We’re looking at each other and going – okay that’s I’m sure....

M ....

(Very poor audio, almost inaudible)

Ann

I’m sure that’s another framework that someone put forward and as you can see the ... it fits well with ... framework expansion and as ... couple ... to it but the same general process is there ... would be to describe ... focus and then analyze would be the – of course the step ... and then accurate steps to.... I think you know, the ... framework would probably say make sure that ... your assessment ... that’s even the hard question. If you’re defining assessment in a very, very broad way and you include the stakeholder involvement in ... steps, that’s all the ... framework ... takes out to make sure that we’re – we emphasize. I think many evaluation frameworks ... why do people do that....

Samantha

Okay great thank you.

Operator

Miss Karen Marsanek you may ask your question.

Sue Ann

Oh hello, this is Sue Ann Jakerson with Karen Marsanek in Alaska. We’re wondering what CDC’s time table for training on all the steps. I’ve seen you’ve gone through the first two today and would CDC consider extending the deadline of December 15th until at least a month after all that training is complete?

(Poor audio)

W ...Going to take question?

(Inaudible)

W ....

(Very bad audio and cut-outs)

Maureen

We will have this training today and then on ... this week there’ll be Tom ... training on the description and.... These two trainings together should get you well into your evaluation plans. If you feel that you need more help with the later steps in the plan, before, you even ... those, you can feel free to ... into the plan and say ... their going to seek training and technical assistance on developing our specific methods.... This should get us through to step one, step two and step three. After that it becomes much more technical and we would rather ... go use – how you’re going analyze your data questions like that.

When Tom talks to you too on Thursday, I’m sure that Anne’s all ready been saying – you’re all ready learning part of step six. So I would like to see some of that in the plan that we receive on the.... As far as filling in the message section, which is what we probably won’t be able to get to ... probably ... after people will at least turn their plans. If you feel uncomfortable with that, feel free to call ... or myself and know that we will have some additional training on that coming up. You can say in your plans, we’re developing these particular items and we expect to have that done ... or whatever’s comfortable for you, just put that in your....

(Inaudible)

Dan ....

(Poor audio, audio-cut-outs)

Maureen

Dan notes that it’s very important to speak with your program people ... as well because they can have lots of suggestions for ideas and one of the things that we do want to stress with everyone is to keep the dialogue open and that we want things to be very productive ... for everybody. ...For any ... that says please don’t everybody call up on the 15th of December. We’re here right now and would be happy to talk to you and....

(Really bad audio, cut-outs)

Ann

Just to build on my point, between today and Thursday that should in terms of building your plan according to the Evaluation Framework get you up through identifying your evaluation questions.... These are – you know thinking about your goals, who your stakeholders are, how you’re going to involve them and then focusing your evaluation on really the core conceptual pieces.... ...These two sessions should be able to get you up there and then....

M

You also may reiterate to folks that you also all ready provided a guide and had a development evaluation plan, plus an excellent template that’s available to use in helping in developing the plan as well.

(Bad audio, loud background noise)

W

Did everybody hear that? That was ... comment. ...It’s ... that we do have the guide on ... development and if you know if you walk through the exact process. If anybody does not know whether ... that.... ...Internet Web site of ... and if you have any trouble finding it again, call ... or myself or ... presentation....

(Inaudible)

W ....

(Very poor audio, cut-outs)

W

...Yes back in here you can just replay the entire Webinar and ... if you’d like....

M

Yes and the template’s a great source, a great resource....

(Inaudible)
W ...Captive audience. So thank you....

M You’re welcome. There’s another question ... too right?

W No this is.... ...Other questions?

Operator There’s no other questions from the phone lines.

(Inaudible)

W ....

(Audio cut-outs)

W

Does anybody have any examples that they’d like to share with us or any other comments about ... stakeholders ... questions....?

(Almost inaudible)

W

I mean I have a couple of examples but.... ...Examples about ... engage their stakeholders....

Operator Question directions.

(Bad audio)

Maureen

...Simply walk us through the examples?

Ann

Yes.

(Very bad audio and cut-outs, background noise)

Maureen

Then we’ll close on that and then we’ll close on that. I do want to say, if anybody has any suggestions or ideas for additional presentations, please e-mail me or.... ...Call ... 446-395320. The second presentation will be Thursday at 1:00 and we will send out messages about additional presentations after that right? Yes we now have to let Ann talk about ... examples here.

(Background noise, poor audio)

Ann

I just want to give a little ... example of something that I’ve seen in another – a plan that I thought was very useful. It’s a stakeholder assessment and engagement plan. It’s a table, it’s a three by four table, it’s very simple and it takes the – each separate group of stakeholders. So those involved in program operations, those that are ... or affected by the program and the intended users of the ... and puts them in a real ... table. Then there are columns in this table that talks about their interests or perspectives, their role in the evaluation and try to engage them.

(Background noise, poor audio)

So for example, under the persons who are involved in the program implementation, they’ve listed public health nurses, TB clinic clerks, clinic physicians, agent managers and state managers. Their interest or perspectives is attributing to – this is really very thoughtful, that the staff may or may not be reductive to change. The staff may perceive the evaluation as a personal judgment and the staff may be questioning the validity of the information collected. So is their entry for perspective and you ... there might be a critic amongst that group.

(Very bad audio)

Then offering their role in the evaluation to be something about asking the staff how they do their jobs and what sort of process they do. Interviewing them to understand the context of the program. Discussing their concerns about the evaluations, having them help you collect date it with pulling records and other sorts of the data. Then also suggesting possible modifications ... the evaluations process.

For each one of the different kinds of stakeholders, this table goes through this. Under two, you’ll note that I didn’t talk about how ... engage the stakeholders because what I think is very interesting is to think about at what point you want to engage these certain stakeholders and what point perhaps they could step out of the process. So the second group is those affected by the program. This is patients, contact of cases, persons at facilities where the investigations occur and the general public.

They may have concerns about their confidentiality a bit and burdens of the process because they’re part of the evaluations. Their ... has – so they can fear about TB or about testing. They want also – there’s a certain sense of security that they want to feel secure about ... in the public in general. These are very often the folks where the records and the data that are going to be coming from. Also with providers, these are important people to share your findings with.

(Poor audio, background noise)

Then the third group are the primary users of the evaluations. These are program staff, state contacts, TB program administrators and you’ll note that – especially in the third group, you’ll find that there’s overlap with the first two groups. So you’ll have stakeholders in various places. They may want to streamline the processes. They want to identify opportunities for transferable lessons. Their role in the evaluation may be to define specific evaluation questions to reflect the decisions that need to be made. Interpret findings and planning on for the use of the evaluation climate.

So what can also – when we’re thinking about these stakeholders, we’ve identified the stakeholder category, we’ve talked about their interest or perspective and again to sort of reiterate what’s on the slide, we thought about their interest and perspective in terms of things like credibility, implementation, advocacy and.... So you want to think about your stakeholders in terms of how can their perspective increase the credibility of the evaluation. How they might be part of the evaluation and how they might advocate for you for the program for the evaluation and whether they’re funders and they’ll advocate for funding or make funding recommendations for example.

(Poor audio, cut-outs)

The last one is list ways to keep them engaged ... and I apologize that you can’t see this – the thing that I’m looking at, because I think it’s a very concise way of presenting your stakeholders, thinking about who they are and how to involve them. So what’s interesting about the way this is ... thinking about this, is that ... those that are involved in program operations, they’re probably going to be part of your evaluation nearly all the way through. This is not always true but there are certainly people you want to talk to in the beginning when you’re planning and focusing. They’re probably going to be collecting data with you. They certainly will be involved in using the recommendations from you that....

So group one tends to be involved all the way through. Group two, while very important and very key stakeholders, their role you might be finding more limited, because these are the folks that you might be collecting data from. These are the people who might be, look to be and audience for your evaluation results. They may be only involved in steps, for example for in six or four, five and six. So it’s a slightly different perspective.

In the third group, the ... evaluation findings, although they might be involved all the way through, users are – you know you would want to talk to them very early on in the planning process. Have a meeting with them and interview them and talk with them and help them ... the planning process.

Then they may be less important during the whole implementation phase – still keep and perhaps not someone you’re checking in with on a weekly basis or monthly basis. Yet they become very important again in the end, when you’re interpreting your data and analyzing – you’re interpreting your data, making recommendations, disseminating the plans and trying to get their recommendations implemented. Again ... using the evaluation findings may come back in. So I described that to you, I hope it’s not completely obtuse but it’s really an illustration of how key stakeholders – they’re important to every step but again they make come in and out of frequent contact as you’re planning implementing any ... results.

(Very poor audio, background noise)

Maureen

I’m just going to jump in – this is Maureen again. Just that, Ann is looking at an example that was written from – an example that was sent in from.... So the ... is talking about – and you can see very clearly in the template itself and if you follow the guidelines that she’s here giving these today, well within talking with us, you should be able to fill ... and follow through that table. If you have any questions or comments, feel free to call us and we can help you think through how to – I think everybody knows how to identify the stakeholders quite well but it ... on how to prioritize them, how to engage them and what sort of processes. I’m going to ask you a quick question on, how do you find....?

(Poor audio)

Ann

Well there’s no substitute for talking to them. I think that identifying and engaging your stakeholder is really required and ... your stakeholders. So I’ve scheduled meetings with them, maybe to talk to them by phone. If there are public meetings that they’re holding that are important to your program and useful to your program, you can go to the public meetings and listen in on what it is that they’re talking about. You can even schedule time sometimes at public meetings and kind of have an open forum discussion about what you might want to use in your evaluation.

Further on in that, you can – they can be part of your data collections team. They also I think are very important members of the – like I say ... add a data interpretations team. When I was conducting an evaluation, an international evaluation, we found that because there was so much variations in the context of what was going on and the challenges that the different programs had in 16 different countries, we relied on stakeholders in every site to help us interpret the data and help us come up with some of our conclusions and justify those conclusions, so that we weren’t putting forth something that was really inaccurate because we didn’t understand the context. ..Answer?

Maureen

Yes.

Ann

Okay, any more questions?

Operator

Thank you and ma’am we have no questions on the phone lines.

Ann

Okay well then I think if we have no more questions, I think we’ll sign off for the day and thank you, everyone very much for your attention, your attendance and your questions. At this time I’m going to remind you that part two is going to be on Thursday at 1:00 p.m. and Tom Tackle will be presenting that and he’ll talk about step two and three of the Evaluation Framework. So good bye.

(Loud background noise)

Coordinator

Thank you, that will complete today’s conference call, you may disconnect at this time.