Why Evaluate?
Evaluation is a critical tool for determining whether a program is working and how it can be improved. More specifically—"Program evaluations are individual, systematic studies that use objective measurement and analysis to answer specific questions about how well a program is working." (U.S. General Accounting Office, #GAO/GGD-00-204 Program Evaluation)
Evaluation generally answers questions such as:
- How well does the program work?
- Does the program do what we intended it to do?
- Does the program work for the reasons we think it does?
- Is the program cost-effective? Are the benefits worth it?
- What are the unintended consequences of the program?
Reasons for conducting a program evaluation vary. You may undertake an evaluation to meet your own program management needs or to meet the requirements of an NIH research or training program, as stated in the funding announcement. You may also be part of a larger program evaluation initiated by NIGMS to examine the effectiveness of overarching policies and practices. For instance, your program may be part of a large, collaborative effort that will be evaluated as a whole.
Evaluation is simply one of several program management tools considered useful for:
- Gaining insight about a program and its activities,
- improving program activities in order to increase the likelihood of success and
- assessing the impact of a program and determining whether goals are being met.
In addition, evaluation can help you demonstrate results and gain public support for your program. Increasingly, Federal agencies are expected to focus on achieving results and demonstrating specific outcomes. At NIH, program evaluation is one important tool that agency administrators and program staff use to determine the extent to which programs are operating efficiently and achieving their intended goals.
As a steward of taxpayer funds, NIGMS supports evaluation to determine if programs are achieving their intended goals, to build a stronger knowledge base for public policymaking, and ultimately to help determine how to invest in programs.
Evaluation vs. Research
Evaluation and research are not the same thing. Research is scientific inquiry based on intellectual curiosity, and it produces generalizable knowledge that advances a field. In contrast, evaluation judges the worth or merit of a particular program. It focuses on information for decision-making, taking into account specific program goals and stakeholder interests.
Types of Evaluation
Evaluations can be categorized into two broad categories: formative and summative. Formative evaluations are conducted during the development and implementation of a program. They provide useful information about how to achieve program goals or how to improve a program. Summative evaluations occur once programs are well established. They indicate the extent to which a program is achieving its goals.
Within the two evaluation categories, there are more specific types of evaluations: needs assessments (formative), process evaluations (formative) and outcome evaluations (summative).
- Needs assessments consider the needs of stakeholders in order to develop appropriate programs goals. This type of formative evaluation helps to design or modify a program to achieve its goals and is often a tool for strategic planning and priority-setting.
Typical questions asked include: What is the nature and the extent of the issues the program should address? Whom does this program serve? To what extent are their needs being met? What are the program’s documented goals?
- Process evaluations look at program operations to determine if they are being conducted as planned, whether output is being produced or how processes can be improved. This type of formative evaluation often employs a comparison group as a recognized standard of operations.
Typical questions asked include: Is the program being implemented as planned? If not, why not? Does the program meet recognized standards of performance?
- Outcome evaluations examine program accomplishments and effects to determine if a program is meeting its intermediate and long-term goals. This type of summative evaluation often compares current program performance against prior program performance, a comparable control group or recognized standards of performance.
Typical questions asked include: To what extent has the program achieved its goals? Has the program been more or less successful than other comparable programs? What are the unintended effects of the program?
Rather than focusing on what type of evaluation you need to do, however, it's more important that you determine first what you need to learn about the program. Knowing which questions you want to answer will determine the type of study you'll need to do and point you to the type of data you'll need to collect.