Measuring and Analyzing Performance to Find What Works
Goal-setting is a first step toward improving performance, but agencies must also measure progress toward their goals. Also, when agencies measure characteristics of the problems they are trying to tackle and of opportunities that arise, it helps them set priorities and tailor agency actions more precisely. Measurement and analysis complement agency goal-setting. They let an agency know if it is on or off track to meet its targets. Even more important, they help agencies diagnose problems, identify drivers of future performance, evaluate risk, support collaboration, and inform follow-up actions. Analyses of patterns, anomalies, and relationships help agencies discover ways to achieve more value for the taxpayer’s money.
The Department of Veterans Affairs (VA) used deeper analysis of its data to yield improvements. Increased analysis of its customer satisfaction, quality, and service measures enabled VA to increase the percentage of medical care new patient appointments completed within 14 days of a patient’s desired date to 89 percent in 2011, up from 84 percent the previous year. Also, an initiative at VA medical centers to reduce health care associated infections, featured in articles by the Wall Street Journal and the Commonwealth Fund, helped VA enlist leaders at the local, regional, and national levels to identify and spread highly effective infection-reducing practices. To accomplish this, VA mentored facilities with persistent sub-par performance and employed a multidimensional report card with quarterly measures of ICU and acute care performance, resulting in a clear payoff – one type of infection rate, for example, experienced more than a three-fold reduction over five years.
Similarly, the Federal Aviation Administration has implemented an agency-wide safety management approach – the Safety Management System (SMS). This system employs complex algorithms not only to analyze the causes of incidents, problems, and accidents including hazards that could arise from industry or agency processes. Because hazards usually arise from the interaction of various factors, putting the Safety Management System into operation has also had the effect of breaking down the stovepipes that contain and isolate various pieces of the aviation safety picture. At Seattle-Tacoma Airport, for example, an increase in ramp accidents connected with employees loading planes and driving around terminals was traced to the fact that people with varying skills and responsibilities performed diverse jobs in these risk areas. The airport addressed this by holding separate, targeted training for each group of employees and painting a line to distinguish the area of operation for tarmac personnel from that of runway personnel. As a result, the number of accidents fell sharply.
These are just some of the ways that Federal agencies are strengthening their measurement and analytic practices to find what works and ways to reduce costs. At the same time, they are developing a better sense of different kinds of measurement useful for different situations, including annual performance reports, dashboards and databases, and more frequent, actionable information, and integrating performance measurement and analysis with other evidence.
...Data is a powerful tool to determine results. We can't ignore facts. We can't ignore data.
President Barack Obama, July 24, 2009
Annual Performance Measurement
Each year, agencies report progress in their annual performance reports. Increased leadership attention is leading to improvements in many of these reports. The Department of the Interior’s annual performance plan and report, for example, has a number of noteworthy features. It shows performance trends, trends in spending, and a narrative discussion of changes in performance, causal factors, and agency actions. The “Digging Deeper” section maps the mission area to key outcome goals, end outcomes, intermediate outcomes, key performance indicators, and resources. The “Evaluations” section has an easy-to-read overview of completed evaluations and actions being taken in response to the findings.
Data and Dashboards
In addition to the Administration’s path-breaking site Data.gov, opening unprecedented volumes of Federal data to the public, agencies are developing a wide variety of approaches to share their data and analyses with the public. Dashboards and databases provide a more complete picture of progress on long and short-term priorities, both for objectives under direct agency control but also for those shared with delivery partners, as the following examples illustrate.
Education Dashboard
To monitor the country's progress towards reaching the President’s goal that, by 2020, the United States will again have the highest proportion of college graduates in the world, the Department of Education developed the United States Education Dashboard. The Dashboard displays information about progress towards this important goal in a simple, easy to read format that allows readers to compare across States, graph results, and drill down into the data.
U.S. Patent and Trademark Office Dashboards
Last year, the USPTO launched the USPTO Data Visualization Center on its web site. The Patents Dashboard provides more refined pendency information than was previously available, as well as critical performance indicators such as the number of applications in the backlog, production, actions per disposal and our staffing levels. The Trademarks Dashboard provides the public access to information on trademark pendency, quality, application filings, registrations, and an assessment of inventory for pending applications as well as new applications awaiting the examiner’s first office action. The Office of Policy and External Affairs (OPEA) dashboard gives the public an overview of USPTO’s contributions in the development of strong intellectual property systems, domestically and abroad. It also provides metrics on the number of Patent Prosecution Highway (PPH) cases filed with the USPTO, as well as the number of programs, officials, and represented countries trained by the Global Intellectual Property Academy (GIPA).
EPA's Enforcement and Compliance History Online (ECHO)
ECHO provides fast, integrated searches of EPA and State data for regulated facilities. ECHO has numerous features, including a search for criminal enforcement cases and web developer tools that make it easy to tap into ECHO reports and maps. It also includes a tool that gives the public important information about pollutants that are released into local waterways. The Discharge Monitoring Report (DMR) Pollutant Loading Tool brings together millions of records and allows for easy searching and mapping of water pollution by local area, watershed, company, industry sector, and pollutant. Americans can use this new tool to protect their health and the health of their communities.
Searches using the DMR Pollutant Loading Tool return “top ten” lists to help users easily identify facilities and industries that are discharging the most pollution and which water bodies are most affected. When discharges are above permitted levels, users can view the violations and link to details about enforcement actions that EPA and States have taken to address these violations.
.
FAA’s Destination 2025 Performance Report
The Federal Aviation Administration publishes detailed quarterly information on its performance measures in the Destination 2025 Performance report. The reported measures fall into the categories of the Next Level of Safety, the Workplace of Choice, Delivering Aviation Access through Innovation, Sustaining Our Future, and improved Global Performance through Collaboration. Here the public can view data on issues from the commercial air carrier rate for 2011 (a reassuring zero percent), runway incursions, performance based navigation, to noise and emissions exposures.
National Highway Traffic Safety Administration
The National Highway Traffic Safety Administration offers several databases on its website that cover a wide range of traffic safety issues. For example, the Special Crash Investigations (SCI) page offers in-depth and detailed crash investigation data. In particular, it examines the safety impact of new, emerging, and rapidly changing technology (such as air bags and alternative fuel systems) and offers ways to explore alleged or potential vehicle defects. The Fatality Analysis Reporting System (FARS) is a nationwide census that gives NHTSA, Congress and the American public yearly data about fatal injuries suffered in motor vehicle crashes.
Occupational Health and Safety Information
The Occupational Safety and Health Administration’s website has a wealth of information that covers workplace injury and illness statistics and inspection data. It also has a section on commonly used statistics that includes data on the annual number of worker injuries, illnesses and fatalities (also available from the Bureau of Labor Statistics here), the four leading categories of events leading to fatal work injuries (transportation incidents, assaults and violent acts, contact with objects and equipment, and falls – together responsible for more than half (55%) of worker deaths in 2010), and the Top 10 most frequently cited OSHA standards violated in FY 2011 (Federal OSHA data).
USDA’s Food Environment Atlas
The Food Environment Atlas assembles statistics on three broad categories of food environment factors--food choices, health and well-being, and community characteristics. Statistics on food choices include indicators of the community's access to and acquisition of healthy, affordable food. Health and well-being statistics include indicators of the community’s success in maintaining healthy diets, such as: food insecurity; diabetes and obesity rates; and physical activity levels. Community characteristics that may influence the food environment include demographic composition, income and poverty, population loss, metro-nonmetro status, natural amenities, and recreation and fitness centers. The Atlas currently includes 168 indicators of the food environment. The year and geographic level of indicators vary to better accommodate data from a variety of sources. Some data are from the Census of Population in 2000 while others are as recent as 2009. Some are at the county level while others are at the State or regional level. The most recent county-level data are used whenever possible.
