This is the accessible text file for GAO report number GAO-09-680G 
entitled 'Assessing the Reliability of Computer-Processed Data' 
which was released on July 13, 2009.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Applied Research and Methods: 

July 2009: 

External Version I: 

Assessing the Reliability of Computer-Processed Data: 

GAO-09-680G: 

Contents: 

Preface: 

Section 1: Introduction: 

Section 2: Understanding Data Reliability: 

Section 3: Deciding Whether a Data Reliability Assessment Is Necessary: 

Section 4: Determining the Extent of the Assessment: 

Section 5: Planning a Data Reliability Assessment: 

Section 6: Steps in the Assessment: 

Section 7: Making the Data Reliability Determination: 

Section 8: Including Appropriate Language in the Report: 

Appendix I: Collecting Information for Reliability Assessments: 

Appendix II: Sample Interview Questions and Issues Related to Process 
and System Controls: 

Appendix III: Sample Language for Reporting on Data Reliability: 

Figures: 

Figure 1: Factors That Help Decide Whether to Use Data: 

Figure 2: Determining the Need for a Data Reliability Assessment: 

Figure 3: The Framework of the Data Reliability Assessment Process: 

Figure 4: Steps in Assessing Data Reliability: 

Figure 5: Making the Final Determination: 

Abbreviations: 

GAGAS: Generally accepted government auditing standards: 

GPRA: Government Performance and Results Act: 

[End of section] 

Preface: 

Computer-processed data from outside sources are often central to audit 
reports. While these data are simply another type of evidence to rely 
on, assessing them may require more technical effort than other types. 
Computer-processed data, resulting from computer processing or entering 
data into a computer system, can vary in form. They may be data in 
electronic files or tables in published reports, including paper 
copies. (More specific examples are discussed in section 2.) 

Intended to demystify the assessment of computer-processed data, this 
guide is consistent with the Yellow Book--the 2007 Government Auditing 
Standards--which defines generally accepted government auditing 
standards (GAGAS), and it replaces the 2002 Assessing the Reliability 
of Computer-Processed Data.[Footnote 1] 

Various tests of sufficiency and appropriateness are used for all types 
of evidence to assess whether the evidence standard is met. Because 
assessing computer-processed data requires more technical tests, it may 
seem that such data are subject to a higher standard of testing than 
other evidence. This is not the case. For example, we apply many of the 
same tests of sufficiency and appropriateness that we apply to other 
types of evidence, but in assessing computer-processed data, we focus 
on one test in the evidence standard--appropriateness. Appropriateness 
includes validity and reliability, which in turn includes the 
completeness and accuracy of the data. 

This guide therefore provides a flexible, risk-based framework for data 
reliability assessments that can be geared to the specific 
circumstances of each engagement. The framework gives structure to 
planning and reporting, facilitates the right mix of skills on each 
engagement, and ensures timely management acceptance of assessment 
strategies. The framework is built on: 

* making use of existing information about the data, 

* conducting only the amount of work necessary to determine whether the 
data are reliable enough for your purposes, 

* maximizing professional judgment, and: 

* bringing the appropriate people, including management, to the table 
at key decision points. 

The ultimate goal of data reliability assessment is to determine 
whether you can use the data for your intended purposes. This guide is 
designed to help you make an appropriate, defensible assessment in the 
most efficient manner. With any related questions, call Sidney 
Schwartz, the Director of the Center for Design, Methods, and Analysis 
in the Applied Research and Methods team, at (202) 512-7387. 

Signed by: 

Nancy Kingsbury: 
Managing Director, Applied Research and Methods: 

[End of section] 

Section 1: Introduction: 

This guide explains what data reliability means and provides a 
framework for assessing the reliability of computer-processed data. It 
includes guidance on determining when to do a data reliability 
assessment, factors contributing to the extent of the assessment, and 
suggestions for steps to take in conducting the assessment. 

The ultimate goal of a data reliability assessment is to gather and 
evaluate the information needed to make the following decision: Can we 
use the data to answer the research question? Figure 1 gives an 
overview of the factors that help inform that decision. Not all the 
factors in the figure may be necessary for all research projects. 

Figure 1: Factors That Help Decide Whether to Use Data: 

[Refer to PDF for image: illustration] 

Factors That Help Decide Whether to Use Data: 
Importance of data to message; 
Strength of corroborating evidence; 
Risk of using data; 
Review of existing information (documentation, interviews); 
Results of electronic testing; 
Results of tracing to or from source documents; 
Results of review of selected; system controls. 

Source: GAO. 

In addition, the guide suggests appropriate language for different 
circumstances in reporting the results of your assessment. Finally, it 
describes in detail all the stages of an assessment. 

[End of section] 

Section 2: Understanding Data Reliability: 

For the purposes of this guidance, data reliability refers to the 
accuracy and completeness of computer-processed data, given the uses 
they are intended for. Computer-processed data may be data (1) entered 
into a computer system or (2) resulting from computer processing. In 
this guide, "data" always means computer-processed data. 

Computer-processed data can vary in form--from electronic files to 
tables in published reports. The definition of computer-processed data 
is therefore broad. Some specific examples of computer-processed data 
are: 

* data extracts from databases, data warehouses, or data repositories; 

* data maintained in Microsoft Excel or Access or similar commercial 
products; 

* data extracts from enterprise software applications supported by 
information technology departments or contractors; 

* public use data or other replicated detail or summary-level databases 
accessible through an application other than the original source 
system; 

* data collected from forms and surveys on Web portals; and: 

* data summarized in a report or copied from a table in a document. 

While the focus here is on computer-processed data, some of the 
principles and assessment tasks also apply to other kinds of data. 

This guide will help you design a data reliability assessment 
appropriate to your project's purpose and then evaluate the results of 
the assessment. According to the Yellow Book, auditors should assess 
the sufficiency and appropriateness of computer-processed information, 
regardless of whether this information is provided to auditors or they 
extract it independently.[Footnote 2] A data reliability assessment 
should be performed for computer-processed data that materially support 
findings, conclusions, or recommendations. 

In this context, reliability means that data are reasonably complete 
and accurate, meet your intended purposes, and are not subject to 
inappropriate alteration. 

* Completeness refers to the extent that relevant records are present 
and the fields in each record are populated appropriately. 

* Accuracy refers to the extent that recorded data reflect the actual 
underlying information. 

* Consistency, a subcategory of accuracy, refers to the need to obtain 
and use data that are clear and well defined enough to yield similar 
results in similar analyses. For example, if data are entered at 
multiple sites, inconsistent interpretation of data entry rules can 
lead to data that, taken as a whole, are unreliable. 

While this guide focuses only on the reliability of data in terms of 
completeness and accuracy, other data quality considerations are just 
as important. In particular, consider validity. Validity (as used here) 
refers to whether the data actually represent what you think is being 
measured. For example, if we are interested in analyzing job 
performance and a field in the database is labeled "annual evaluation 
score," we need to know whether that field seems like a reasonable way 
to gain information on a person's job performance or whether it 
represents another kind of evaluation score. 

Consider data validity and reliability issues early on a job. Data 
analysts, methodologists, information technology specialists, 
statisticians, and other technical specialists can assist you. 

Assessments of reliability are made in the broader context of the 
particular characteristics of your research project and the risk 
associated with the possibility of using insufficiently reliable data. 
A decision that computer-processed data are reliable does not 
necessarily mean that the data are error-free. Errors are considered 
acceptable in this circumstance: You have assessed the associated risk 
and conclude that the errors are not substantial enough to cause a 
reasonable person, aware of the errors, to doubt a finding, conclusion, 
or recommendation based on the data. 

[End of section] 

Section 3: Deciding Whether a Data Reliability Assessment Is Necessary: 

To decide whether a data reliability assessment is necessary, consider 
the planned use of the data. Figure 2 illustrates the decision process. 

Figure 2: Determining the Need for a Data Reliability Assessment: 

[Refer to PDF for image: decision chart] 

What is the type of engagement? 
For Financial and financial-related audits: 
Use guidance in Financial Audit Manual(GAO-08-585G, GAO-08-586G, and 
GAO-07-1173G) and Federal Information System Controls Audit Manual (GAO-
09-232G). 

All other engagements: 

Do you anticipate that data will materially support findings, 
conclusions, or recommendations? 
If no, If primarily background or contextual information that does not 
materially affect findings, determine if from best available source; 
If yes, continue. 

Does the research question require a determination of the reliability 
of an information system? 
If yes, Conduct a computer system review and disclose in the section on 
objectives, scope, and methodology the work done, results, and any 
limitations found; 
If no, continue. 

Will the data be used in multiple future engagements? 
If yes, Should you do a computer system review?
If yes, Conduct a computer system review and disclose in the section on 
objectives, scope, and methodology the work done, results, and any 
limitations found; 
If not at this time: Continue with a data reliability assessment. 

Source: GAO. 

[End of figure] 

Conditions Requiring Data Reliability Assessment: 

You should assess reliability if the data to be analyzed are intended 
to materially support your findings, conclusions, or recommendations. 
Keep in mind that a finding may include only a description of the 
condition, as in a purely descriptive report. Remember, too, that data 
can include record-level data, summary or aggregate data, and estimates 
or projections based on computer-processed data. 

In your audit plan, you should discuss briefly how you plan to assess 
data reliability, as well as any limitations that may exist because of 
shortcomings in the data. 

Conditions Not Requiring Data Reliability Assessment: 

You do not need to assess the reliability of data if their use in the 
report does not materially affect findings, conclusions, or 
recommendations. In most circumstances, information presented as 
background, context, or example does not require an assessment. For 
example, data not needing an assessment might simply set the stage for 
reporting the project's results or provide information that puts the 
results in proper context. Such information could be the size of the 
program or activity you are reviewing. While such data may not need an 
assessment, you should still ensure that they are from the best 
available sources. 

For instance, a finding might include the number of uninsured Americans 
and you might want to put this number in the context of the overall 
U.S. population. While the estimate of the number of Americans who are 
uninsured would require a data reliability assessment of some kind, as 
long as the estimate of the U.S. population were determined to have 
come from a reliable source (for instance, the U.S. Census), this 
number would not require an assessment. 

Sometimes data that seem like background information may materially 
affect the findings. If data in the report appear to provide context 
but also serve as an impetus for the audit or are likely to be 
subjected to a high degree of scrutiny, you should conduct an 
assessment. For example, if an estimate of the amount of dietary 
supplements Americans take is presented as a basis for conducting an 
audit of a regulatory agency, you should conduct a data reliability 
assessment to be reasonably confident of the estimate's accuracy. 

In addition, if an audit relies on information that is used for widely 
accepted purposes and is obtained from sources generally recognized as 
appropriate, it may not be practical or necessary to conduct procedures 
to verify the information. Such information could include, for example, 
economic statistics that government agencies issue for adjusting for 
inflation or other such information authoritative organizations issue. 
Deciding to use such information without further assessment calls for 
professional judgment by individuals with appropriate knowledge of the 
nature of the information and how it is being used in the audit (for 
example, technical specialists). 

Finally, for financial audits, you should not follow this guidance in 
assessing data reliability. For financial audits, which include 
financial statements and financial-related audits, you should follow 
the Financial Audit Manual and the Federal Information System Controls 
Audit Manual.[Footnote 3] In an information system review, all controls 
in a computer system--for the full range of application functions and 
products--are assessed and tested. This includes: 

1. examining the general and application controls of a computer system, 
[Footnote 4] 

2. testing whether those controls are being complied with, and: 

3. testing data produced by the system.[Footnote 5] 

Information technology specialists can help you design an appropriate 
information system review, given your research question, and connect 
you with the resources you need. 

[End of section] 

Section 4: Determining the Extent of the Assessment: 

The ultimate goal of a data reliability assessment is to determine 
whether you can use the data to answer the research questions. Perform 
assessments only for the portions of the data that are relevant to the 
project. You may need to assess only a few elements of a database or 
you may need to assess many variables in various modules of a data 
collection system. The extent of an assessment depends on the: 

* expected importance of the data to the final report, 

* strength or weakness of any corroborating evidence, and: 

* anticipated level of risk in using the data. 

Expected Importance of the Data in the Final Report: 

In making an assessment, consider the data in the context of the final 
report: 

* Will the project team depend on the data alone to answer a research 
question? 

If the data are the sole source of information leading to findings and 
recommendations, a more extensive assessment may be necessary than if 
you have strong corroborating evidence. 

* Will the data be summarized or will detailed information be reported? 

Although the data elements underlying the summary data still need to be 
assessed, the presentation of more detailed information may require a 
deeper assessment. If you plan to report detailed information, then the 
assessment should focus on whether the data are reliable at the level 
you plan to report. For example, if you need to report only total 
dollars spent, you may have to do an assessment that does not go as 
deep as if you planned to report on expenditures in specific 
categories. 

* Is it important to have precise data? 

Do you need to do an assessment that allows you to report approximate 
data or do you need to do a more in-depth assessment that would allow 
you to report exact numbers? For example, when assessing the ability of 
charities to respond to a disaster, is it enough to know that resources 
will shelter a range of 400,000 to 500,000 people or do we need to know 
the exact figure? 

Corroborating Evidence: 

Consider the extent to which corroborating evidence exists and will 
independently support the findings, conclusions, or recommendations. 
Corroborating evidence is independent evidence that supports 
information in a database or derived from one. Such evidence, if 
available, can be found in alternative databases or expert views. 
Corroborating evidence is unique to each review, and its strength--or 
persuasiveness--varies. 

For help in deciding the strength or weakness of corroborating 
evidence, consider the extent to which the corroborating evidence is: 

* consistent with Yellow Book standards of evidence--sufficiency and 
appropriateness; 

* able to provide crucial support; 

* drawn from multiple sources; 

* drawn from multiple types of evidence, such as testimonial, 
documentary, and physical; and: 

* independent of other sources. 

Risk Level in Using the Data: 

Risk is the likelihood that using data of questionable reliability 
could have substantial negative consequences on the decisions of 
policymakers and others. To do a risk assessment, consider the 
following risk conditions, in which the data: 

* could be used to inform legislation, policy, or a program that could 
have substantial effect; 

* could be used to inform important decisions by individuals or 
organizations with an interest in the subject; 

* will be the basis for numbers that are likely to be widely quoted, as 
in the statement, The United States owes the United Nations about $1.3 
billion for the regular and peacekeeping budgets; 

* are relevant to a sensitive or controversial subject; 

* have been judged for their quality by experts or external 
stakeholders who have taken positions on the information. 

Bear in mind that any one condition may have more importance than 
another, depending on the project. 

[End of section] 

The assessment process should take these factors into account, along 
with what is learned during the assessment. The process is likely to 
differ from one job to another. However, it should include sufficient 
work to allow the auditor to have a good understanding of how the data 
were collected, the systems they were extracted from, and the process 
and system controls related to the key data elements. Technical 
specialists can help you consider these factors and plan your work. 

[End of section] 

Figure 3 illustrates the overall framework of the process for data 
reliability assessment. The framework identifies several key stages in 
the assessment, as well as actions to take and decisions to expect as 
you move through the process. The framework allows you to identify the 
appropriate mix of assessment steps to fit the particular needs of the 
job. In most cases, not all the elements in figure 3 would be necessary 
to complete the assessment. (Specific actions for each stage are 
discussed in sections 6 and 7.) 

Figure 3: The Framework of the Data Reliability Assessment Process: 

[Refer to PDF for image: illustration] 

All phases of assessment are influenced by: 
* importance of data to message, 
* strength of corroborating evidence, and, 
* risk of using data. 

Plan the assessment: 
* Review existing information from agency, GAO, and others; 
* Determine if data are appropriate[A]; 
* Request and receive additional information if needed. 

Perform data assessment with appropriate mix of work: 
* Review existing information; 
* Trace sample; 
* Electronic testing; 
* Review selected system controls. 

Make determination: 
Enough information for reliability determination? 
If no, Request and receive additional information if needed; 
If yes, determination: 
* Sufficiently reliable; 
* Not sufficiently reliable; or, 
* Undetermined reliability. 

Source: GAO. 

[A] After a review of initial information, you may determine that the 
data are not appropriate for answering the research question (for 
example, the database may not contain relevant data elements). 

[End of figure] 

[End of section] 

Section 5: Planning a Data Reliability Assessment: 

When you plan a data reliability assessment, you need to decide on the 
timing--when to perform the assessment--and how to document your plans 
for the assessment and the assessment itself. In addition, important 
decisions about obtaining data at the summary or record levels of 
detail will affect how you can use the data in the report and the depth 
of your data reliability assessment. 

Timing an Assessment: 

Generally, a data reliability assessment is performed as early as 
possible on a project, preferably during the design phase. The audit 
plan helps by reflecting data reliability issues and any additional 
steps that still need to be taken in assessing the reliability of 
critical data. The audit team generally takes initial steps to test the 
data and review existing information about the data and the system that 
produces them before making the audit plan final. Examining this 
information early is also necessary to help the team determine whether 
the data would be appropriate for addressing the research question in 
the first place. 

In some instances, the timing of the project may be very short. Section 
6 has some suggestions for meeting data reliability assessment 
requirements in a short period of time. 

Level of Detail of the Data: 

Record-level data give the greatest opportunity to analyze the data and 
fully assess their reliability. This opportunity may be most important 
for data that are key to your research objectives. Summary-level data 
or a subset of data still require a data reliability assessment, but 
testing and understanding of the data may be more limited. It will also 
be important to understand any process used for summarizing or 
extracting the data; you may need to request the computer code or 
queries used to derive the data. Obtaining the code used to derive the 
records allows you a greater ability to see whether the correct 
criteria were used in providing you with the records, decreasing the 
chance of missing records. In general, it is preferable to obtain 
record-level data because they permit a more comprehensive data 
reliability assessment. 

[End of section] 

For example, auditors might be reviewing the timeliness of agency 
decisions. If you obtained the detailed data for all decisions, you 
might be able to report timeliness data at the national, regional, and 
field office levels. In addition, with this record-level data, you 
could check their reliability to see if important information was 
missing or whether duplicate records were in the file. You could also 
determine, if you were given beginning and ending dates, whether the 
agency was calculating timeliness accurately. The record-level data 
request could give you more reporting flexibility, more opportunities 
to find data problems which could lead to a recommendation, and a 
greater ability to use the data in the findings. A request for only 
national, summary-level data would not allow you to report data at the 
regional and field office levels, might not allow you to fully test 
data reliability, and depending on the intended use of the data, could 
preclude using the data in the findings section of the report. 

Documenting the Assessment: 

All work performed as part of the data reliability assessment should be 
documented and included in the project's documentation. Required 
documentation includes a plan for steps you will take in the 
assessment, as well as the results from all testing, documentation 
review, and interviews related to data reliability. 

In addition, decisions made during the assessment, including the final 
determination of whether the data are sufficiently reliable for the 
overall purposes of the review, should be summarized in the 
documentation. The documentation should make clear what steps the 
project team took and what conclusions they reached. 

[End of section] 

Section 6: Steps in the Assessment: 

Data reliability as a process includes a range of possible steps, as 
shown in figure 4. Assessing data reliability can entail reviewing 
existing information about the data, including conducting interviews 
with officials from the organization being audited; performing tests on 
the data, including advanced electronic analysis; tracing to and from 
source documents; and reviewing selected system controls. 

Figure 4: Steps in Assessing Data Reliability: 

[Refer to PDF for image: illustration] 

* Review existing information; 
* Electronic testing; 
* Trace sample; 
* Review selected system controls. 

Source: GAO. 

[End of figure] 

Deciding which steps to take is an iterative process. Most often you 
may start with the relatively simple steps of reviewing existing 
information and basic testing. The outcome of these steps may lead you 
to take other steps in order to gather enough information. 

The mix of steps you take depends on any potential weaknesses you 
identify as you proceed and circumstances specific to the job, such as 
the importance of the data to the review and corroborating evidence. 
Focus particularly on the aspects of the data that pose the greatest 
potential risk, especially for the more labor-intensive activities. 
Some audits may take an extremely short time to complete; this section 
provides some advice for this situation. 

As discussed in section 5, these steps take place early in the project 
and include the audit team members, as well as appropriate technical 
staff. The time and extent needed to take any of or all these steps 
will depend on the project and the amount of risk involved. 

Reviewing Existing Information: 

A review of existing information helps you determine what is already 
known about the data and the computer processing. The related 
information you collect can indicate both the accuracy and completeness 
of the entry and processing of the data, as well as how data integrity 
is maintained. This information can be in the form of reports, studies, 
or interviews with individuals who are knowledgeable about the data and 
the system. Sources for related information include the U.S. Government 
Accountability Office (GAO), the agency under review, and others. 

GAO: 

Figure 18: GAO may already have related information in its reports 
available at [hyperlink, http://www.gao.gov]. Consider whether 
information in any relevant GAO report is timely and appropriate for 
your uses. 

GAO's Web site also provides other useful information. For example, in 
conducting the annual governmentwide consolidated financial audit, 
GAO's Information Technology team has been involved in reporting on the 
effectiveness of controls for financial information systems at major 
federal agencies, and relevant reports may be found on the site. 

Agency under Review: 

Another source of information is the organization being reviewed. You 
can obtain documentation about a system, such as users' manuals, data 
dictionaries, system documentation, table layouts, codebooks, and data 
quality assurance program materials. You can also ask officials 
questions about their system and how it is used. You can often learn 
initial information about data and data reliability by interviewing 
agency officials and computer system specialists. 

Ideally, as you engage in a project, interviews take place early. You 
can often identify potential reliability issues with the data in the 
initial steps of the assessment from interview questions, before you 
have done further assessment work. Interviewing agency officials early 
about how appropriate the data are for your research questions can help 
you make decisions as you plan further work to assess the reliability 
of the data. Interview questions focus on the completeness and accuracy 
of the data and the internal controls surrounding the information 
system that produces the data. Use what you know about the program 
under review and the computer system to focus interview questions on 
the specific issues that most directly affect the reliability of the 
data you plan to use in the audit. 

In addition, agency officials are often aware of evaluations of their 
computer data or systems and usually can direct you to them. However, 
keep in mind that information from agency officials may be biased. 
Consider asking appropriate technical specialists to help in evaluating 
this information. (Appendixes I and II have sample questions on 
document requests, accuracy and completeness concerns, and process and 
system control issues.) 

Agency information also includes reports under the Federal Managers' 
Financial Integrity Act and the Clinger-Cohen Act, Government 
Performance and Results Act (GPRA) plans and reports, and Chief 
Information Officer and Inspector General reports.[Footnote 6] Some of 
this information can be found in agency home pages on the Web. 

Other Sources: 

Other sources include organizations and data users, as well as 
libraries of relevant literature. To help you identify them, you can 
use a variety of databases and other research tools that include the 
Congressional Research Service Public Policy Literature Abstracts and 
other organizations' Web sites. Additionally, agency officials may be 
able to identify outside users of their data. 

Statistics collected and published by federal government statistical 
agencies constitute a significant portion of the available information 
about the U.S. economy, population, natural resources, environment, and 
public and private institutions. Standards and guidelines governing 
federal statistical agencies are intended to ensure that their surveys 
and studies are designed to produce reliable data as efficiently as 
possible and that their methods are documented and results presented in 
a manner that makes the data as accessible and useful as possible. In 
most cases, federal statistical agencies have information on their 
statistical collection procedures and methods readily available on the 
Internet. Often, this published information serves as much of the 
documentation you will need to review in conducting your data 
reliability assessment. 

Although data that federal statistical agencies collect are generally 
reliable for their purposes, you must still assess whether these data 
are sufficiently reliable for your purpose. For example, census data 
indicate how many natural-born children are living with respondents, 
but these data are not reliable for determining how many natural-born 
children a respondent has ever had, because some children might be 
living independently or with other relatives or living in college or 
the military. 

It is also possible to inappropriately use otherwise reliable federal 
statistical data. For example, an audit team might want to determine 
from Current Population Survey data the proportion of law enforcement 
officers who are Asian. Because this information is at the intersection 
of two separate subpopulations--race and occupation--the number of 
people will be too small to be reliable because of the sampling design 
used to collect these data. Consider these kinds of data reliability 
issues when planning to use federal statistical agency data. 

Performing Data Testing: 

Data testing can be done by applying logical tests to electronic data 
files or paper copies of reports. For record-level electronic data, you 
can use computer programs to test all entries of key data elements in 
an entire data file.[Footnote 7] Keep in mind that you test only the 
data elements you plan to use in your review. 

For paper copy or summarized data--provided by the agency or retrieved 
from the Internet--ask for the electronic data file that was used to 
create them. If you are unable to obtain electronic data, use the paper 
copy or summarized data and, to the extent possible, manually apply the 
tests to all instances of key data elements or, if the report or 
summary is voluminous, to a sample of them. 

Whether you have an electronic data file or a paper copy report or 
summary, you can apply the same types of tests to the data. The tests 
you conduct will vary for each assessment and can include: 

* checking total number of records provided against agency totals; 

* testing for missing data, either entire missing records or missing 
values in key data elements; 

* looking for duplicate records; 

* looking for invalid or duplicate identifiers; 

* testing for values outside a designated range; 

* looking for dates outside valid time periods or in an illogical 
progression; 

* following up on troubling aspects of the data--such as extremely high 
values associated with a certain geographic location--found while 
analyzing the data; 

* testing relationships between data elements (sometimes by merely 
doing a cross tabulation), such as whether data elements follow a skip 
pattern from a questionnaire; and: 

* verifying that computer processing is accurate and complete, such as 
testing a formula used in generating specific data elements, or testing 
to ensure that edit checks are working correctly. 

Depending on what will be tested, this testing can require a range of 
programming skills--from creating cross tabulations on related data 
elements to duplicating an intricate automated process with more 
advanced programming techniques. Consider asking appropriate technical 
specialists to help in conducting this testing. 

Be sure to keep a log of your testing to include in the project's 
documentation. 

Tracing to and from Source Documents: 

When record-level data are available, tracing a sample of data records 
to source documents helps you determine whether the computer data 
accurately and completely reflect these documents. In deciding what and 
how to trace, consider the relative risks of overstating or 
understating conclusions drawn from the data. For example, if you are 
particularly concerned that questionable cases might not have been 
entered into the computer system and that, as a result, the degree of 
compliance may be overstated, consider tracing from source documents to 
the database. However, if you are more concerned that ineligible cases 
have been included in the database and that, as a result, the potential 
problems may be understated, consider tracing from the database back to 
source documents. 

The reason to trace only a sample is that sampling saves time and cost. 
To be useful, however, the sample should be random and large enough to 
estimate the error rate within reasonable levels of precision. Tracing 
an appropriate random sample can allow you to estimate the error rate 
and the magnitude of errors for the entire data file. It is this error 
rate that helps you determine the data reliability. (Consult 
statisticians to help you select the sampling method most suited to 
your project.) 

Generally, every data file has some degree of error--here, example 1 
shows error rate, example 2 magnitude of errors: 

Example 1. In a random sample, 10 percent of the data records have 
incorrect dates, and those dates are off by an average of 3 days. 
Depending on what the data are used for, 3 days may not compromise 
reliability. 

Example 2. The value of a data element was incorrectly entered as 
$100,000 rather than $1,000,000. The documentation of the database 
showed that the acceptable range for this data element was between $100 
and $5,000,000. Therefore, the electronic testing would have confirmed 
that the value of $100,000 fell within that range. In this case, the 
error could be caught not by electronic testing but only by tracing the 
data to source documents. 

Tracing to Source Documents: 

Consider tracing to source documents when (1) they are available 
relatively easily or (2) the possible magnitude of error is especially 
critical. 

To trace a sample to source documents, match the entered data with the 
corresponding data in the source documents. In attempting to trace 
entered data back to source documents, several problems can arise. 
Source documents may not be available because they were destroyed, were 
never created, or are not centrally located. 

Several options are possible if source documents are not available. For 
documents that were never created--for example, when data may be based 
on electronic submissions--use interviews to obtain related 
information, any corroborating evidence obtained earlier, or a review 
of the adequacy of system controls. 

Tracing from Source Documents: 

Consider tracing from source documents instead of, or in addition to, 
tracing a sample to source documents when you have concerns that the 
data are not complete. To trace a sample from source documents, match 
the source documents with the entered data. Such tracing may be 
appropriate to determine whether all data are completely entered. 
However, if source documents were never created or are now missing, you 
cannot identify the missing data. 

Reviewing Selected System Controls: 

Your review of selected system controls--the underlying structures and 
processes of the computer where data are maintained--can provide some 
assurance that the data are sufficiently reliable. Examples of system 
controls are limits on access to the system and edit checks on data 
entered into the system.[Footnote 8] Controls can reduce to an 
acceptable level the risk that a significant mistake could occur and 
remain undetected and uncorrected. Limit the review to evaluating the 
specific controls that can most directly affect the reliability of the 
data in question. 

Choose areas for review on the basis of what is known about the system. 
Sometimes you identify potential system control problems in the first 
steps of the assessment. Other times, you may learn that source 
documents are not readily available. Therefore, a review of selected 
system controls is a good way to determine whether data were entered 
reliably. If needed, consult information system auditors or other 
technical specialists for help in evaluating system controls. 

Using what you know about the system, concentrate on evaluating the 
controls that most directly affect the data. These controls will 
usually include (1) certain general controls, such as logical access 
and control of changes to the data, and (2) the application controls 
that help ensure that the data are accurate and complete, as well as 
authorized. 

The steps for reviewing selected system controls are: 

* gain a detailed understanding of the system as it relates to the data 
and: 

* identify and assess the application and general controls that are 
critical to ensuring the reliability of the data required for the 
audit. 

Working within Short Time Periods: 

In some instances, a project may have a time period that is very short. 
Despite this, you may have time to review existing information and test 
data that are critical for answering a research question. For example, 
you can question knowledgeable agency staff about data reliability or 
review GAO or Inspector General reports to quickly gather information 
about data reliability issues. 

In addition, critical data elements can generally be tested 
electronically for obvious errors of completeness and accuracy in a 
short time on all but the most complicated or immense files. From that 
review and testing, you will be able to make a more informed 
determination about whether the data are sufficiently reliable to use 
for the purpose of your review and to decide whether further 
investigation is needed. 

[End of section] 

Section 7: Making the Data Reliability Determination: 

Review the results of your work periodically and decide whether (1) the 
data are sufficiently reliable for your job's purpose, (2) the data are 
not reliable for that purpose, or (3) additional work is needed before 
a determination can be reached. Keep in mind that you are not attesting 
to the overall reliability of the data or database. You are determining 
only the reliability of the data as needed to support the review's 
findings, conclusions, or recommendations. As you gather information 
and make your judgments, consult appropriate technical specialists for 
assistance. 

Factors to Consider in the Determination: 

To determine whether the data reliability for the engagement is 
sufficient, consider all factors related to aspects of your engagement 
as well as assessment work performed to this point. As shown in figure 
5 (and discussed in section 4), these factors include: 

* the expected importance of the data in the final report, 

* corroborating evidence, 

* level of risk of using the data, and: 

* the results of assessment work conducted so far. 

Figure 5: Making the Final Determination: 

[Refer to PDF for image: illustration] 

Factors: 
* Importance of data to message; 
* Strength of corroborating evidence; 
* Risk of using data; 
* Review of existing information (documentation, interviews); 
* Results of electronic testing; 
* Results of tracing to or from source documents; 
* Results of review of selected system controls. 

All factors combine to assist in answering the question: 

What is the final determination of reliability? 
* Sufficiently reliable; 
- Use data and disclose limitations. 

* Not sufficiently reliable; 
- Take alternative actions. 

Source: GAO. 

[End of figure] 

Considering the Results of Your Assessment Work: 

Before making a decision about the reliability of the data for your 
purposes, consider the results of all the steps you took in conducting 
the assessment. Appropriately document and review the results before 
entering into the decision-making phase of the assessment, because 
these results will, wholly or in part, provide the evidence that the 
data are sufficiently reliable--and therefore appropriate enough--or 
not sufficiently reliable for the purposes of your audit engagement. 
Remember that you may decide that you need to take further steps to 
come to a conclusion about the reliability of the data for your 
purposes. 

Outcomes to Consider in the Assessment: 

The strength of corroborating evidence and the degree of risk can 
suggest different data reliability decisions. If the corroborating 
evidence is strong and the risk is low, the data are more likely to be 
considered sufficiently reliable for your purposes. If the 
corroborating evidence is weak and the risk is high, the data are more 
likely to be considered not sufficiently reliable. If data testing did 
not raise any questions and answered all issues in the review of 
existing documentation, then the data are more likely to be considered 
sufficiently reliable for your purposes. 

The overall determination is a professional judgment that the project 
team makes in discussions with team management and technical 
specialists. 

The determination categorizes the data as sufficiently reliable, not 
sufficiently reliable, or of undetermined reliability. Each category 
has implications with respect to whether you need to take further steps 
in the assessment and whether you can use the data for your intended 
purposes. 

Sufficiently Reliable Data: 

You can consider the data sufficiently reliable when you conclude the 
following: The results of your work (including testing results and 
reviews of existing information) provide assurance that (1) the 
likelihood of significant errors or incompleteness is minimal and (2) 
the use of the data would not lead to an incorrect or unintentional 
message. You could have some problems or uncertainties about the data, 
but they would be minor, given the research questions and intended use 
of the data. 

In certain cases, after collaboration with the producers of the data, 
you may be able to make corrections that make the data sufficiently 
reliable for your purposes. You may also be able to alter your research 
question or planned use of the data to take into account any data 
limitations discovered. When your final determination indicates that 
the data are sufficiently reliable, use the data. 

Not Sufficiently Reliable Data: 

You can consider the data to be not sufficiently reliable when you 
conclude the following: The results of your work indicate (1) 
significant errors or incompleteness in some of or all the key data 
elements and (2) that using the data would probably lead to an 
incorrect or unintentional message, given the research questions and 
intended use of the data. 

When the determination indicates that the data are not sufficiently 
reliable, consider seeking evidence from other sources, including 
alternative computerized data--the reliability of which would also be 
assessed--or original data in other forms, such as surveys, case 
studies, or expert interviews. 

Coordinate with the requester if your attempts to seek reliable 
evidence from other sources are unsuccessful. Inform the requester that 
such data, necessary in order to respond to the request, are 
unavailable. Reach an agreement with the requester to: 

* redefine the research questions to eliminate the need to use the 
data, 

* use the data with appropriate disclaimers, or: 

* end the engagement. 

Remember that you and your audit team are responsible for deciding what 
data to use. Although the requester may want information based on 
insufficiently reliable data, you are responsible for ensuring that 
data are used appropriately to respond to the requester. If you decide 
you must report data that you have determined are not sufficiently 
reliable for the engagement's purpose, make the limitations of the data 
clear, so that incorrect or unintentional conclusions will not be 
drawn. Consult with appropriate management on your project before you 
agree to use data that are not sufficiently reliable. 

Sometimes, when conducting data reliability work, you encounter issues 
that might lead you to consider recommending changes to the data or 
data system. Consider further investigating data reliability issues 
where there is a strong likelihood that the data problems you have 
found could (1) materially change publicly disseminated agency 
information; (2) materially change organizational decisions where the 
organization uses these data; (3) materially misrepresent an agency's 
program or an organization's operational inputs, clients, or outcomes; 
(4) call into question whether the entity was in compliance with 
federal laws or regulations; or (5) undermine internal controls over 
high-risk operations or financial resources. 

However, if the data reliability issues are the result of the auditor's 
attempting to use the data for purposes other than those the 
organization uses them for and if they do not result in issues outlined 
above, then recommendations might not be warranted, unless the auditor 
can make a strong case that the data should be sufficiently reliable 
for the use the auditor intended. A strong case might be that these 
data are essential to document a condition critical to effective 
decisions or operations where an agency is not currently using these 
data. 

When the types of data reliability issues described above exist, 
consider making a recommendation that addresses the data problems or 
issuing a management letter to the audited organization. A management 
letter addresses management or operational issues that were found but 
that are beyond the substance of the audit. 

Data of Undetermined Reliability: 

In your assessment of work performed so far, you may be unable to 
determine whether or not the data are sufficiently reliable. For 
example, the review of some information or testing may have raised 
questions about the data's reliability, or the work has provided too 
little information to judge reliability. In these cases, you may need 
to do additional work to determine reliability. If you are unable to 
perform additional work, the data are of undetermined reliability. 

You can consider the data to be of undetermined reliability if specific 
factors are present--such as limited access to the data source, a wide 
range of data that cannot be examined with current resources, data 
limitations that prevent an adequate assessment, short time periods, 
the deletion of original computer files, or a lack of access to needed 
documents. 

For example, you may have limited or no access to information about the 
data source. This is particularly likely when international agencies, 
other countries, or private organizations produce data or when there 
are privacy concerns with the data. It can occur where there is no 
audit authority to ask for more information or when insufficient 
information exists in the form of source documents or documentation 
about the data. In such cases, an attempt is made to gather as much 
information as possible, by contacting data owners or users or by 
looking for corroborating evidence, before concluding that the data are 
of undetermined reliability. Finding sufficient corroborating evidence, 
for example, may enable you to determine that the data are reliable 
enough for your purposes. 

Alternatively, a wide range of data may have been gathered that is 
impossible to examine, such as in a survey of 50 state organizations 
asking for data that may have been collected differently within each 
state. You might then try to determine the overall reliability of the 
information, but may have insufficient resources to examine it all. 

Finally, you may have conducted a data reliability assessment and still 
be unable to determine whether the data are sufficiently reliable, 
because data limitations prevented you from doing this. For example, 
you might have found that financial data of interest are self-reported 
by other countries, affected by differences in exchange rates, and 
based on varying definitions. These limitations and lack of further 
access to the countries might prevent you from determining the 
reliability of the data. 

To minimize last-minute crises, address data reliability issues in the 
planning phase of engagements, set realistic deadlines, and be prepared 
to ask for more time to assess data if it arrives later than expected. 
Inadequate planning earlier in the engagement is not a sufficient 
reason to use data of undetermined reliability, particularly if the 
data are being used as key evidence. Even though you may sometimes work 
within extremely tight time periods or may have received data or 
supporting documentation very late in an engagement, you will not want 
to use data that can lead to an incorrect message. GAO follows this 
principle, for example, to help ensure that GAGAS is met. 

As noted with regard to insufficiently reliable data, when you decide 
that the data are of undetermined reliability, inform the audit's 
requester that sufficiently reliable data needed to respond to the 
request are unavailable. Remember that you and your audit team are 
responsible for deciding what data to use. Although the requester may 
want information based on data of undetermined reliability, you are 
responsible for ensuring that appropriate data are used. Consult with 
appropriate team management before you agree to use data of 
undetermined reliability. If you decide to use such data, clearly state 
their limitations, so that incorrect or unintentional conclusions will 
not be drawn. 

[End of section] 

Section 8: Including Appropriate Language in the Report: 

You should include in the report's methodology section a statement 
about having conformed to generally accepted government auditing 
standards. These standards include the appropriateness of the data 
being used. You conform to GAGAS by discussing in the report what you 
did to assess the data, disclose any data concerns, and make a judgment 
about the reliability of the data used in the report. 

Further, in the methodology section, discuss your assessment of data 
reliability and the basis for your determination. The language in this 
discussion will depend on whether the data are sufficiently reliable, 
not sufficiently reliable, or of undetermined reliability. You may need 
to discuss the reliability of the data in other sections of the report 
as well. Whether you do so depends on how important the data are to the 
message. (Appendix III has samples of reporting language.) 

Sufficiently Reliable Data: 

Present your basis for determining that the data are sufficiently 
reliable, given the research questions and intended use of the data. 
This presentation includes (1) noting the kind of assessment you relied 
on, (2) explaining the steps in the assessment, (3) describing any 
corrections made to the data, and (4) disclosing any data limitations. 
Such disclosure of limitations includes: 

* telling why using the data would not lead to an incorrect or 
unintentional message, 

* explaining how limitations could affect any expansion of the message, 
and: 

* pointing out that any data limitations are minor in the context of 
the engagement. 

Not Sufficiently Reliable Data: 

Present your basis for determining that the data are not sufficiently 
reliable, given the research questions and intended data use. This 
presentation should include the kind of assessment you relied on and 
explain the steps in the assessment. In this explanation, (1) describe 
the problems with the data, as well as why using them would probably 
lead to an incorrect or unintentional message, and (2) state that the 
data problems are significant or potentially significant. In addition, 
if the report contains a conclusion or recommendation supported by 
evidence other than these data, state this fact. Finally, if the data 
you assessed are not sufficiently reliable, consider whether to include 
this finding in the report and recommend that the audited organization 
take corrective action (section 7 discusses factors to consider). 

Data of Undetermined Reliability: 

Present your basis for assessing that the data are of undetermined 
reliability. Include such factors as the deletion of original computer 
files, data limitations that prevent an adequate assessment, short time 
periods, and the lack of access to the data source or to needed 
documents. Explain the reasonableness of using the data--for example, 
the data are supported by credible corroborating evidence, are widely 
used by outside experts or policymakers, or are used to present a 
general indication and not to support specific findings. 

In addition, make the limitations of the data clear, so that incorrect 
or unintentional conclusions will not be drawn from them. For example, 
indicate how using these data could lead to an incorrect or 
unintentional message. Finally, if the report contains a conclusion or 
recommendation supported by evidence other than these data, state this. 

[End of section] 

Appendix I: Collecting Information for Reliability Assessments: 

This appendix suggests ways to help you think about questions related 
to data reliability assessments. It includes sample documentation 
requests and interview questions. Using your own judgment, select or 
modify items according to the relevance to your research objectives. 
Not all items will apply in every case; focus on the specific data 
elements that you will be using. 

Data reliability assessment is often iterative, requiring some 
revisiting of issues as they arise in interviews, electronic testing, 
and data analysis. Once you have obtained the data, you may see 
unexpected elements or characteristics (for instance, a date or text 
entries in a numeric field). In such cases, it may be necessary to 
contact the source again. 

It may be helpful to obtain documentation about the data if it is 
available, whether from a large and complicated system or a simple 
spreadsheet, and to review it before questioning individuals 
responsible for and familiar with the data. Established systems are 
likely to have many processes documented. Some documentation may be 
available on the Internet. 

When information is not available beforehand, it can be requested in an 
interview. However, reviewing the documentation may require follow-up 
interviews to resolve questions brought up during document review. 

Relevant documentation to request could include: 

* information on a system's purpose and structure, such as user 
manuals, system flow charts, or design specifications; 

* information on data elements (or fields) in the system, their 
definitions, descriptions, codes, and values (as in a data dictionary); 

* financial statement audit reports, if data are used in the entity's 
financial statements; 

* the survey form used to collect the data, if applicable; 

* reviews of the quality of the data, including: 

- Inspector General or internal audit reports, 

- internal reviews and studies, 

- contractor or consultant studies, 

- reports of congressional hearings or copies of congressional 
testimony related to the data, and: 

-summaries of ongoing or planned audits, reviews, or studies of the 
system or data. 

Consider asking officials in an interview or written request some of 
the following questions if they are relevant and cannot be obtained 
from documentation you may have reviewed: 

* When was the system created, and what is its purpose? 

* How does the organization use the data from the system? 

* Who are its primary users? How do users access the system? 

* How and where are data collected? Who is responsible for data entry? 

* How current are the data? How frequently are data entered? 

* Who has access to enter or update information in the database? 

* What procedures ensure that the data system consistently captures all 
data occurrences (records, observations) and all data elements? Is 
there written documentation of these procedures? 

* Does the system have any edit checks or controls to help ensure that 
the data are entered accurately? For instance, 

- Does someone review at least a sample of data entries to ensure that 
key fields are accurate, nonduplicative, and sensible? (For example, 
the date an injury claim was filed should precede the date of 
adjudication.) If so, how often? 

- Are there electronic safeguards, such as error messages for out-of- 
range entries or inconsistent entries? 

- What are the procedures for follow-up if errors are found, and who is 
responsible for correcting them? 

- Do systematic reviews or exception reports examine accuracy and 
present error rates? How frequently? 

* Have there been changes to any of these procedures (including how a 
data element is defined, entered, or maintained) over the period of 
time for which you are requesting data? 

* Has the system had problems that would affect the quality of the 
data, such as system crashes during which data were lost? 

To assess the reliability of the data for your purposes, it may be 
useful to discuss with agency officials or other users of the data, 
such as academic researchers, how you intend to use the data. In that 
discussion, consider asking the following questions: 

* What is your opinion of the quality of the data, specifically their 
completeness and accuracy? Are there any data limitations such as data 
elements that are often incomplete or incorrect? 

* How would any limitations affect the intended use of the data? 

* Are there concerns about timeliness or usability? 

* Are there any purposes for which the data should not be used? 

* What steps have others taken to clean or otherwise improve the data 
in order to conduct an analysis (for example, imputation of missing 
fields, weighting)? 

* Is the organization taking any action to correct problems? 

In asking these questions, you are looking for information on known 
limitations of the data. You are not looking for confirmation that the 
data are reliable. You must use your judgment to make the assessment. 

You may be using data from statistical databases or data derived from 
samples or surveys, such as the Current Population Survey. If so, you 
may also need information on the following (which, for established 
systems, may be publicly available from the source): 

* population definition; 

* sample design; 

* description of data editing procedures, including imputation, if 
used; 

* impact of imputation; 

* unit and item nonresponse rates; 

* nonsampling error; 

* comparability with related data, if any; 

* information on limitations obtained from users, not producers, if 
applicable. 

[End of section] 

In developing your interview questions or information request, 
incorporate the questions or documents from above that are relevant for 
your assessment. You can start an interview or information request with 
language like the following, specifying the purpose of the request and 
data to be used: 

We are conducting a review of _______________. In this review, we plan 
to use data from your agency's ____ database or ____ program. We are 
following government auditing standards which require that we assess 
the reliability of data we use in our products. Therefore, we would 
like to ask you questions about the completeness and accuracy of the 
data and the information system that produces the data. The data fields 
we are interested in using are _____ for the purpose of _____. 

[End of table] 

[End of section] 

Appendix II: Sample Interview Questions and Issues Related to Process 
and System Controls: 

Your detailed understanding and review of selected process and system 
controls can help ensure that the data are sufficiently reliable. 
Process controls refer to an organization's policies and procedures 
that could affect the accuracy and completeness of data. System 
controls refer to the underlying structures and programming of the 
computer system that could affect the accuracy and completeness of 
data. Process and system controls differ but often interact. Both 
should be considered the internal controls surrounding the 
organization's input and use of data. 

Process and system controls can reduce to an acceptable level the risk 
that significant data mistakes could occur and remain undetected and 
uncorrected. You can often identify potential process and system 
control problems in an assessment's initial steps through interview 
questions aimed at program officials and computer system specialists. 
The issues and questions below provide some additional guidance on 
developing interview questions as they relate to system and process 
controls. 

Interviewing an agency's officials about process and system controls 
can help you make decisions about whether you need to plan further work 
to assess the reliability of the data. Use what you know about the 
program under review and the computer system to focus interview 
questions on the specific process and system controls that most 
directly affect the reliability of the data you plan to use. 

Process Controls: 

Process controls that could affect the accuracy and completeness of 
data include, among others, training, case control, guidance, incentive 
structure, interaction with stakeholders, management reviews, and 
system changes. 

Training: 

Is data system training made available to users entering data into the 
system? What is the quality of the data system training? How is the 
training implemented--for example, do all new users have to go through 
the training? Is refresher training made available? 

Case Control: 

Are procedures in place to ensure that all cases are entered into a 
data system? Can a case or transaction be processed without being 
entered into the data system? Can a case move to the next step of a 
process without having been entered into the data system? Are 
procedures in place to prevent the duplicate recording of the same 
record? 

Guidance: 

Does the agency or organization provide clear guidance for data entry 
in grey areas? For example, if a case could be accurately described in 
more than one way, is there guidance on how the case should be 
categorized when entered into the data system? 

Incentive Structure: 

How does measuring employee or agency performance affect the quality of 
data entered into the system? For example, if employees are measured on 
the timeliness of case processing, could they enter incorrect dates 
into the system, indicating that cases were completed in a timely 
manner when in fact they were not? 

Interaction with Stakeholders: 

Do users of the data or individuals whose programs are the subject of 
data records receive periodic updates regarding data in the system? Do 
these users or stakeholders have a chance to bring attention to 
incorrect data or data that need to be updated? 

Interaction with stakeholders can help make sure that the people most 
likely to have knowledge of the correct data can work to ensure its 
accuracy as it is captured in the system. 

Management Reviews: 

Does the organization's management review data informally or 
systematically? Informal management reviews could include reviews of 
summary-level reports to look for outliers or the evaluation of period- 
to-period changes, looking for differences from historic trends. 
Outliers and unusual changes could (but do not necessarily) signal 
problematic data issues. 

Do agency systematic management reviews include a random sample of 
cases that management reviews during each period? Does the computer 
system generate exception reports for unusual data being generated? 
Does management systematically review these exception reports? 

System Changes: 

What are the organizational procedures regarding changes to the system? 
For example, are reporting requirements created by policy personnel 
correctly translated into programming requirements for system 
technicians? 

Policy personnel might request reporting on the number of cases meeting 
specific criteria. Does the implemented programming generate accurate 
reporting of all cases in the system that meet those criteria? Are some 
cases meeting the criteria not reported because of programming logic 
that has errors? Are programming changes first conducted in a test 
environment before being implemented? What procedures define new data 
elements? What procedures are in place to change data elements? 

System Controls: 

System controls that could affect the accuracy and completeness of data 
include, among others, edit checks, access controls, system assigned 
data, and case history. 

Edit Checks: 

When personnel enter information in the system, do they receive error 
messages when they enter obviously incorrect data? For instance, edit 
checks could demand certain precision for dates that can be entered. If 
money is being obligated for a current fiscal year, does the system 
allow only dates from the current fiscal year? 

The precision of data entry that edit checks demand can be important in 
determining the reliability of the data. Sometimes the edit checks are 
not precise enough to ensure data quality. Conversely, the precision of 
the edit checks could affect data quality negatively. For instance, if 
only some data entries are allowed by edit checks in the system, do 
personnel enter data that are allowed by the system but that are 
incorrect so they can avoid the edit checks? 

Access Controls: 

Who can access the system? What controls limit access to only the 
appropriate people? What are the controls on who has "read" access 
versus "write" access to the system? Who is able to change programming 
in the system? 

System-Assigned Data: 

Another system control could have the computer assign data instead of 
their being entered by agency personnel. For instance, does the 
computer generate a time and date stamp? This could ensure that dates 
are accurate and not susceptible to manipulation. 

Case History: 

Does the system maintain historic data about the case? For instance, if 
a case moves from an old to a new status, is this history captured, or 
is the old status overwritten? 

While auditors can learn about process and system controls through 
interview procedures, they should take additional steps to validate the 
effectiveness of process and system controls. The amount of validation 
needed is affected by the expected importance of the data to the final 
report. Validation could occur through inspecting case entry procedures 
as a case moves through a program. An auditor could examine personnel 
interactions with the data system at various stages in a process. To 
check for accuracy, auditors could choose a small sample of source 
documents and compare information in physical files with data in the 
system. Validation of programming requirements and access controls can 
be technically difficult and auditors might consult with information 
technology specialists if needed. 

[End of section] 

Appendix III: Sample Language for Reporting on Data Reliability: 

In a report's introductory paragraphs and section on objectives, scope, 
and methodology, include a statement about conformance to generally 
accepted government auditing standards. These standards include the 
appropriateness of the data being used. 

You conform to GAGAS by discussing in the report what you did to assess 
the data, any data concerns, and your judgment about the reliability of 
the data for use in the product. When data are used to answer one or 
more of the researchable questions, summarize these points in the 
introductory section of the report. 

General Examples: 

Here are four general examples. 

Example 1: 
We assessed the reliability of _______ data by (1) performing 
electronic testing of required data elements, (2) reviewing existing 
information about the data and the system that produced them, and (3) 
interviewing agency officials knowledgeable about the data. We 
determined that the data were sufficiently reliable for the purposes of 
this report. 

Example 2: 
We assessed the reliability of _______ data by (1) performing 
electronic testing of required data elements, (2) reviewing existing 
information about the data and the system that produced them, and (3) 
interviewing agency officials knowledgeable about the data. In 
addition, we traced a statistically random sample of data to source 
documents (see appendix x for details). We determined that the data 
were sufficiently reliable for the purposes of this report. 

Example 3: 
To assess the reliability of _______'s data, we (1) performed 
electronic testing for obvious errors in accuracy and completeness; (2) 
reviewed related documentation, including contractor audit reports on 
data verification; and (3) worked closely with agency officials to 
identify any data problems. When we found discrepancies (such as 
nonpopulated fields or data entry errors), we brought them to _______'s 
attention and worked with _______ to correct the discrepancies before 
conducting our analyses. We determined that the data were sufficiently 
reliable for the purposes of our report. 

Example 4: 
To assess the reliability of the data elements needed to answer the 
engagement objectives, we (1) performed electronic testing of required 
data elements, (2) reviewed related documentation, and (3) interviewed 
agency officials knowledgeable about the data. The results of our 
electronic testing showed that data elements key to our review 
contained high percentages of missing data. (See appendix x for further 
details.) Therefore, we determined that the data were not sufficiently 
reliable for the purposes of this report. To answer the research 
question, we... 

Examples Adapted from GAO Reports: 

Sufficiently Reliable: 

Here, adapted from GAO reports, are five examples of sufficiently 
reliable data, with no or few caveats. 

Example 1: 
To assess the reliability of the Federal Trade Commission's cost and 
fee collection data, we talked with agency officials about data quality 
control procedures and reviewed relevant documentation. We determined 
that the data were sufficiently reliable for the purposes of this 
report. 

Source: GAO, Telemarketing: Implementation of the National Do-Not-Call 
Registry, GAO-05-113 (Washington, D.C.: Jan. 28, 2005). 

Example 2: 
To assess the reliability of the FBI's October 2002 through May 2003 
criminal fingerprint submission data, we (1) reviewed existing 
documentation related to the data sources, (2) electronically tested 
the data to identify obvious problems with completeness or accuracy, 
and (3) interviewed knowledgeable agency officials about the data. We 
determined that the data were sufficiently reliable for the purposes of 
this report. 

Source: GAO, Law Enforcement: Information on Timeliness of Criminal 
Fingerprint Submissions to the FBI, GAO-04-260 (Washington, D.C.: Jan. 
27, 2004). 

Example 3: 
We obtained and analyzed data on the time associated with the grant 
award and distribution processes. We reviewed these data for obvious 
inconsistency errors and completeness and compared them for the five 
selected states with paper documents we obtained from these states. 
When we found discrepancies, we brought them to the attention of the 
Office for Domestic Preparedness and state and local officials and 
worked with them to correct the discrepancies before conducting our 
analyses. From these efforts, we determined that the time period data 
were sufficiently reliable for the purposes of this report. [Discussion 
of use of background data] 

...We also obtained and analyzed grant funding and expenditure data 
from selected states and local jurisdictions. Given that the grant 
funding and expenditure data are used for background purposes, we did 
not assess their reliability.[Involved some tracing to source documents 
and working with agency to resolve discrepancies] 

Source: GAO, Homeland Security: Management of First Responder Grant 
Programs Has Improved but Challenges Remain, GAO-05-121 (Washington, 
D.C.: Feb. 2, 2005). 

Example 4: 
This documentation included information on staffing requirements and 
the number of bags per hour that can be screened by in-line explosives 
detection systems, compared with stand-alone explosives detection 
systems and explosives trace detection machines. We also interviewed 
officials from the Transportation Security Administration (TSA), air 
carriers, airports, explosives detection systems equipment 
manufacturers, and airport industry associations to obtain information 
on TSA's efforts to improve checked baggage screening operations using 
explosives detection system machines. 

Although we could not independently verify the reliability of all this 
information, we compared it with other available supporting documents 
to determine data consistency and reasonableness. From these efforts, 
we believe the information we obtained is sufficiently reliable for 
this report. [Use of corroborating evidence] 

Further, we reviewed the results from unannounced, undercover, covert 
testing of checked baggage screening operations that TSA's Office of 
Internal Affairs and Program Review conducted, and we questioned TSA 
officials about the procedures used to ensure the reliability of the 
covert test data. From their answers, we believe that the covert test 
data are sufficiently reliable for the purposes of our review. 

Source: GAO, Aviation Security: Systematic Planning Needed to Optimize 
the Deployment of Checked Baggage Screening Systems, GAO-05-365 
(Washington, D.C.: Mar. 15, 2005). 

Example 5: 
We obtained online access to the DAISY, MIDAS, DODAAD, and FEDLOG 
programs, and we obtained copies of the SAMMS databases for fiscal 
years 2002 and 2003 and Government Liquidation LLC databases for June 
2001 through December 2004. For each Department of Defense (DOD) system 
and database we used in our work, we (1) obtained information from the 
system owner or manager on its data reliability procedures; (2) 
reviewed systems documentation; (3) reviewed related DOD Inspector 
General reports, Defense Logistics Agency (DLA) comptroller budget 
data, and independent public accounting firm reports related to these 
data; and (4) performed electronic testing of commodity purchase and 
excess inventory databases to identify obvious errors in accuracy and 
completeness. [Reviewed various system documentation and reports] 

We verified database control totals, where appropriate. We also 
received FEDLOG training from the Defense Logistics Information Service 
(DLIS) service provider. When we found obvious discrepancies, such as 
omitted national stock number data in the DLA commodity purchases 
databases and transaction condition coding errors in the Defense 
Reutilization and Marketing Service (DRMS) excess property systems 
data, we brought them to the attention of agency management for 
corrective action. We made appropriate adjustments to transaction data 
used in our analysis, and we disclosed data limitations with respect to 
condition coding errors and the omission of national stock number data 
that affected our analysis. [Worked with agency to resolve 
discrepancies and disclosed limitations in report] 

Our data analysis covered commodity purchases and excess commodity turn-
ins and disposal activity during fiscal years 2002 and 2003. In 
addition, we statistically tested the accuracy of excess inventory 
transactions at five Defense Reutilization and Marketing Offices (DRMO) 
and five DLA supply depots. We also reviewed summary data and selected 
reports on DRMS compliance reviews of 91 DRMOs during fiscal year 2004 
to determine the extent to which DRMS had identified problems with 
adherence to DOD and DRMS policies, made recommendations for corrective 
actions, and monitored DRMO actions to address its recommendations. 
From these procedures, we are confident that the DOD data were 
sufficiently reliable for the purposes of our analysis and findings. 
[Performed statistical testing] 

Source: GAO, DOD Excess Property: Management Control Breakdowns Result 
in Substantial Waste and Inefficiency, GAO-05-277 (Washington, D.C.: 
May 13, 2005). 

Here, adapted from GAO reports, are four examples of sufficiently 
reliable data, with caveats and specific purpose stated. 

Example 1: 
To address the staffing effort for the Coalition Provisional Authority 
(CPA), we collected and analyzed information CPA, the United States 
Agency for International Development, the Department of State, and the 
Army Corps of Engineers provided. We interviewed officials of these 
organizations as well as from the departments of Justice and Treasury. 
[Limitations noted and specific purpose for use of data stated] 

We relied primarily on staffing data from the CPA personnel office, as 
its data were the most comprehensive and it was responsible for 
processing and managing CPA personnel requirements. To assess the 
reliability of these data, we (1) interviewed the officials at CPA who 
are responsible for compiling these data and (2) performed some basic 
reasonableness checks of the data against other sources of information. 
According to CPA officials, the staffing data are only about 90 percent 
accurate because of difficulties in tracking personnel entering and 
exiting Iraq. We determined that the data from March 2004 onward were 
sufficiently reliable to make comparisons of the type of personnel 
directly supporting CPA. 

Source: GAO, Rebuilding Iraq: Resource, Security, Governance, Essential 
Services, and Oversight Issues, GAO-04-902R (Washington, D.C.: June 28, 
2004). 

Example 2: 
To obtain fiscal year 2003 expenditure data for personal protection 
equipment (PPE), we asked the U.S. Coast Guard to survey all 188 
stations and their oversight units. Each station and unit was asked to 
provide the total amount of fiscal year 2003 funds spent on PPE for 
personnel assigned to the station during the year. These totals 
included expenditures for station personnel at the group and district 
levels. [Tracing to source documents with results reported] 

To verify the accuracy of these data, we reviewed original expenditure 
documentation for a judgmentally selected sample of 29 stations. From 
this documentation, we independently quantified PPE expenditures for 
each station. Our count of total PPE purchases at the 29 stations was 9 
percent higher than the total the Coast Guard provided--our count was 4 
percent less than the Coast Guard's, after removing expenditures for 
one outlier station. Coast Guard officials attributed the difference to 
errors station personnel made when compiling the expenditure data. 
[Limitations noted and specific purpose for use of data stated] 

As a result of these differences, however, we refer to the total 
expenditure for fiscal year 2003 as an estimate. Because Coast Guard 
officials considered gathering expenditure data for fiscal year 2002 
too labor intensive for station personnel, given their current 
workloads, we used the Coast Guard's data on planned PPE expenditures 
for fiscal year 2002. After reviewing possible limitations in the PPE 
data provided us, we determined that the data were sufficiently 
reliable for the purpose of providing estimates of expenditures. 

Source: GAO, Coast Guard: Station Spending Requirements Met, but Better 
Processes Needed to Track Designated Funds, GAO-04-704 (Washington, 
D.C.: May 28, 2004). 

Example 3: 
To assess the reliability of the data on the pledges and disbursements 
international donors made, we (1) interviewed the official at the 
Department of State who is responsible for compiling these data, based 
on information provided by the government of Afghanistan, and (2) 
performed some basic reasonableness checks of the data against other 
sources of information. We determined that the data were sufficiently 
reliable for the purpose of making a broad comparison of U.S. 
contributions to those of other major donors and the combined total for 
all other donors. [Data used for broad comparisons rather than precise 
amounts, with limitations noted] 

However, we also noted several limitations in the data--notably that 
the data were largely self-reported by donor nations to the Afghan 
government and were affected by differences in exchange rates. In 
addition, donors both overreported and underreported, because of 
different definitions of disbursement. Furthermore, the data on larger 
donors are considered more reliable than the data on smaller donors, 
according to the Department of State. 

Source: GAO, Afghanistan Reconstruction: Deteriorating Security and 
Limited Resources Have Impeded Progress; Improvements in U.S. Strategy 
Needed, GAO-04-403 (Washington, D.C.: June 2, 2004). 

Example 4: 
To assess the reliability of cost data federal agencies provided on our 
questionnaire, we examined the cost information for obvious errors and 
inconsistencies, and we examined responses to the questionnaire items 
requesting information on the development of the cost data. When 
necessary, we contacted respondents to clarify responses, and we 
reviewed documentation about the cost data. Federal agencies generated 
their cost data from various sources such as their financial accounting 
systems, credit card logs, and security services contracts. [Examined 
reliability of data obtained through survey] 

This cost information is not precise and the costs are not likely to 
represent all additional costs for the Code Orange alert periods. In 
some cases, we have concerns about the reliability of the cost data 
source within particular agencies. For example, 6 of the 16 federal 
agencies reported that they extracted some of the Code Orange alert 
cost data from their financial accounting systems. As reported in the 
fiscal year 2005 President's Budget, 5 of these agencies' financial 
management performance reports had serious flaws as of December 31, 
2003. [Detailed limitations and specific purpose noted] 

Despite these limitations, we believe the cost data are sufficiently 
reliable as indicators of general ranges of cost and overall trends. 
However, they should not be used to determine the cumulative costs for 
all federal agencies for Code Orange alert periods. 

...We reported cost data that the Department of Homeland Security (DHS) 
collected from states and localities for the three Code Orange alert 
periods only to illustrate the range of costs that states reported to 
DHS for reimbursement. Cost information states submitted to DHS does 
not include all costs for states and localities during the Code Orange 
alert periods. In particular, not all states submitted costs to DHS for 
reimbursement, and it may be that not all state agencies and localities 
in states that submitted cost information reported costs to their 
states for submission to DHS. 

In addition, the cost information states submitted does not include 
additional costs for training or equipment and material purchases 
during Code Orange alert periods, because these costs are not 
reimbursable through the critical infrastructure protection grant 
programs. Moreover, some states have not finished validating costs they 
plan to submit for reimbursement. [Detailed limitations and specific 
purpose noted] 

Despite these limitations, we believe the cost data are sufficiently 
reliable as indicators of general ranges of costs that states submitted 
for reimbursement to DHS and overall trends. However, because this cost 
information from states and localities is not complete, it should not 
be used to reach conclusions about the financial effect of Code Orange 
alerts on states and localities. 

Source: GAO, Homeland Security: Communication Protocols and Risk 
Communication Principles Can Assist in Refining the Advisory System, 
GAO-04-682 (Washington, D.C.: June 25, 2004). 

Not Sufficiently Reliable: 

Here are two examples with reference to data of insufficient 
reliability for some purposes. 

Example 1: 
Staff of the Office of Records Services of the U.S. Citizenship and 
Immigration Service (USCIS) provided cost estimates for existing change 
of address processing costs and for an annual nonimmigrant alien 
address reporting requirement. We tried to obtain supporting 
explanations and documentation to verify these estimates but were not 
provided information on them all. 

On the basis of our efforts to determine the reliability of the 
estimates for which supporting information was provided--which included 
verifying calculations and bringing any discrepancies we found to 
USCIS's attention--we believe that they are sufficiently reliable for 
the purposes of this report. We did not use cost estimates for which 
supporting information was not provided. [Some data not used for lack 
of supporting information] 

Source: GAO, Alien Registration: Usefulness of a Nonimmigrant Alien 
Annual Address Reporting Requirement Is Questionable, GAO-05-204 
(Washington, D.C.: Jan. 28, 2005). 

Example 2: 
Although we did not independently verify the accuracy of the self-
reported information these agencies provided, we took a series of 
steps--from survey design through data analysis and interpretation--to 
minimize potential errors and problems. To identify potential 
questions, we spoke with numerous transportation experts, agency 
officials, and officials at organizations relevant to transportation 
planning and decision making, including the American Association of 
State Highway and Transportation Officials, the American Public 
Transportation Association, and the Association of Metropolitan 
Planning Organizations (AMPO). [Examined reliability of data obtained 
through survey] 

To verify the clarity, length of time of administration, and 
understandability of the questions, we pretested the questionnaire with 
12 transit agencies, state departments of transportation, and 
metropolitan planning organizations. We also had the questionnaire 
reviewed by a survey expert and AMPO staff. In addition, we examined 
survey responses for missing data and irregularities. We analyzed the 
survey data by calculating descriptive statistics of state 
transportation and transit agency responses.[A] 

We also surveyed state transportation departments about the analysis of 
benefits and costs of transit projects and the importance of different 
factors in decision making, for capacity-adding transit projects in 
their states. However, from the inconsistencies and irregularities of 
the survey responses, low response rate, and telephone conversations 
with survey respondents, we concluded that the information from this 
survey was not sufficiently reliable for our purposes. Therefore, we 
did not use the information from this survey in our analysis or include 
it in the report. [Some data not used because of problems found; 
explicit statement that did not use] 

Source: GAO, Highway and Transit Investments: Options for Improving 
Information on Projects' Benefits and Costs and Increasing 
Accountability for Results, GAO-05-172 (Washington, D.C.: Jan. 24, 
2005). 

Here are three examples of data of insufficient reliability leading to 
agency changes or recommendations. 

Example 1: 
To assess the reliability of [early and late] release data, we reviewed 
the process by which the District of Columbia Department of Corrections 
tracks these data and the extent to which each relevant data element is 
complete and accurate. To do this, we interviewed department staff 
about the processes used to capture early and late release errors, the 
controls over those processes, and the data elements involved. For late 
release errors, we also traced data to their corresponding source 
documents. 

We identified inconsistencies in the information, prompting the 
department to review its methodology for identifying late releases. 
This review led it and us to conclude that its methodology had been 
incomplete and had produced an undercount of the true number of late 
releases. The department modified its methodology in April 2004 to be 
more comprehensive. 

Because the department did not have complete data on early and late 
inmate releases, it does not know the full extent to which they 
occurred and may not discover an early release error until long after 
an inmate has been released. With respect to late releases, the 
department used an incomplete methodology and, therefore, may have 
understated the actual number of late releases. During our review, the 
department modified the methodology to more accurately identify the 
number of late releases. [Data problems found during review led to a 
statement of possible effect and modification by agency] 

Source: GAO, District of Columbia Jail: Management Challenges Exist in 
Improving Facility Conditions, GAO-04-742 (Washington, D.C.: Aug. 27, 
2004). 

Example 2: 

From Results in Brief: Our review of prospective ruling request cases 
showed that the Legal Case Inventory System (LCIS), the Office of 
Regulation and Rulings' (OR&R) automated database, continued to face 
data reliability challenges potentially hindering its effectiveness as 
a tool for tracking and monitoring the progress and history of cases 
and measuring timeliness. For example, our comparison of LCIS data to 
case files showed that 88 of the 325 cases we reviewed were 
inaccurately coded as rulings in LCIS. [Because of the quality of the 
data, database reliability became a reporting objective] 

In response to recommendations we made in our September 2000 report, 
and to data errors we found in this review, OR&R has taken corrective 
actions to improve the accuracy and reliability of LCIS data, such as 
developing uniform procedures for recording cases in LCIS. However, 
they may not resolve the LCIS data reliability challenges. Although the 
corrective actions include goals such as correctly coding cases and 
entering timely and accurate information into the database, some of the 
actions lack specific procedures for effective implementation. For 
example, OR&R did not provide specific guidance as to how, when, and by 
whom information letters are to be coded. This report contains a 
recommendation to the OR&R Assistant Commissioner regarding continued 
assessment of LCIS data reliability to determine whether the corrective 
actions are sufficient. 

From Objectives, Scope, and Methodology: To determine whether OR&R 
resolved the data reliability challenges it faced with LCIS, we 
interviewed OR&R management officials, reviewed case file information 
for our sample of 325 OR&R headquarters cases categorized in LCIS as 
prospective rulings, and collected and reviewed other available 
information. This information included the July 2002 Standard Operating 
Procedure, intended to ensure a consistent process for receiving, 
acknowledging, assigning, recording, tracking, updating, signing, and 
closing ruling cases in LCIS. [Data collection included a file review 
of randomly selected cases, with comparison to the database and review 
of documents such as standard operating procedures] 

In reviewing OR&R's case files for our sample of cases and noting 
discrepancies with LCIS data for "type of case code," "case category 
code," "date assigned," and "date closed," we did not discuss each case 
with OR&R officials to determine the reasons that case file data did 
not match LCIS data or data were missing from case files. To do so 
would have been time consuming and complex, for us as well as OR&R, 
with little likelihood of determining the reason for each discrepancy. 
In carrying out the work for our September 2000 report on OR&R 
headquarters rulings, we asked OR&R officials to explain the reasons 
for discrepancies. However, we reported that we could not always 
identify the reasons why LCIS data were inaccurate for the cases we 
reviewed. 

From Recommendations: To help ensure that LCIS data are accurate and 
that OR&R can reliably use the database as a management tool to record 
and monitor prospective rulings and measure timeliness, we recommend 
that the OR&R Assistant Commissioner take steps to continue to assess 
LCIS data reliability to determine whether recent improvements 
sufficiently correct past problems. 

Source: GAO, U.S. Customs Service: Prospective Rulings More Timely, but 
Database Reliability Questions Remain, GAO-03-828 (Washington, D.C.: 
Aug. 6, 2003). 

Example 3: 
In our effort to examine General Services Administration's (GSA) FAIRS 
systems, we reviewed the extent and quality of controls over federal 
aircraft data. In doing so, we sought to determine whether (1) GSA had 
management controls in place to provide reasonable assurance that the 
FAIRS data included in its report were valid and reliable and (2) FAIRS 
data were sufficiently reliable for our intended use. 

We identified and evaluated GSA's management controls over the 
processes to collect, analyze, and report costs, use, and numbers of 
government aircraft. We did not audit the data that agencies submit to 
FAIRS, and we did not audit the data produced by FAIRS or the 
information GSA included in its annual reports. We conducted background 
research and site visits, interviewed GSA officials, and collected and 
reviewed documentation on GSA and FAIRS to gain an understanding of 
GSA's operations and FAIRS processes, its inherent and control risk 
factors, and existing management controls. We documented our 
understanding of the processing of aircraft inventory, cost, and use 
data in FAIRS, and the identified internal controls in a process flow 
chart. For each relevant process identified, we assessed the overall 
effectiveness of existing controls by conducting a walk-through of the 
system and performing control testing--physical observation of how 
controls actually operated. [Engagement involved review of internal 
controls for system, as well as reliability of information in the 
database] 

Further, we evaluated the results of our analyses and testing to 
conclude whether GSA management controls provide reasonable assurance 
that the FAIRS data included in GSA's annual report are valid and 
reliable. We found that information in the database was not 
sufficiently reliable to accurately determine the composition and cost 
of federal aircraft programs. However, we used the information to 
provide descriptive and summary statistics (in app. II). As a result, 
we developed recommendations for improving or establishing management 
controls to help ensure FAIRS data quality. 

From Recommendations: To improve the completeness and accuracy of the 
FAIRS database so that it captures all aircraft program costs and is 
useful for conducting detailed analyses of the condition and 
performance of the federal aircraft fleet, we are making the following 
three recommendations to the Administrator of GSA: 

* Clarify existing FAIRS guidance to agencies to identify the cost 
elements that all aircraft programs should report to the FAIRS system, 
make the reporting of those elements mandatory, and develop a mechanism 
to ensure that agencies comply with reporting requirements; 

* Expand existing FAIRS guidance to require that programs report 
additional aviation costs associated with acquiring aircraft, not 
currently required; this would provide more complete and accurate data 
on the composition and cost of the federal aircraft fleet and, thus, 
enhance GSA's annual report on federal aircraft operations. At a 
minimum, agencies should be required to report acquisition, financing, 
and self-insurance costs; 

* Test the FAIRS database periodically to ensure that existing systems 
controls are working as designed and work with the Interagency 
Committee for Aviation Policy to identify, develop, and implement 
additional controls as necessary. 

Source: GAO, Federal Aircraft: Inaccurate Cost Data and Weaknesses on 
Fleet Management Planning Hamper Cost Effective Operations, GAO-04-645 
(Washington, D.C.: June 18, 2004). 

[End of section] 

Footnotes: 

[1] Comptroller General of the United States, Government Auditing 
Standards: July 2007 Revision, [hyperlink, 
http://www.gao.gov/products/GAO-07-731G] (Washington, D.C.: Government 
Accountability Office, July 2007), and GAO, Assessing the Reliability 
of Computer-Processed Data, [hyperlink, 
http://www.gao.gov/products/GAO-03-273G] (Washington, D.C.: October 
2002). 

[2] Comptroller General of the United States, Government Auditing 
Standards, section 7.23-27, pp. 134-37, and section 7.65, p. 151. 

[3] See GAO and President's Council on Integrity and Efficiency, 
Financial Audit Manual, vol. 1, [hyperlink, 
http://www.gao.gov/products/GAO-08-585G] (Washington, D.C.: July 2008), 
vol. 2, GAO-08-586G (Washington, D.C.: July 2008), and vol. 3, 
[hyperlink, http://www.gao.gov/products/GAO-07-1173G] (Washington, 
D.C.: Aug. 28, 2007), and GAO, Federal Information System Controls 
Audit Manual, [hyperlink, 
http://www.gao.gov/products/GAO-09-232G](Washington, D.C.: February 
2009). 

[4] General controls refers to the structure, policies, and procedures--
in all or a large segment of an organization's information systems-- 
that help ensure proper operation, data integrity, and security. 
Application controls refers to the structure, policies, and procedures 
that apply to individual application systems, such as inventory or 
payroll. 

[5] Guidance for reviewing general and application controls is in GAO, 
Federal Information System Controls Audit Manual. 

[6] See Federal Managers' Financial Integrity Act of 1982, Pub. L. 97- 
255, Sept. 8, 1982, 96 Stat. 814, 31 U.S.C. � 3512; Clinger-Cohen Act 
of 1996, Pub. L. 104-106, divs. D, E, Feb. 10, 1996, 110 Stat. 642, 
679, 40 U.S.C. � 1401 et seq.; Government Performance and Results Act 
of 1993, Pub. L. 103-62, Aug. 3, 1993, 107 Stat. 285, 31 U.S.C. � 1101; 
and Federal Information Security Management Act of 2002, 44 U.S.C. � 
3541 et seq. 

[7] An in-depth discussion of quality assurance practices to be used in 
electronic testing and analyses is beyond the scope of this guide. It 
is nonetheless important to perform appropriate checks to ensure that 
you have obtained the correct file. All too often, auditors receive an 
incorrect file (an early version or an incomplete file). Appropriate 
steps include counting records and comparing totals with the 
responsible agency or source. 

[8] For more information about system controls, and how specific 
controls contribute to internal control and the reliability of computer 
processed data, see GAO, Standards for Internal Control in the Federal 
Government, [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] 
(Washington, D.C.: November 1999), and Internal Control Management and 
Evaluation Tool, [hyperlink, http://www.gao.gov/products/GAO-01-1008G] 
(Washington, D.C.: August 2001). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO�s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO�s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: