Formats

• Executive Summary (PDF) 1.0 MB  


• Help with Viewers, Players, and Plug-ins

Table of Contents

• Front Matter
• User's Guide to Registries Evaluating Patient Outcomes: Summary
• Overview
• Evaluating Registries
• Tables

Front Matter

The Effective Health Care Program of the Agency for Healthcare Research and Quality (AHRQ) conducts and supports research focused on the outcomes, effectiveness, comparative clinical effectiveness, and appropriateness of pharmaceuticals, devices, and health care services. More information on the Effective Health Care Program and electronic copies of this report can be found at www.effectivehealthcare.ahrq.gov.

This report is the summary of Registries for Evaluating Patient Outcomes: A User's Guide, which was produced under contract to AHRQ as a collaborative effort between two centers involved in the Effective Health Care Program: The Outcome DEcIDE Center (Developing Evidence to Inform Decisions about Effectiveness) under Contract No. HHSA29020050035I TO1, and the Duke EPC (Evidence-based Practice Center) under Contract No. 290-02-0025 TO1. The AHRQ Task Order Officer for this project was Elise Berliner, Ph.D. The findings and conclusions in this summary and the larger document are those of the authors, who are responsible for the contents; the findings and conclusions do not necessarily represent the views of AHRQ or the U.S. Department of Health and Human Services. Therefore, no statement in this report should be construed as an official position of AHRQ or the U.S. Department of Health and Human Services.

Copyright Information

Registries for Evaluating Patient Outcomes: A User's Guide and User's Guide to Registries Evaluating Patient Outcomes: Summary are both copyrighted by the Agency for Healthcare Research and Quality (AHRQ). The products and their contents may be used and incorporated into other materials on the condition that the contents are not changed in any way (including covers and front matter) and that no fee is charged by the reproducer of the products or their contents for their use. The products may not be sold for profit or incorporated into any profit-making venture without the expressed written permission of AHRQ.

Specifically:

1. When one of the documents is reprinted, it must be reprinted in its entirety without any changes.
2. When parts of the documents are used or quoted, the following citation should be used.

 

*Note: Registries for Evaluating Patient Outcomes: A User's Guide contains material copyrighted by others. For material noted as copyrighted by others, the user must obtain permission from the copyright holders identified in the document. This is a summary of a longer, more comprehensive report.

Citation for full report should be used: Gliklich RE, Dreyer NA, eds. Registries for Evaluating Patient Outcomes: A User's Guide. (Prepared by Outcome DEcIDE Center [Outcome Sciences, Inc. dba Outcome] under Contract No. HHSA29020050035I TO1.)

AHRQ Publication No. 07-EHC001-1. Rockville, MD: Agency for Healthcare Research and Quality. April 2007.

Full Report: AHRQ Publication Number 07-EHC001-1

Summary Report: AHRQ Publication Number 07-EHC001-2

 

User's Guide to Registries Evaluating Patient Outcomes: Summary

Patient registries are organized systems that collect data for scientific, clinical, or policy purposes. Registries are a valuable complement to randomized controlled trials in determining real-world outcomes in the practice of medicine. They do not generally have restrictive inclusion or exclusion criteria, nor do they specify what therapy the health care provider must adhere to. They can be used to evaluate outcomes for diverse purposes ranging from the natural history of a disease, to the safety of drugs or devices, to the real-world effectiveness of therapies.

The Effective Health Care Program of the Agency for Healthcare Research and Quality (AHRQ) conducts and supports research focused on the outcomes, effectiveness, comparative clinical effectiveness, and appropriateness of pharmaceuticals, devices, and health care services. As part of the Effective Health Care Program, Outcome Sciences, Inc., a DEcIDE (Developing Evidence to Inform Decisions about Effectiveness) center, and the Duke EPC (Evidence-based Practice Center) collaborated in a study of registries and the many elements involved in creating a registry.

Outcome Sciences, with the assistance of the Duke EPC, coordinated a group of 39 contributors and 35 reviewers to develop a handbook to serve as a guide to the design, implementation, analysis, interpretation, and evaluation of the quality of a registry for understanding patient outcomes.

This is a summary of the full handbook. The "Overview" presents basic information on the main areas to consider in setting up a registry, from selection of data elements and protection of patient privacy to analyzing results and publishing findings. "Evaluating Registries" outlines elements of quality to be considered in setting up or evaluating a registry. Basic good registry practices are given, as well as future directions for practices that could enhance scientific rigor but may not be practical for every registry.

The full report is available online at www.effectivehealthcare.ahrq.gov. Free print copies are available by calling 800-358-9295. Ask for Registries for Evaluating Patient Outcomes: A User's Guide, AHRQ Publication Number 07-EHC001-1.

Overview

For the purpose of this document, a patient registry is an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure, and that serves a predetermined scientific, clinical, or policy purpose(s). The registry database is the file (or files) derived from the registry.

Although registries can serve many purposes, the focus here is on registries that are created for one or more of the following purposes: to describe the natural history of disease, to determine clinical effectiveness or cost effectiveness of health care products and services, to measure or monitor safety and harm, and/or to measure quality of care. Registries are classified according to how the populations are defined. For example, product registries include patients who have been exposed to biopharmaceutical products or medical devices. Health services registries consist of patients who have had a common procedure, clinical encounter, or hospitalization. Disease or condition registries are defined by patients having the same diagnosis, such as cystic fibrosis or heart failure.

Planning

There are several key steps in planning a patient registry, including articulating the purpose of the registry, determining whether the registry is an appropriate means for addressing the research question, identifying stakeholders, defining the scope and target population, assessing feasibility, and securing funding. The registry team and advisors should be selected based on expertise and experience. The plan for registry governance and oversight should clearly address such issues as overall direction and operations, scientific content, ethics, safety, data access, publications, and change management. It is also helpful to plan for the entire lifespan of a registry, including how and when the registry will end and any plans for transitioning the registry at that time.

Registry Design

A patient registry should be designed with respect to its major purpose, with the understanding that different levels of rigor may be required for registries that are designed to address focused analytical questions to support decisionmaking, in contrast to those intended primarily for descriptive purposes. The key points to consider in designing a registry include formulating a research question; choosing a study design; translating questions of clinical interest into measurable exposures and outcomes; choosing patients for study, including deciding whether a comparison group is needed; determining where data can be found; and deciding how many patients need to be studied and for how long. Once these key design items have been determined, the registry design should be reviewed to evaluate potential sources of bias (systematic error); these should be addressed to the extent that is practical and achievable.

The specific research questions of interest will guide the registry design, identification of exposures and outcomes, and definitions of the target population (the population to which the findings are meant to apply). The registry population should be designed to approximate the characteristics of the target population as much as possible. The number of study subjects desired and length of observation (followup) should be planned in accordance with the overall purpose of the registry. The desired study size (in terms of subjects or person-years of observation) is determined by specifying the magnitude of an expected clinically meaningful effect or the desired precision of effect estimates. Study size determinants are also affected by practicality, cost, and whether or not the registry is intended to support regulatory decisionmaking. Depending on the purpose of the registry, internal, external, or historical comparison groups strengthen the understanding of whether the observed effects are indeed real and, in fact, different from what would have occurred under other circumstances.

Registry study designs often restrict eligibility for entry to individuals with certain characteristics to assure that they will assemble enough information for analysis (e.g., age restriction) or use some form of sampling-random selection, systematic sampling, or a haphazard, nonrandom approach. The potential for bias refers to opportunities for systematic errors to influence the results. The information value of a registry is enhanced by its ability to provide an assessment of the potential for bias and to quantify how this bias could affect the study results.

Data Elements

The selection of data elements requires balancing such factors as their importance for the integrity of the registry and for the analysis of primary outcomes, their reliability, their contribution to the overall burden for respondents, and the incremental costs associated with their collection. Selection begins with identifying relevant domains. Specific data elements then are selected with consideration for established clinical data standards, common data definitions, and the use of patient identifiers. It is important to determine which elements are absolutely necessary and which are desirable but not essential. In choosing measurement scales for assessing patient-reported outcomes, it is preferable to use scales that have been appropriately validated, when such tools exist. Once data elements have been selected, a data map should be created, and the data collection tools should be pilot tested. Testing allows assessment of respondent burden, accuracy, and completeness of questions, and potential areas for missing data. Inter-rater agreement for data collection instruments can also be assessed, especially in registries that rely on chart abstraction. Overall, choice of data elements should be guided by parsimony, validity, and a focus on achieving the registry's purpose.

Data Sources

A single registry may integrate data from various sources. The form, structure, availability, and timeliness of the required data are important considerations. Data sources can be classified as primary or secondary. Primary data are collected for direct purposes of the registry. Secondary data are comprised of information that has been collected for purposes other than the registry, and they may not be uniformly structured or validated with the same rigor as primary data. Sufficient identifiers are necessary to guarantee an accurate match between secondary sources and registry patients. Furthermore, a solid understanding of the original purpose of the secondary data and how they were collected is advised, because the way that those data were collected and verified or validated will help shape their use in a registry. Common secondary sources of data linked to registries include medical records systems, institutional or organizational databases, administrative health insurance claims data, death and birth records, census databases, and related existing registry databases.

Ethics, Data Ownership, and Privacy

Critical ethical and legal considerations should guide the development and use of patient registries. The Common Rule is the uniform set of regulations on the ethical conduct of human subjects research from the Federal agencies that fund such research. Institutions that conduct research agree to comply with the Common Rule for federally funded research and may opt to apply that rule to all human subjects activities conducted within their facilities or by their employees and agents, regardless of the source of funding. The Health Insurance Portability and Accountability Act of 1996 (HIPAA) and its implementing regulations (collectively, the Privacy Rule) are the legal protections for the privacy of individually identifiable health information created and maintained by health care providers, health plans, and health care clearinghouses (called "covered entities"). The research purpose of a registry, the status of its developer, and the extent to which registry data are individually identifiable largely determine applicable regulatory requirements. Other important ethical and legal concerns include transparency of activities, oversight, and data ownership.

Patient and Provider Recruitment and Management

Recruitment and retention of providers (as registry sites) and patients are essential to the success of a registry. Recruitment typically occurs at several levels, including facilities (hospital, practice, pharmacy), providers, and patients. The motivating factors for participation at each level and the factors necessary to achieve retention differ according to the registry. Factors that motivate participation include the perceived relevance, importance, or scientific credibility of the registry, as well as the risks and burdens of participation and any incentives for participation. Because provider and patient recruitment and retention can affect how well a registry accurately represents the target population, well-planned strategies for enrollment and retention are critical. Goals for recruitment, retention, and followup should be explicitly laid out in the registry planning phase, and deviations during the conduct of the registry should be continuously evaluated for their risk of introducing bias.

Data Collection and Quality Assurance

The integrated system for collecting, cleaning, storing, monitoring, reviewing, and reporting on registry data determines the utility of those data for meeting the registry's goals. A broad range of data collection procedures and systems are available. Some are more suitable than others for particular purposes. Critical factors in the ultimate quality of the data include how data elements are structured and defined, how personnel are trained, and how data problems are handled (e.g., missing, out-of-range, or logically inconsistent values). Registries may also be required to conform to guidelines or standards of specific end users of the data (e.g., 21 Code of Federal Regulations, Part 11). Quality assurance aims to affirm that the data were, in fact, collected in accordance with established procedures and that they meet the requisite standards of quality to accomplish the registry's intended purposes and the intended use of the data.

Requirements for quality assurance should be defined during the registry's inception and creation. Because certain requirements may have significant cost implications, a risk-based approach to developing a quality assurance plan is recommended. It should be based on identifying the most important or likely sources of error or potential lapses in procedures that may impact the quality of the registry in the context of its intended purpose.

Adverse Event Detection, Processing, and Reporting

The U.S. Food and Drug Administration defines an adverse event (AE) as any untoward medical occurrence in a patient administered a pharmaceutical product, whether or not related to or considered to have a causal relationship with the treatment. AEs are categorized according to the seriousness and, for drugs, the expectedness of the event. Although AE reporting for all marketed products is dependent on the principle of "becoming aware," collection of adverse event data falls into two categories: those events that are intentionally solicited (meaning data that are part of the uniform collection of information in the registry) and those that are unsolicited (meaning that the AE is volunteered or noted in an unsolicited manner).

Determining whether the registry should use a case report form to collect AEs should be based on the scientific importance of the information for evaluating the specified outcomes of interest. Regardless of whether or not AEs constitute outcomes for the registry, it is important to develop a plan for detecting, processing, and reporting AEs for any registry that has direct patient interaction. If the registry receives sponsorship, in whole or in part, from a regulated industry (for drugs or devices), the sponsor has mandated reporting requirements, and the process for detecting and reporting AEs should be established and registry personnel trained on how to identify AEs and to whom they should be reported. Sponsors of registries designed specifically to meet requirements for surveillance of drug or device safety are encouraged to hold discussions with health authorities about the most appropriate process for reporting serious AEs.

Analysis and Interpretation

Analysis and interpretation of registry data begin with answering a series of core questions. Who was studied? How were the data collected, edited, and verified, and how were missing data handled? How were the analyses performed? Four populations are of interest in describing who was studied: the target population, the accessible population, the intended population, and the population actually studied (the "actual population"). The representativeness of the actual population to the target population is referred to as generalizability.

Analysis of registry outcomes first requires an analysis of the completeness of data collection and data quality. Considerations include an evaluation of completeness for most if not all important covariates and an understanding of how missing data were handled and reported. Analysis of a registry should provide information on the characteristics of the patient population, the exposures of interest, and the endpoints. Descriptive registry studies focus on describing frequency and patterns of various elements in a patient population, whereas analytical studies concentrate on associations between patients or treatment characteristics and health outcomes of interest. A statistical analysis plan describes the analytical plans and statistical techniques that will be used to evaluate the primary and secondary objectives specified in the study plan. Interpretation of registry data should be provided so that the conclusions can be understood in the appropriate context and so that any lessons from the registry can be applied to the target population and used to improve patient care and outcomes.

Evaluating Registries

Although registries can provide useful information, there are levels of rigor that enhance validity and make the information from some registries more useful for guiding decisions than the information from others. The term "quality" can be applied to registries to describe the confidence that the design, conduct, and analysis of the registry can be shown to protect against bias and errors in inference-that is, erroneous conclusions drawn from a registry. Although there are limitations in any assessment of quality, this handbook uses a quality component analysis to evaluate high-level factors that may affect results and differentiates between research quality (which pertains to the scientific process) and evidence quality (which pertains to the data/findings emanating from the research process). Quality components are classified as either "basic elements of good practice," which can be viewed as a basic checklist that should be considered for all patient registries, or as "potential enhancements to good practice" that may strengthen the information value in particular circumstances. The results of such an evaluation should be considered in the context of the disease area(s), the type of registry, and the purpose of the registry, and should also take into account feasibility and affordability.

Registries are undertaken for many purposes, ranging from descriptive studies intended to contribute to scientific understanding of patient outcomes to studies used to inform policy decisions. Some are undertaken with great urgency, whereas others proceed with more deliberation. Budgetary support ranges from spartan to adequate. Most importantly, registries often serve multiple purposes and change over time to accommodate these various purposes-in fact, these are hallmarks of registries. Although all registries can provide useful information, there are levels of rigor that enhance validity and make the information from some registries more useful for guiding decisions than others.

To date, no standards have been developed by which to guide evaluation of registries, and the research into quality aspects of registries has been sparse. This is an overview of key components of the design, execution, and analysis of a registry that promote reliability and validity of data on patient outcomes.

The aim here is to provide a simple and user-friendly system that allows registries to be described and evaluated in the context of the purpose for which they are conducted. Information is presented to help distinguish between:

• Basic good registry practices that are desirable to meet certain purposes.
• Future directions for practices that could enhance scientific rigor but may not be achievable because of practical constraints.

The items listed as "basic elements of good practice" are applicable to all patient registries. While it may not be practical or feasible to achieve all of the basic elements of good practice, it is useful to consider these characteristics in planning and evaluating registries. The information described in this handbook, and particularly in this chapter, is also designed to be used in reporting registry study results, much as CONSORT guidelines have been used to improve reporting of clinical trials.

Defining Quality

A definition of "quality" for registers was adapted from a definition developed for randomized controlled trials; the term is used to refer to the confidence that the design, conduct, and analysis of the trial or registry can be shown to protect against bias (systematic error) and errors in inference-that is, erroneous conclusions drawn from a study. As used here, quality refers both to the data and to the conclusions drawn from analyses of these data. For more information about bias, validity, and inference, readers are encouraged to consult the handbook and epidemiologic textbooks.

Measuring Quality

There are two major difficulties with assessing quality in registries:

• It can often be difficult to differentiate between the quality of the design, the study conduct, and the information available.
• There is a lack of empirical evidence for evaluating parameters purported to indicate quality and impact on the evidence produced from registries.

In addition, registries vary widely in methodology, scope, and objectives, and therefore attributes that are important in one scenario may be less important in another. Furthermore, registries may be very useful vehicles for providing clinically relevant real-world information, even when they meet relatively few of the basic elements of good practice (typically because of budgetary limitations). In many cases, some data are better than no data, and even registries that fall short of including all the basic elements of good registry practice may still provide valuable insights about real-world medical and consumer practices and disease etiology. Evaluations of the quality of any registry must therefore be done with respect to the context-specific purpose of the registry, must take into account both the internal and external validity of the data, and should be tempered by considerations of cost and feasibility.

The most commonly used method to assess quality of studies is a quality scale; there are numerous quality scales of varying length and complexity in existence, with strong views being expressed both for and against their use. Different scales emphasize distinctive dimensions of quality and therefore can produce disparate results when applied to a given study. In most situations, a summary score is derived by adding individual item scores, with or without weighting. This method, however, ignores whether the various items may lead to a bias toward the null (suggesting the erroneous interpretation that there is no effect) or tend to exaggerate the appearance of an effect when none really exists, and the final score produced does not reflect individual components.

Rather than develop a checklist, the approach suggested here is to undertake a quality component analysis, an investigation of the components that may affect the results obtained. In the quality component analysis, a differentiation is made between two domains: research quality, which pertains to the scientific process (in this instance, the design and operational aspects of the registry), and evidence quality, which relates to the data/findings emanating from the research process. According to Lohr, "The level of confidence one might have in evidence turns on the underlying robustness of the research and the analysis done to synthesize that research."^*

Several key elements identified in previous research studies were reviewed in selecting the quality components for analysis presented here.

The results of the quality component analysis must be considered in conjunction with context-specific substantive components that relate to the disease area, the type of registry, and the purpose of the registry. (See Table 1.) For example, a disease-specific registry that has been designed to look at natural history should not be deemed low quality simply because it is not large enough to detect rare treatment effects.

Quality Domains

For research, the quality domains are planning; design; data elements and data sources; and ethics, privacy, and governance. For evidence, the quality domains are described separately for registry participants; data elements and data sources; data quality assurance; analysis; and reporting.

Table 2 shows the basic elements of good registry practice for research, and Table 3 shows additional practices that have the potential to enhance scientific rigor, and thus the validity and reliability of information resulting from registries. Similarly, Table 4 shows the basic elements of good registry practice for evidence, and Table 5 shows additional practices that may enhance the evidence quality. It is important to weigh efforts taken to promote the accuracy and completeness of evidence in balance with the public health urgency of a problem, the types of interventions that are available, and the risks to public health from coming to a wrong conclusion. These lists of components are most likely incomplete, but the level of detail provided should be useful for high-level quality distinctions.

Most importantly, the basic elements of good practice, as well as the potential enhancements to good practice, depend to a great extent on the resources and budget available to support registry-based research.

---

^* Lohr KN. Rating the strength of evidence: relevance for quality improvement programs. Int J. Qual Health Care 2004;16(1):9-18.

Tables

Table 1: Overview of Registry Purposes

Determining clinical effectiveness, cost effectiveness, or comparative effectiveness of a test or treatment, including evaluating the acceptability of drugs, devices, or procedures for reimbursement. Measuring or monitoring safety and harm of specific products and treatments, including comparative evaluation of safety and effectiveness. Measuring or improving quality of care, including conducting programs to measure and/or improve the practice of medicine and/or public health. Assessing natural history, including estimating the magnitude of a problem; determining the underlying incidence or prevalence rate; examining trends of disease over time, conducting surveillance; assessing service delivery and identifying groups at high risk; documenting the types of patients served by a health provider; and describing and estimating survival.

Table 2: Research Quality for Registries—Basic Elements of Good Practice
Planning	Sufficient thought has been given to identifying and capturing all the necessary aspects that are feasible to collect from the outset. A written registry plan documents the goals; design; target population; methods for data collection, including patient recruitment; data elements and data sources; a high-level data management plan; plans for protecting human subjects and for data review for quality; and a high-level analysis plan that contains sufficient detail to explain the main focus and proposed methods of analysis. The process for identifying serious events is described and a plan is created for reporting, as appropriate and consistent with regulatory requirements. A plan for communication of study results is addressed. Appropriate personnel and facilities are available, including facilities for secure storage of data. A process is established for documenting subsequent modifications to the registry plan.
Design	The literature has been reviewed to guide appropriate data collection. The target population is described, including plans to recruit study subjects. Specific eligibility, inclusion, and exclusion criteria are specified. The size required to detect an effect, should one exist, or achieve a desired level of precision is specified, whether or not the sample size requirement is met. The followup time required to detect events of interest is specified, whether or not it is feasible to achieve; however, the followup time planned is adequate to address the main objective. Plans are made for how the analysis will be conducted, including what comparative information, if any, will be used to support study hypotheses or objectives.
Data elements and data sources	Outcomes are clinically meaningful and relevant in that the information is useful to the medical community for decisionmaking. Operational definitions of outcomes are clearly defined. Important exposures, risk factors, and mitigating (or protective) factors are identified and collected to the extent feasible. The individual(s) responsible for the integrity of the data, computerized and hard copy, are identified; it is determined that they have the training and experience to perform the assigned tasks. Data collectors are trained using standard techniques. A data and coding dictionary is maintained to provide explicit definitions and describe coding used. A quality assurance plan has been created and addresses data editing and verification, as appropriate.
Ethics, privacy, and governance	The issues of protection of human subjects—including privacy, informed consent, data security, and study ethics—have been carefully considered and addressed in accordance with local, national, and international regulations. The registry has received review by any required oversight committees (e.g., ethics committee, privacy committee, or institutional review board, as applicable).

Table 3: Research Quality for Registries—Potential Enhancements to Good Practice
Planning	A formal protocol covers all the topics listed as basic elements of a study plan, covering some elements in depth. The protocol also includes objectives or hypotheses; governance, privacy, and ethics; plans for data entry; and reporting of study results. It may be helpful for stakeholders to have input in reviewing the protocol before it is finalized to assure clinical relevance and feasibility. The protocol includes a plan for training registry and site personnel about how to identify and report serious events that occur during the observation period and that could be causally related to the product or process under study, as appropriate. An advisory board has been established. A feasibility study or pilot test may be useful in certain situations, such as when studying hard-to-reach populations, when sensitive data are sought, and when critical registry methods are new or have not otherwise been tested. Feasibility assessment may include evaluation of factors such as means and likelihood of recruiting appropriate patients, as well as establishing and fine-tuning what data will be collected and the methods for data collection. A plan for quality assurance is described in the protocol. The sampling process is part of a risk-based strategy that focuses on detecting and quantifying the most likely causes of error and the types of error that are most likely to impact the registry purpose. For example, a registry might compare a random sample of patient data (e.g. 5 percent to 20 percent of patients and specific data variables) with patient charts or with a sample of registry sites based on “for-cause” reasons, or a combination of these approaches. The plan for generating and/or reviewing publications and presentations is defined. It includes review by knowledgeable parties. Plans for timely dissemination of information and a process for others to access the data are considered.
Design	Use of concurrent comparators may offer an advantage over historical or external comparison groups in situations where treatments are evolving rapidly. The methods of data collection do not limit site participation such that the representativeness of site selection is compromised. While single methods of data collection to a centralized database (e.g., via Web) are most efficient, a single method may not suit all registries. Multiple methods of data collection may be required for some purposes (e.g., where access to computers or Internet is limited). Formal statistical calculations may be used to specify the size of the registry (number of patients or patient-years of observation) needed to measure an effect with a certain level of precision or to meet a specified statistical power to detect an effect, should one exist, whether or not the desired size is achievable within the practical study constraints. Precision and power considerations must be balanced against budgetary and feasibility constraints, and should not be used as a reason to avoid conducting research in areas where little exists.
Data elements and data sources	Whenever possible, coding used is consistent with nationally approved coding systems to promote comparability of information among studies. Standardized data dictionaries, such as the ICD-9 (International Classification of Diseases, 9th Revision), are used where applicable. It is preferable to use scales and tests that have been validated when such tools exist for the purpose needed. Rigor can be enhanced by external validation for a sample of data and/or data review by an adjudication committee for complex conditions or endpoints for which established procedures and/or coding are not used. To reduce losses to followup, safety studies can be enhanced by collecting enough information on individual identifiers to permit linkage with external databases such as the National Death Index where such databases exist, as appropriate. However, the desire for long-term followup should be balanced by considerations relating to the challenges posed by collecting individually identifiable data (as opposed to “de-identified data”), especially with regard to institutional review policies. Levels of quality assurance activities may be adapted based on observed performance. For example, they would be increased for sites that appear to be having difficulty in study conduct or data entry.
Ethics, privacy, and governance	Potential conflicts of interest are considered and managed appropriately. Plans for timely review and dissemination of results are established at the outset. Publication policies are specified in advance of collecting data. Publishing results in the peer-reviewed literature is a desirable means of introducing information into the public domain.

Table 4: Evidence Quality for Registries—Basic Elements of Good Practice
Registry participants	Registry participants are similar to the target population, and attention has been paid to minimizing selection bias to the extent feasible. Eligibility (in terms of inclusion and exclusion criteria) is confirmed upon patient enrollment. For safety studies, personnel are appropriately trained to ask about complaints or adverse events in a manner that is clear and specific (e.g., solicited vs. unsolicited) and to know how information should be reported to manufacturers and health authorities. Completeness of information on eligible patients has been evaluated and described.
Data elements and data sources	Information has been collected for relevant key exposures, risk factors, and mitigating or protective factors. Patient outcomes are clinically relevant (in terms of information that will assist medical professionals with decisionmaking) and clearly defined. Definitions are provided, especially for complex conditions or outcomes that may not have uniformly established criteria (e.g., specify how an “injection site reaction” is operationally defined). The followup period is reasonably sufficient to capture the main outcomes of interest.
Data quality assurance	Data are reasonably complete. Reasonable efforts have been expended to assure that appropriate patients have been systematically enrolled and followed in as unbiased a manner as possible. Reasonable efforts have been devoted to minimize losses to followup. Data checks are employed using range and consistency checks.
Analysis	Accepted analytic techniques are used; these may be augmented by new or novel approaches. The role and impact of missing data and potential confounding factors have been explored.
Reporting	A report describes the methods, including target population and selection of study subjects, compliance with applicable regulatory rules and regulations, data collection methods, any transformation of variables and/or construction of composite endpoints, statistical methods used for data analysis, and a description of any circumstances that may have affected the quality or integrity of the data. Results are reported for all the main objectives. Followup time is described so that readers can assess the impact of the observation period on the conclusions drawn. The report includes a clear statement of any conclusions drawn from the analysis of the registry’s primary and secondary objectives and any implications of study results, as appropriate. All authors who are acknowledged have had a meaningful role in the design, conduct, analysis, or interpretation of results.

Table 5: Evidence Quality for Registries—Potential Enhancements to Good Practice
Registry participants	Selection bias is evaluated. The external validity is described (i.e., registry subjects are shown typical of the target population). It may also be informative to describe how the actual population was selected. For studies of comparative effectiveness and safety, contemporaneous data are collected for a comparison group to the extent that this is ethical and feasible, and that other clinically relevant, robust comparative data are not available. For registries where practice characteristics may impact outcome, diverse clinical practices are represented.
Data elements and data sources	The exposure data used to support the main hypothesis are as specific as possible. For example, data identify a specific product, including manufacturer, if available. Results that can be confirmed by an unbiased observer—such as death, test results, and scores from validated measures for patient-reported results or clinical rating scales—enhance accuracy and reliability. The followup period is sufficient to capture outcomes of interest.
Data quality assurance	Reproducibility of coding is evaluated. Potential sources of errors relating to accuracy and falsification are rigorously evaluated and quantified (e.g., through database and site reviews). For studies of safety, effectiveness, and comparative effectiveness, a sample of data are compared with patient records. Followup is reasonably complete for the registry purpose. Validated analytic tools are used for the main analysis (e.g., commercially available analytic packages are used).
Analysis	Loss to followup is characterized at key stages during the conduct of the study. For safety studies, the risks and/or benefits of products, devices, or processes under study are quantitatively evaluated beyond simply evaluating statistical significance (e.g., rates, proportions, and/or relative risks are reported). Sensitivity analyses are useful to examine the effect of varying the study population inclusion/exclusion criteria, the assumptions regarding exposure, and the definitions of potential confounders and outcomes on the association between the a priori exposure of interest and the outcome(s). If models are used, the specific data elements that are included are described.
Reporting	Consistency of results is compared and contrasted with other relevant research. Inferences about causal effects are based on a variety of factors, including the strength of the association, biases, and temporal relations. The practice of making inferences about causation largely on the outcome of tests of statistical significance is discouraged.

Return to Top of Page