Develop Your Own?
Contents
Before You Begin
Developing Detailed Specifications
Data Issues to Keep in Mind
Sampling
Reliability and Validity
Field Testing
An Alternative to Consider: Customizing Existing Measures
State-Designed and State-Negotiated Quality Measures
Want More Information?
Existing measurement sets assess a limited number of factors. There may be health conditions, health services, or specific population subgroups of great interest for which established measures are lacking.
Examples: A State may want to:
- Measure whether its child population is receiving lead screening.
- Assess whether enrolled children receive service for an urgent medical condition within a set time frame.
- Assess services received by children who are severely overweight.
Where established measures are lacking, States may wish to develop their own quality measures. Additionally, Title V Maternal and Child Health Programs are required to develop their own "State-negotiated" quality measures as part of their Title V Maternal and Child Health Block Grant responsibilities.
This section:
- Discusses questions to consider Before You Begin as you contemplate developing your own measure.
- Describes some key characteristics of quality measures and the measurement development process, including:
- Discusses Customization of Existing Measures as a possible strategy.
- Provides some examples.
Online Resources:
For more information on existing measurement sets, go to:
http://www.ahrq.gov/chtoolbx/measures.htm#measures
For more information on Title V Maternal and Child Health Programs, go to:
http://www.ahrq.gov/chtoolbx/measure5.htm
For more information on "State-negotiated" quality measures, go to:
http://www.ahrq.gov/chtoolbx/measure8.htm
Return to Contents 
Before You Begin
The process of developing quality measures is time consuming and resource intensive. The goal is to create accurate measures that can withstand critical scrutiny. Quality measurement strives for results that cannot be called into question because of unclear definitions, ambiguous procedural specifications, or flawed assumptions concerning whether and how the involved health care-providing entity affects the factor being measured.
Development of measures that produce such solid results is a multiyear venture, requiring:
- Technical expertise.
- Senior-level oversight and championing.
- Explanation to and involvement of multiple stakeholders.
- Significant funding.
Before quality measures are ready for use, the following are required:
- Careful planning.
- Expert input.
- Rigorous testing.
Undertaking to develop your own quality measure is a significant venture. As you contemplate developing your own quality measure, you may wish to consider the following questions:
Return to Contents
Have You Identified the Significant Health Problems of Your Target Population(s)?
Even if your overall objectives are already set, alternative quality measures are possible. In general, the aspects of health care most useful to measure:
- Have a significant impact on the health of the relevant population(s).
- Are important to key stakeholder groups.
- Consume significant health care resources.
- Offer clear opportunities for improvement.
Return to Contents
Do Measures Already Exist that Accurately Assess Quality in Your Area of Interest?
Have you:
- Looked at existing quality measures in general use?
- Searched AHRQ's National Quality Measures Clearinghouse™ (NQMC) compendium of quality measures?
- Consulted with your peers?
- Consulted with professional organizations?
- Reviewed the literature?
If you are considering a survey of perceptions of care, have you:
- Reviewed existing survey instruments?
- Done focus groups with relevant populations?
- Consulted with consumers and advocates?
Online Resources:
For more information on existing quality measures, go to: http://www.ahrq.gov/chtoolbx/measures.htm
For more information from the National Quality Measures Clearinghouse™ (NQMC) on 74 child health measures, go to: http://www.qualitymeasures.ahrq.gov/
Return to Contents
Have You Clearly Defined the Anticipated Uses of the Resulting Quality Information?
Will you use the results for:
- Quality improvement?
- Accountability?
- Program management?
Clear and meaningful plans for the use of the resulting quality information should be in place before any measure development begins.
Online Resource: For more information on Uses of Quality Measurement, go to: http://www.ahrq.gov/chtoolbx/uses.htm
Return to Contents
Are There Other Methods of Obtaining Comparable Information?
Alternative ways of getting comparable information using existing data or information sources should be thoroughly explored prior to beginning measure development.
Return to Contents
Have You Examined Your Assumptions About Causality?
The concept of quality measurement is based on the assumption that there is a meaningful connection between the operations of an identifiable health care-providing entity and the health care results being measured.
Example: School absenteeism may be affected by child health programs, but there are too many other contributing factors for absenteeism to serve as a useful measure of child health program effectiveness.
Many factors affect child health, and many of them may be considerably beyond the reach of any health care plan, program, or system.
Return to Contents
Developing Detailed Specifications
Quality measurement tools are designed to produce accurate measures of quality that can withstand critical scrutiny. Well-designed quality measures generate information that cannot be called into question because of:
- Unclear definitions.
- Ambiguous procedural specifications.
- Flawed assumptions concerning connections between health care providers and the aspects of health care being measured.
In order to ensure such a result, quality measurement tools must include clearly defined detailed terminology or specifications that are:
Objective. The definitions of terms are quantifiable and capable of external verification.
Standardized. Definitions of data elements, data collection, and data analyses are sufficiently precise and comprehensible that they can be understood and applied in the same way regardless of who refers to or applies it.
Comprehensible. The terminology can be understood without specialized training or expert knowledge.
A quality measure is a relatively simple concept that describes a rate; a rate has a numerator and a denominator. Successful quality measure development requires that the specifications of these key data elements (numerator and denominator) must also be:
- Objective.
- Standardized.
- Comprehensible.
This can be complex. To illustrate, consider a quality measure that assesses "the percentage of children with an urgent medical condition receiving care within a specific time frame." This quality measure could be illustrated as follows:
The number of children with an urgent medical condition receiving care within a specific time frame
The total number of children with an urgent medical condition
To calculate the quality measure, the numerator (above the line) is divided by the denominator (under the line), thus generating the quality measurement or rate.
The denominator of this measure is the "total number of children with an urgent medical condition." Some of the questions that must be answered in order to define this denominator are:
Ages Included. What are the ages of the children who are included?
Measurement Period. What is the measurement period?
- A calendar year?
- A State fiscal year?
- Some other period?
Age Specification. If, for example, children ages 1 through 5 are defined as the denominator population:
- What about children who turned 1 during the reporting year?
- Children who turned 6?
- Which are included?
- Which are excluded?
Systems or Units Assessed. What service delivery systems or units of analysis are you assessing?
- One managed care organization (MCO)?
- All MCOs?
- A primary-care case management (PCCM) program?
- Fee for service (FFS)?
- Local, regional, or statewide?
- Specific program(s)?
Title V. In assessing Title V programs:
- How is the relevant child population defined?
- What is the link between entities providing public health care and the resulting care?
SCHIP or Medicaid. In assessing the State Children's Health Insurance Program (SCHIP) or Medicaid, can SCHIP or Medicaid children be accurately identified?
Definitions. It is necessary to determine:
- What definition of an "urgent medical condition" will be used.
- What authority will be used in establishing this definition.
- What experts or established clinical standards will be called upon.
Methods. In determining methods to be used:
- What standardized method will be used to ensure that all of the children whose care is surveyed have an "urgent" level of medical need?
- What codes will be used?
- What data sources will be surveyed?
Once these questions have been resolved, the answers are translated into a comprehensive description of the denominator. The goal is to describe the measure specifications so precisely and in such detail that reasonably knowledgeable persons across a wide variety of settings would be able to understand and apply them in a consistent manner. Some examples of specifications of existing measures may be of interest.
Online Resource: For more information on examples of specifications of existing measures, go to: http://www.ahrq.gov/chtoolbx/uses.htm
After the denominator is defined, the numerator must also be defined. In the example given, the numerator is "the number of children with an urgent medical condition receiving care within a specific time frame." The definitions of ages, of the populations to be used (plan or program enrollment, type of service delivery system, etc.), and of "urgent medical condition" will have already been agreed upon and defined, as noted. However, further questions remain:
- What is the appropriate time frame within which a child with an urgent medical condition should receive care?
- What authority will be used in establishing this definition?
- What experts or established clinical standards will be called upon?
- What data sources will be used to get this information?
Once these questions have been answered, the core definitions for this quality measurement will be in place.
In 2002, the Centers for Medicare & Medicaid Services issued a protocol for calculating performance measures as part of overall guidance for the mandatory external quality review of Medicaid MCOs. The protocol describes a State Medicaid agency's activities related to the specification, collection, and verification of performance measures.
Online Resource: Go to: http://www.cms.hhs.gov/home/medicaid.asp.
Return to Contents
Data Issues to Keep in Mind
For many child health issues, it may be hard to identify data that are:
- Readily available.
- Comparable.
- Complete.
- Accurate.
For statistical reasons, the number of cases to be studied must also be large enough that information on the quality of health care services received by the study group can be reliably assumed to apply to others.
Quality measurement can make difficult demands on already limited data sources. In the example of determining the percentage of children receiving the urgent care they need within a given time frame, calculation of the quality measure requires accurate data on the timing of the receipt of care and on a variety of diagnoses.
Quality measurement is a data-driven activity. The feasibility of calculating a quality measure depends heavily on the availability of routinely collected data or on the resources to collect data that are not readily available. Specification of the nature and sources of the data to be used in any given quality measure and of how the data are to be collected is important for achieving credible measurement results.
The following sections present some issues to consider, particularly if you are going to develop your own data specifications and data collection methods.
Return to Contents
Push the Envelope—But Not Too Hard
Data availability issues are central to any quality measure. On the one hand, ambitious measures that rely on data sets that are fragmentary, incomplete, inaccurate, or just plain nonexistent ensure that the resulting quality measurement results will be flawed. On the other hand, exclusive reliance on existing and readily available data may limit what is possible and prevent progress. Therefore:
- Include those in charge of data in your planning so they can see the potential of quality measurement.
- Consult them about increasing data sources and improving data collection.
Because data completeness and accuracy are critical to all quality measures, whether they are part of standardized measurement sets or measures you develop yourself, data specialists are a key to success.
Return to Contents
Learn and Use the "Typical" Data Sources
Most quality measures use one of the following:
- Administrative data.
- Medical records data.
- Survey data.
Administrative Data. All health programs, plans, and systems maintain administrative data. You should:
- Understand what information these data contain.
- Learn the limits as well as the strengths of the data.
- Investigate whether the data are complete and accurate.
Because administrative data are regularly and routinely collected, their use in calculating quality measures will lessen both the burden and the cost. Encounter and claims data are examples of administrative data.
Online Resource: For more information on these types of data, go to Consider Using Encounter or Claims Data—Very Carefully at http://www.ahrq.gov/chtoolbx/develop#encounter.
Medical Records. All providers maintain medical records with detailed information on patient conditions and treatments, including important information on preventive and well-care services. In many settings much of that information is found only in medical records, particularly where the information does not pertain to a procedure or service that is individually billable.
In some cases, systems have been established to transfer some medical records information into administrative databases, making it easier to use for quality measurement. In most settings, medical records review is a costly but necessary component of the calculation of selected quality measures.
Survey Data. Survey data are as good as the procedures used to:
- Develop the survey instrument.
- Draw the sample.
- Administer the survey.
Study the protocols developed for such public domain survey instruments as Consumer Assessment of Health Plans CAHPS®, Experience of Care and Health Outcomes Survey (ECHO™), Promoting Healthy Development Survey (PHDS), and the Young Adult Health Care Survey (YAHCS) for models in these areas.
Online Resources:
For more information on CAHPS®, go to: http://www.ahrq.gov/chtoolbx/measure2.htm
For more information on ECHO™, go to: http://www.ahrq.gov/chtoolbx/measure9.htm
For more information on PHDS, go to: http://www.ahrq.gov/chtoolbx/measure6.htm
For more information on YAHCS, go to: http://www.ahrq.gov/chtoolbx/measure7.htm
Return to Contents
Consider Using Encounter or Claims Data—Very Carefully
Encounter and claims data are varieties of administrative data. Some States use encounter or claims data as the basis for the calculation of Health Plan Employer Data and Information Set (HEDIS®) measures or for other variations on established quality measures. The data are on hand and are gathered continuously as a part of normal operations.
Before embracing these data sources for the purposes of quality measurement, however, two caveats apply:
- First, are the data complete? Claims data are likely to be complete for those services or procedures that are individually reimbursed or billable and less likely to be complete for services or procedures that are not individually reimbursed. Encounter data completeness also varies.
- Second, are the data accurate and consistent? This can be especially problematic with encounter data, where such issues as inconsistent coding can pose significant challenges.
Some States have made significant investments in cleaning claims or encounter data to ensure accuracy and completeness. Where such precautions are taken and maintained, such data can be an important data source for the calculation of quality measures.
In May 2002, CMS released a protocol for Validating Performance Measures submitted by MCOs. Although intended primarily for Medicaid use, these protocols offer useful examples to others concerning how to ensure data completeness and accuracy.
Online Resources:
For more information on HEDIS®, go to: http://www.ahrq.gov/chtoolbx/measure4.htm
Download the protocol for Validating Performance Measures. Go to: http://www.cms.hhs.gov/home/medicaid.asp.
Return to Contents
Sampling
In many cases, a sample of children is chosen for quality measurement studies. Methods of choosing samples must be precisely described and followed to avoid inadvertent bias that undermines the credibility of the result.
Example: If data collection methods make it easier to submit electronic data rather than data on paper, bias is introduced, and data on paper may be underrepresented in the sample.
If sampling methods are not scrupulously designed so as to select a sample that is representative of the entire population in question, results could be questioned.
Examples:
- If a sample of children with urgent medical conditions overrepresents children in urban areas, the results cannot be applied to children in rural areas.
- If a sample is not large enough, the results cannot be generalized beyond the sample itself.
The small numbers of children with specific special health care conditions make it particularly difficult to develop quality measures in this area.
Return to Contents
Reliability and Validity
Quality measures must be tested for reliability and validity as a part of the measurement development process.
Reliability. A "reliable" measure will give the same result every time that it is applied to the same aspect of health care. This can be compared to a thermometer, which will show the same temperature each time it is used to measure the temperature in a particular location in a living room if, in fact, the temperature has not changed.
Validity. A "valid" measure will be reliable and will also measure the intended aspect. This can be compared to ensuring that the reading on the thermometer measuring the average temperature in the living room is not instead measuring the temperature of the corner next to the hot fireplace.
Specific statistical tests exist to assess reliability and validity. Such testing requires specific expertise. If in-house expertise is not available, outside assistance is needed to ensure that these critical steps in measure development are successfully completed.
Return to Contents
Field Testing
Field testing is another essential component of the measurement development process. In field testing, a quality measure that appears to be ready for general use is used in an actual setting on a trial basis.
Example: If the quality measure is intended for use in MCOs and PCCM programs, volunteer MCOs and PCCM programs with experience in quality measurement will be identified. They will put the measure to use and report on how it works.
Field testing answers questions such as:
- Can the denominator and numerator specifications be interpreted readily and consistently?
- Are the anticipated data sources actually available?
- Can the procedures for data collection be followed in a reasonable fashion?
- Can quality measure data be analyzed and rates calculated using the recommended procedures?
Return to Contents
An Alternative to Consider: Customizing Existing Measures
Because quality measurement development is demanding and resource intensive, some States have found that customization of existing, rigorous, field-tested quality measures is a better alternative. This is particularly true for HEDIS® and CAHPS®, two measurement sets characterized by methodological precision and rigor.
Some States have used existing HEDIS® measures as a starting point, then customized specifications.
Examples of such customization are:
- The age ranges to which a measure is applied.
- The required length of "continuous" enrollment.
New York and Arizona have used customized HEDIS® measures as part of their Quality Assurance Reporting Requirements, which include a measure of lead screening. Other States, including Massachusetts, have expanded the use of HEDIS® and/or CAHPS® measures beyond MCOs to collect and report data for PCCM programs and fee-for-service Medicaid or to calculate State averages. Still others have calculated HEDIS® results using MCO-reported encounter data rather than the data sources specified in the HEDIS® measure descriptions.
Such HEDIS® customization builds on the strengths of the HEDIS® measurement development process. However, any changes in specifications can compromise the comparability of the results. True comparisons of results using such quasi-HEDIS® measures can only be made with other results calculated using the same customized specifications.
Example: If all MCOs in a State use the same customized quasi-HEDIS® measure, comparisons can be made, but only among MCOs within that State.
Further, if specifications are customized, it is essential that the same precision and rigor be used in making the change that was used in developing the original measure. If customized specifications introduce vagueness or ambiguity, precision is lost and both comparability and the reliability and validity of the results suffer.
The CAHPS® measurement set is designed to allow user flexibility:
- CAHPS® includes pretested modules of supplemental questions on a wide variety of issues.
- CAHPS® specifications indicate where such modules can be added.
- CAHPS® also allows user-designed questions to be added in accordance with CAHPS® procedures.
Nonetheless, CAHPS® also includes specifications and protocols designed to ensure reliability, validity, and comparability. Customization of CAHPS® must be done with care in order to avoid compromising the interpretation of results. Keep in mind:
- The sequencing of questions has been carefully structured and tested to ensure that the order of questions does not in itself bias responses.
- The Child Core set itself cannot be altered without compromising comparability.
- CAHPS® supplemental modules must be used in accordance with CAHPS® specifications concerning where they should be inserted in order to avoid bias.
- For the same reason, any user-designed questions must be added with care.
To avoid biasing results through faulty positioning or through the addition of too many new items, user-designed items should be added only in consultation with CAHPS® user support staff.
Online Resources:
For more information on HEDIS®, go to: http://www.ahrq.gov/chtoolbx/measure4.htm
For more information on CAHPS®, go to: http://www.ahrq.gov/chtoolbx/measure2.htm
For more information on the Quality Assurance Reporting Requirements of New York, go to: http://www.ahrq.gov/chtoolbx/measure8.htm#ny
For more information on the Quality Assurance Reporting Requirements of Arizona, go to: http://www.ahrq.gov/chtoolbx/measure8.htm#az
Return to Contents
State-Designed and State-Negotiated Quality Measures
Since 1996, Ohio's Department of Mental Health has been designing the Mental Health Consumer Outcomes System, which includes a standard, statewide approach to identifying and measuring consumer outcomes and quality of Ohio's mental health systems. Created through a broad-based effort including consumers, advocates, providers, and community mental health boards, the system measures outcomes in the areas of: clinical status, quality of life, functional status, and safety and health.
The measures are intended to have multiple uses, including: local and State benchmarking; quality improvement by provider agencies, community mental health boards, and Ohio's Department of Mental Health; and accountability to stakeholders, including consumers and consumer advocates.
Online Resource: For extensive information on the Ohio Mental Health Consumer Outcomes System, go to: http://www.mh.state.oh.us/oper/outcomes/outcomes.index.html.
Rhode Island's RIteCare program includes Medicaid and SCHIP. It is administered by the Rhode Island Department of Human Services (DHS), which has developed and applied its own quality measures. The Rhode Island DHS has named these "health indicators." In 2003, Rhode Island's DHS issued a report monitoring trends in the diagnosis and treatment of asthma.
Online Resources:
Download the report in PDF format (179 KB). Go to: http://www.dhs.ri.gov/dhs/reports/ristats%20vol%202%20issue%203.pdf. Plugin Software Help.
For a manual describing the process Rhode Island used, go to: http://www.dhs.state.ri.us/dhs/reports/dhcresys.htm.
The Title V Maternal and Child Health Program includes numerous examples of State-negotiated quality measures.
Online Resource: For more information on such measures, look under "search" at: https://performance.hrsa.gov/mchb/mchreports/states_narratives/states_narrative.asp
Return to Contents
Want More Information?
For more information on related issues, you may want to review the following article:
McGlynn EA, Asch SM. Developing a clinical performance measure. Am J Prev Med 1998;14:14-21.
Abstract available on PubMed®:
http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=9566932&dopt=Abstract
Return to Contents
Internet Citation:
Child Health Care Quality Toolbox: Develop Your Own?. July 2004. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/chtoolbx/develop.htm
540 Gaither Road Rockville, MD 20850 