Tutorials on Quality Measures > Validity of Clinical Quality Measures
Some clinical quality measures produce less valid results than other measures. The validity of results depends upon how the measure is built and whether it addresses the purpose chosen by the user. The NQMC measure summary captures key building blocks that can be used to assess the validity of a measure for specified purposes.
NQMC contains clinical quality measures grouped into five measure domains:
- Measures of process, access, outcome, and patient experience assess the quality of care provided by health care professionals and organizations.
- Measures of structure assess the capacity of health care professionals and organizations.
Users of measures should consider how well the intended use matches the measure developer's intended use. The use intended by the measure developer is captured in the NQMC measure summary section called State of Use of the Measure, especially the Current Use field.
Six key questions may help to determine a measure's validity as a measure of quality. The NQMC measure summary includes specific fields that provide an answer to each of these questions.
Question 1. How strong is the scientific evidence supporting the validity of this measure as a quality measure?
The key section from the measure summary for this question is Evidence Supporting the Measure.
Example: For an access measure "Mental Health Intensive Case Management," the field Type of Evidence Supporting the Criterion of Quality for the Measure characterizes the evidence as consisting of "A clinical practice guideline or other peer-reviewed synthesis of the clinical evidence," as well as, "One or more research studies published in a National Library of Medicine (NLM) indexed, peer-reviewed journal."
Question 2. Are all individuals in the denominator equally eligible for inclusion in the numerator?
A valid measure of quality of care should exclude individuals who should not receive the indicated care or are not at risk for the outcome. The key fields addressing this question are Denominator Sampling Frame and Denominator Inclusions/Exclusions.
Example: For a maternity care measure that assesses the percent of patients who received substance abuse treatment services, these fields would reveal whether all cases in the denominator are equally eligible to be included in the numerator. Not all mothers delivering newborns need or should receive substance abuse services, only those who have substance abuse problems. For instance, imagine two health plans, A and B: in Plan A, 10% of mothers are substance abusers, and 1% receive treatment. In Plan A then, there is gross underuse—only 10% of the mothers in need receive treatment. In Plan B, only 1% of mothers are substance abusers, and those 1% receive treatment; this is perfect care because 100% of the mothers in need receive treatment. The measure result shows that Plans A and B have identical scores (both score 1%); however, Plan B clearly provides higher quality care (100% of mothers in need receive treatment). This measure can be used to monitor use of services in the two Plans, but cannot provide a direct measure of the quality of care.
Question 3. Is the measure result under control of those whom the measure evaluates?
A measure that is not under control of those evaluated is not classified by NQMC in a Quality Measure domain, but rather as a Related Measure.
Example: A measure of asthma prevalence within a Health Plan is a not a measure of Outcome but of User/Enrollee Health Status. Clinicians can diagnose asthma, but asthma is primarily caused by genetic and environmental risk factors, not by receiving health care. A user should not use this measure to compare health care providers who care for populations that differ in their risk for developing asthma. A measure developer could use this measure to monitor how many enrollees within a Plan are known to have asthma, not as a direct measure of quality. A direct measure of success in detecting asthma would require two steps, firstly identifying all persons with asthma within the population of enrollees, then checking the percentage of these persons who were known by the Plan to have asthma.
Question 4. How well do the measure specifications capture the event that is the subject of the measure?
The key section from the measure summary for this question is Data Collection for the Measure. Within this section, the fields Numerator Inclusions/Exclusions and Denominator Inclusions/Exclusions show the details of measure construction.
Example: For a measure that is used to assess hospital admission rates for long-term complications among patients with diabetes mellitus, the fields Numerator Inclusions/Exclusions and Denominator Inclusions/Exclusions may describe the use of inpatient administrative data with diagnostic codes for renal, eye, neurological, circulatory, and other complications of diabetes. This enables the user to review, for example, whether codes for appropriate complications are included in the measure.
Question 5. Does the measure provide for fair comparisons of the performance of providers, facilities, health plans, or geographic areas?
For some measures, it may be important to account for differences in the characteristics of individuals who receive care from different providers, facilities, or health plans, or that live in different geographic areas. This may be done through statistical adjustment or stratification of the sampled population for the measure. The key fields for this question are Allowance for Patient or Population Factors and Description of Allowance for Patient or Population Factors. These fields reveal whether the measure includes allowance for patient or population factors and describe what those factors are. Allowance for patient or population factors is critical for measures of the outcomes of care because, in addition to the treatments given, demographic and clinical factors may influence the measure result. Allowing for patient or population factors may also be a concern for other types of measures.
Example: Results for an outcome measure for mortality rates among acute stroke patients will be largely determined by the patients' underlying health conditions present before the onset of the stroke. The field Allowance for Patient or Population Factors may show the methods of risk adjustment used in the measure to enable fair comparison of the quality of care in hospitals.
Question 6. Does the measure allow for adjustment of the measure to exclude patients with rare performance-related characteristics when appropriate?
Example: A measure concerning provision of an evidence-based treatment allows exclusion of patients who refuse the treatment. The field Exclusions/Exceptions reveals whether allowance is made for medical, patient, and system factors that influence whether a particular process of care should be provided to a patient.
