Future Development of Nursing Home Quality Indicators

Abstract

Nursing home quality indicators have been developed over the past 10 years to quantify nursing home quality and to draw systematic comparisons between facilities. Although these indicators have been applied widely for nursing home regulation, quality improvement, and public reporting, researchers and stakeholders have raised concerns about their accuracy and usefulness. We critically evaluate nursing home quality indicators from the standpoint of theory, measurement, and application, and we recommend strategies to make the indicators more valuable as quality assessment tools. We recommend that (a) more comprehensive quality indicators should be developed in conjunction with the new Minimum Data Set 3.0; (b) the validity and reliability of the indicators should be evaluated thoroughly with respect to both measurement and application; (c) statistical criteria should be incorporated explicitly into quality indicator scoring and outlier targeting; (d) the dimensionality and theoretical structure of the quality indicators should be carefully examined; (e) risk adjustment methods should be refined and broadened; and (f) quality indicator reporting systems should be strengthened and tailored to stakeholders' needs.

Although the quality of care in nursing homes has been discussed for decades, only recently have comprehensive administrative systems been developed to quantify nursing home quality and to systematically compare facilities. Similar systems have been designed to assess quality in acute care (Romano et al., 2003) and other long-term-care areas (Shaughnessy et al., 1994). Nursing home quality indicators, or QIs, became feasible after the advent of the Minimum Data Set (MDS), with its standardized collection and reporting of data on nursing home residents. The Centers for Medicare and Medicaid Services (CMS) viewed the MDS data as a vital source of information for quality assessment. Through support from the CMS, researchers at the University of Wisconsin's Center for Health Systems Research and Analysis (CHSRA) developed a set of nursing home QIs from MDS data items (Zimmerman et al., 1995). The CMS followed up in 2002 with a new set of MDS-based quality indicators, which were termed “quality measures” (QMs; Berg et al., 2002). The QMs were similar in concept to CHSRA QIs and they relied on many of the same MDS items, although they used different methods for risk adjustment and reporting (described in the paragraphs that follow).

Fourteen QMs were incorporated into Medicare's Nursing Home Compare public reporting system (CMS, 2004), which contains quality-related information on every nursing home in the nation. In addition, the CHSRA QIs and CMS QMs are used by state nursing home survey agencies, in the Joint Commission for Accreditation of Healthcare Organization ORYX quality monitoring system, and in various provider group or state initiatives (Harris & Clauser, 2002). Furthermore, CMS has contracted with state quality improvement organizations to assist nursing facilities with quality improvement efforts based on the CMS QMs.

Given the widespread dissemination of nursing home QIs (or QMs), they should be firmly grounded clinically, their statistical properties should be well understood, their validity and reliability should be thoroughly tested, and they should have demonstrated effectiveness as decision-making tools for regulators, care providers, and consumers. Unfortunately, many questions remain about the indicators and their application to quality assessment or benchmarking (Mor, Angelelli, Gifford, Morris, & Moore, 2003). In fact, the General Accounting Office (2002) recently concluded that national implementation of the CMS's QM reporting system was premature because of questions about the indicators chosen for the system, the accuracy of underlying MDS data, and the way information would be reported to the public. Nonetheless, the CMS proceeded with implementation of the reporting system, and, for the most part, these questions remain unanswered.

In this article we have two main purposes: (a) to critically evaluate the current state of nursing home QIs from the standpoint of theory, measurement, and application; and (b) to recommend additional steps in the development of the indicators so they can become more valuable quality assessment tools. We draw from recent critiques of QIs by Zimmerman (2003), Mor, Angelelli, and colleagues (2003), and others (Berg et al., 2002; Harris & Clauser, 2002; Schnelle, Bates-Jensen, et al., 2004) who have been actively involved in refinement and application of the indicators. Building on their research, we provide recommendations for the future development, improvement, and refinement of QIs and QMs and the research necessary to achieve this.

Quality Indicators

Quality indicators have focused on measures of the processes and outcomes of care derived from routinely collected assessment data (MDS). They are defined at the resident level according to receipt or nonreceipt of a service; presence or absence of a condition at a single point in time (prevalence); or development of or change in a condition over time (incidence). They are then aggregated to the facility, state, and national levels and expressed as prevalence or incidence rates. All 24 CHSRA QIs are calculated for the nursing home population as a whole, whereas 11 QMs refer to long-term residents (i.e., those with full or quarterly assessments for a given period) and 3 refer to short-term residents (i.e., those with a 14-day prospective payment system MDS in the past 6 months).

Table 1<--?1-->displays the 24 CHSRA QIs and 14 CMS QMs along with MDS items used in defining the indicators or in risk adjustment, and their prevalence or incidence rates drawn from national data sources. Similarly defined CHSRA QIs and CMS QMs have comparable rates of prevalence or incidence (maximum difference of 2%), suggesting considerable overlap between the two systems. Nonetheless, rates for different QIs vary widely. Indicators such as prevalence of fecal impaction are quite rare (2/1,000) and might qualify as sentinel events, whereas other indicators are very common, such as prevalence of bowel or bladder incontinence among high-risk residents (941/1,000). Most indicators have some form of risk adjustment—subpopulations are excluded from the QI denominator (exclusions); the indicator is categorized into high- and low-risk groups with separate rates for each group (CHSRA QIs); or rates are adjusted according to covariates entered into a multiple logistic regression model (CMS QMs) as described by Berg and associates (2002).

Critique of the Quality Indicators

Threats to Reliability and Validity

The QIs have received limited reliability and validity testing. The developers of the CHSRA QIs found high levels of agreement between QI reports and clinical records in small convenience samples of residents who flagged on various QIs (Zimmerman et al., 1995). In addition, facility-level CHSRA QIs displayed reasonable stability over time (calendar quarters) in a sample of approximately 500 facilities (Karon, Sainfort, & Zimmerman, 1999). Researchers tested the reliability and validity of selected CHSRA QIs and all of the CMS QMs in a sample of more than 4,000 residents in 209 facilities from six states (Morris et al., 2003). The interrater reliability of MDS items making up the QIs was generally very high. There were 17 CHSRA QIs and 13 CMS QMs tested: 15 were found to be superior (κ >.75), 14 acceptable (κ =.40–.75), and 1 poor (κ <.40). Researchers assessed construct validity by correlating facility QI rates with the facility's use of what the researchers termed “preventive strategies” (e.g., staff training, higher resource level, and quality-improvement programs) or “responsive strategies” (e.g., facility documentation of resident conditions, changes in resident status, or referral to specialists, such as physicians). Of the QMs, 9 were judged to have top validity, 3 had middle validity, and 1 was not valid. The statistical criteria for these validity assessments were not very stringent: a top validity item had only to obtain a Multiple R of.45 or higher with preventive factors or.55 or higher with combined preventive and responsive factors. A more serious problem is that this study did not test the validity of the QIs at the resident level; the researchers did not attempt to determine if a resident who flagged on a QI was actually receiving poor care. This omission raises the possibility that criterion measures are not associated with care quality at the resident level, or that indicators are not valid in targeting individual residents for quality review or in-depth evaluation.

A potentially serious validity threat may arise from ascertainment bias. This is the paradoxical phenomenon in which the staff in better quality facilities are more likely to ascertain negative health outcomes or conditions, resulting in QI rates that may be higher (worse) than in lower quality facilities where bad outcomes are not ascertained (Mor, Angelelli, et al., 2003; Schnelle et al., 2003). Morris and colleagues (2003) found little evidence for ascertainment bias in their indicator validation study, although they acknowledged that their method for detecting and correcting for bias might have been flawed.

The collecting and reporting of MDS data in the context of regulation or reimbursement presents additional threats to the validity of the QIs. Facilities may deliberately underreport certain indicators (e.g., severity of pressure sores) in order to avoid sanctions (Harrington & Carrillo, 1999). In other instances, such as with case-mix reimbursement systems, facilities may have an incentive to overreport problem conditions if this leads to increased acuity scores and higher payment for care (Chen & Shea, 2002). These administratively induced biases may help to explain the disturbingly wide variation in QI rates among states (Berg et al., 2002; Morris et al., 2003). Some variation in rates may be due to interstate differences in the composition of nursing home populations. Alternatively, states may vary in their assessment practices as a result of differences in reimbursement systems, interpretation of MDS items by nursing home survey teams, or training and expertise of nursing staff.

Insufficient Attention to Sampling or Statistical Error

Sampling error affects the QIs in two important respects. Because of the assessment schedule and the reference periods for different variables, the MDS captures information over only a small sample of time during a resident's stay (Karon et al., 1999; Mor, Angelelli, et al., 2003). The reference period for many MDS items is either 7, 14, or 30 days prior to the assessment, and assessments typically are performed on a 90-day schedule unless a resident has a significant change assessment, which happens infrequently. Thus, many health events or services could be missed because they fall outside the MDS reference periods.

A second source of sampling error is the small number of residents that form the denominators for QI rates in many facilities. The majority of QIs are not sentinel events; they refer to processes or outcomes that may or may not be viewed as problems depending on the facility's QI rates compared with that of its peers (percentile rank) or a statewide average (z score). Ranks are problematic because they obscure the underlying distribution of QI rates in the comparison group. Small sample sizes can lead to very large standard errors for QI rates (Mor, Angelelli, et al., 2003). For example, if a facility has a QI incidence rate of.05 based on a sample of 20 residents, the 95% confidence interval (CI) would be 0.001–0.245. Even a sample of 100 residents yields a 95% CI of 0.01–0.11. A related statistical problem arises because residents are clustered within facilities. Berlowitz and colleagues (2002) took clustering into account when they applied Bayesian techniques in combination with hierarchical modeling in profiling risk-adjusted rates of pressure sores for a sample of 108 nursing homes. They found substantial shrinkage of the QI rates after they applied their statistical method. Compared with the conventional z-score techniques, the range of risk-adjusted QI rates narrowed from 0–14.3% to 1.0–4.8%, and the number of outliers declined from 15 to 2.

Controversy Surrounding Risk Adjustment

Despite the generally recognized need for risk adjustment when quality-of-care comparisons are drawn between facilities, there is little agreement about what risk factors should be selected for adjustment and how it should be carried out. As shown in Table 1, CHSRA QIs adjust for risk by dividing resident populations into low- and high-risk categories and computing separate QI rates for each category (Arling, Karon, Sainfort, Zimmerman, & Ross, 1997). The CMS QMs are risk adjusted with a multiple logistic regression approach in which the observed QM rate is expressed as a ratio to the expected rate based on the risk characteristics of the resident population (Berg et al., 2002). Developers of QIs have taken a conservative approach toward risk adjustment because of their own or stakeholder fears that risk adjustment might “let some facilities off the hook” (Mor, Berg, et al., 2003; Zimmerman, 2003). That is, risk factors that were themselves the result of poor-quality care (e.g., immobility, bowel incontinence, or weight loss) might lower the facility's adjusted QI rate on a major indicator (e.g., pressure sores). Nursing home residents suffer from multiple conditions that interact with different care processes. If risk adjustment excluded any risk factor that possibly could be influenced by facility care, the list of adjusters might be quite short and they probably would explain very little of the variation in risk. Using a comprehensive or minimalist set of risk adjusters can affect QI performance (R. L. Kane, Flood, Bershadsky, & Keckhafer, 2004).

Failure to Identify Key Conceptual Dimensions

The QIs are defined as a series of individual indicators that can be grouped into general categories (domains); however, no underlying dimensions have been identified empirically, and there have been no summary measures for QIs in clinically related areas. Users of the QIs must thus decipher results from a long list of measures, and facilities may score high on some QIs and low on others. It is difficult to get a picture of overall care quality in a facility. Although they acknowledge the conceptual dimensionality of QIs, Mor, Angelelli, and colleagues (2003) conclude that QMs show only weak empirical correlation. There is scant statistical evidence for “consistent and coherent dimensions of quality that can usefully summarize the multiplicity of nursing home quality measures” (p. 263). Interestingly, these authors do not cite any studies that have empirically tested the structure or dimensionality of nursing home QIs. In contrast, researchers have been reasonably successful in identifying dimensionality in nursing home survey deficiencies (Mullan & Harrington, 2001), which correspond with many of the care problems measured by the QIs.

Additional Issues

Additional criticisms have been leveled at the QIs. First, they narrowly focus on clinical areas of care with limited attention to quality of life, which is poorly measured on the MDS (R. A. Kane et al., 2003). Even in the clinical area, a wide range of clinically meaningful MDS items are not candidates for QIs because they are not included on quarterly assessments (Zimmerman, 2003). Second, almost all QIs focus on care problems, directing attention toward avoiding poor care rather than fostering good care (R. A. Kane, 2003; Morris et al., 2003; Zimmerman et al., 1995). Even in the problem-focused approach, little attention is devoted to facilities that have the fewest problems compared with those that have the most problems (Berlowitz et al., 2001; Simmons, Babineau, Garcia, & Schnelle, 2002).

Third, the reliance on peer comparisons (e.g., percentile rank) in the absence of absolute standards or benchmarks can obscure systemic quality-of-care problems. Mediocre or poor care might become the norm, at least implicitly, when the mean or median is the reference point (Karon & Zimmerman, 1996). Conversely, if standards are only relative, providers may feel as if they are aiming at a moving target; an improvement in care will not change the facility's rank if other facilities improve as well. Finally, the QIs are presented to the public with inadequate explanation (GAO, 2002; Harrington, Collier, et al., 2003; Harrington, O'Meara, Kitchener, Simon, & Schnelle, 2003; Harris & Clauser, 2002).

Recommendations for Future Development

A new, more comprehensive set of QIs should be developed. The new indicators should be markedly improved with respect to validity and reliability, theoretical structure, risk adjustment methods, and reporting systems or other applications.

Develop More Comprehensive Indicators in Conjunction With the New MDS 3.0

The MDS 3.0 scheduled tentatively for 2005 provides an opportunity to modify the current QIs and to create new indicators (CMS, 2003). Current QIs have been limited by information available in the MDS. The approach should be reengineered so that the final set of MDS items are decided on in light of desired QIs. The items appearing on quarterly assessments should be chosen with regard to their use as QIs; items appearing on comprehensive assessments but not on the quarterlies are spaced so far apart that they make poor candidates for either incidence or prevalence QIs.

Work is in progress to improve the validity and reliability of items in the MDS 3.0. First, for example, a five-item Geriatric Depression Scale (Sheikh & Yesavage, 1986) and a new set of pain items could improve interrater agreement for these inherently subjective areas. Second, many sections (e.g., pain management, diseases or diagnoses, vaccines, delirium, and falls) have been revised to reflect changes in clinical knowledge, perhaps allowing for more valid assessment. This updated clinical information has special relevance for the development of future postacute QIs; as long-term care becomes more clinically complex, there will likely be a concomitant call for new postacute QIs. Including a new quality-of-life section in MDS 3.0 greatly expands the range of potential indicators.

In addition to the development of new QIs, we recommend a thorough review of the current QIs by clinical experts for their clinical relevance and support of evidence-based practice. For example, the QI “prevalence of occasional or frequent bladder and bowel incontinence without a toileting plan” does not take into consideration those residents who are totally incontinent (all or most of the time). Thus, this QI supports the practice of not assessing or intervening with residents who are totally incontinent. Scheduled toileting interventions could at the very least reduce the number of incontinent episodes for these residents.

Positive QIs are needed to change the tone of quality-assessment efforts from simply avoiding bad care to achieving better care. Such positive indicators might emphasize nursing rehabilitation services, such as range-of-motion, activity of daily living (ADL) skill training, and toileting programs, for residents with potential need for these services; improvements in mood or problem behaviors between assessments; and improvements in ADL skills among individuals with potential for improvement. Quality-of-life items added to the new MDS 3.0 would be excellent candidates for positive indicators. Exemplary care could be recognized as very low scores on problem-focused QIs in combination with very high scores on positively directed indicators.

Thoroughly Evaluate Validity and Reliability of Quality Indicators

The psychometric properties of the QIs should be thoroughly evaluated, as should their validity in practice as quality-assessment tools. The contractors for the MDS 3.0 will no doubt report on the psychometric properties of the items. One would hope that refining the instrument would improve its reliability and validity. Extensive testing of the validity of the QIs (e.g., criterion validity and stability over time) should proceed with facility-level studies based on samples drawn from different states and regions and with facilities selected systematically according to resident acuity or risk, quality of care, size, or other characteristics that may influence the generalizability of findings. In addition, resident-level reliability and validity studies should use sufficient samples. These studies should examine criterion-related validity, comparing QIs against presumptive or actual quality-of-care problems as determined through clinical records, expert opinion of on-site research staff, nursing home survey records, or other criteria.

The QIs should also be evaluated as they are applied to nursing home practice. Despite the extensive use of the QIs by providers and regulators, very little systematic research has been conducted into potential natural experiments involving QI applications. From our experience, providers tend to focus on the “big five” QIs—falls, pressure sores, weight loss, restraints, and psychotropic medications—because they are deleterious outcomes or processes that can lead to deleterious outcomes; they are readily documented; they are often linked to each other; they receive special attention in the survey process; and they are candidates for quality improvement. In contrast, QIs for cognitive decline or incidence of depression receive less attention because of the greater difficulties with assessment and intervention. The members of the nursing staff generally do not attend to incontinence QIs because incontinence is so common it is almost taken for granted. For the same reason, most state survey agencies place relatively little emphasis on this care problem. Rantz and colleagues stand out as researchers who have attempted to study the QIs in natural settings by experimenting with alternative QI reporting systems (Rantz et al., 2003), developing complementary quality-assessment instruments that could be compared with the QIs (Rantz, Popejoy, et al., 1997), and recording their experiences in consulting with facilities on QI applications (Rantz et al., 2001). Schnelle and colleagues have been in the forefront of examining how care is actually being delivered in nursing facilities, shedding light on the validity of the QIs while increasing knowledge about care practice (Schnelle, Simmons, et al., 2004; Schnelle et al., 2003; also see Bates-Jensen et al., 2003; Simmons et al., 2003).

Incorporate Statistical Criteria Into Quality Indicator Scoring

Most QIs will be based on probabilistic models. Even if underlying MDS items were measured without error, we would still have a problem of statistical inference. The most formidable statistical problem with the QIs is the estimation error associated with small sample sizes (QI denominators) and rare or infrequent events (small numerators). The problem of statistical error is accentuated when rates from small and large facilities must be compared; small facilities are more likely to be flagged as outliers and their rates will be less stable over time. The situation is compounded for low-rate QIs, for which just one or two events can greatly influence a facility's position relative to peer comparison facilities.

Reporting systems should include adjusted QI rates and facility-specific confidence intervals around these rates. The construction of intervals should be based on conscious decisions about the intended sensitivity and specificity of the QIs. Adjusting QI rates for facility size may produce greater sensitivity but also increase the potential for false positives. The specificity of the QIs might be increased by use of methods such as empirical Bayes estimation techniques (Berlowitz et al., 2001), but perhaps at the price of lower sensitivity. The rates for rare or infrequent events such as fecal impaction or pressure sores could be treated as sentinel events, for which every incident is considered a quality problem. These are not just technical issues; they have important implications for policy and stakeholder use of the QIs. False positives on problem-oriented QIs can engender provider resentment and divert consumer attention away from true positives. Lack of sensitivity can result in provider complacency or give consumers a false sense of confidence about facility quality.

Investigate the Dimensionality and Theoretical Structure of the Quality Indicators

The lack of evidence for dimensionality of the QIs raises serious questions about their construct validity as well as their capacity to characterize facility performance in a parsimonious way. Facilities may concentrate their efforts in certain care dimensions in response to resource constraints or staff capabilities, or they may approach dimensions of care separately because of tunnel vision about care processes and outcomes. Nonetheless, we would expect clinically related indicators to be correlated with each other both at the resident and facility level. Thus, a facility doing poorly on the care processes of skin care, nutrition, or toileting would be expected to have higher rates of skin breakdown or pressure sores than facilities offering better care in these areas. Moreover, clinically related outcomes such as weight loss, incontinence, and pressure sores should be correlated with each other and would be expected to be higher in facilities with poor care processes in these areas. From a positive perspective, residents receiving nursing rehabilitation services should show higher rates of functional improvement or at least less functional decline.

One method of exploring the dimensionality of the indicators would be to specify and test causal models between processes and outcomes of care. Researchers have had some success in identifying dimensionality in nursing home survey deficiencies (Mullan & Harrington, 2001), although admittedly these data are collected in a different manner than QIs. Relationships between care processes and outcomes, and structural features, such as staffing, could be examined as well. Sainfort and colleagues (Ramsay, Sainfort, & Zimmerman, 1995; Sainfort, Ramsay, Ferreira, & Mezghani, 1994) proposed and tested a causal model of long-term-care quality. Although results from their analysis were disappointing, they offered some evidence for the effect of organizational factors on nursing home QIs.

Refine Risk Adjustment Methods and Treat Risk More Broadly in the Context of Care

Risk adjustment methods used thus far have tried to remove variance in quality that can be attributed to differences in facility populations. Removing risk-related variance is controversial because of difficulties in selecting risk factors that are not themselves related to care quality. It also relegates risk to a confounding variable rather than treating it as an integral part of the care process. Nursing home care is replete with complex interactions between risk, care processes, and outcomes. Resident Assessment Protocols (RAPs), for example, have a series of MDS-based risk factors associated with clinical outcomes that are supposed to trigger provider care planning. Providers' responses to high- and low-risk residents should profoundly influence care outcomes. The causal models of care processes and outcomes should specify and test interactions by resident risk. For example, a risk factor such as cognitive impairment may interact with care processes (e.g., physical restraints or psychotropic medications), heightening their impact on care outcomes (e.g., falls or incidence of incontinence).

In addition, risk characteristics of the facility's resident population may interact with the context of care delivery. For example, facilities specializing in postacute populations may be more effective with high-risk residents whereas conventional nursing facilities may do better caring for lower risk residents (Arling et al., 1997). Multilevel modeling, which takes into account resident- and facility-level variables simultaneously, can be useful in isolating effects of the context or setting in which care is being delivered, the composition of patients in that setting (e.g., risk or acuity), and the interactions between context and composition (Duncan, Jones, & Moon, 1998). Indeed, there is evidence that the multilevel nature of QI rates (i.e., residents nested within facilities and facilities nested within states or regions) calls for hierarchical linear modeling in risk adjusting the QIs and identifying facility outliers (Berlowitz et al., 2002).

Strengthen Quality Indicator Reporting Systems and Tailor Them to Stakeholders' Needs

Some improvements in QI reporting apply equally to all stakeholders (i.e., consumers, providers, and regulators), whereas others should be tailored to specific stakeholder needs. Absolute standards or benchmarks would have universal appeal. Standards could be determined with clinical panels such as those used by Rantz, Petroski, and colleagues (1997). Absolute standards would obviate some of the statistical issues associated with the peer comparison approach; they could be defined to be insensitive to variation in sample sizes. Another benchmarking approach might be to identify best-performing facilities and use their QI rates as benchmarks for the rest of the industry. The challenge lies in identifying excellent performers on most if not all QI dimensions that are consistent over time. Facility QI rates have not been shown to be highly stable, although it is unclear whether this is due to measurement error or whether facility quality fluctuates (Rantz et al., 2004).

The QIs have focused for the most part on processes and outcomes of care. However, many stakeholders are concerned about structural indicators of quality, such as nurse staffing levels (Harrington, O'Meara, et al., 2003; Saliba & Schnelle, 2002). Although studies into the relationship between staffing and care quality have yielded mixed findings, there is general agreement that the number of members on the nursing staff affects care quality (Abt Associates, 2001; R. L. Kane, 2004; Schnelle, Simmons, et al., 2004). Moreover, staff skill mix may relate to different quality-of-care dimensions; clinical outcomes may be more sensitive to the number of registered nurses, whereas quality-of-life indicators may be more sensitive to the number of nursing assistants (Harrington, Zimmerman, Karon, Robinson, & Beutel, 2000).

Beyond the simple numbers and skill mix of staff, a reporting system might consider factors such as staff training, turnover and retention, and supervision. Other organizational features might include the privacy of living quarters (single or semiprivate living areas or bathrooms) or the presence of specialty-care units such as units for individuals with Alzheimer's disease. Finally, reporting systems might incorporate results from consumer satisfaction surveys. Several states have consumer satisfaction systems in place, although they vary in the types and quality of information (Lowe, Lucas, Castle, Robinson, & Crystal, 2003). The California nursing home Web site (California HealthCare Foundation, 2004) is a good example of a multidimensional reporting system that incorporates information about QIs, nursing home survey results, staffing, and other aspects of care. The Nursing Home Compare Web site also combines information about the QMs with staffing and survey deficiency data (CMS, 2004).

Differences in the information needs of stakeholders make multiple reporting systems inevitable. Consumers may want an intuitive interface such as a rating system in which stars are assigned to facilities; some consumers have trouble interpreting more complex figures (Goldstein & Fyock, 2001). Better techniques that capture consumers' attention might be used. For example, one might first ask them which quality elements are of greatest concern and display facilities based on the consumers' interests. In contrast, regulators who work continuously with the QIs and who use them for targeting residents as well as facilities will require a much more extensive system. Providers that apply the indicators to internal quality improvement may place more emphasis on processes of care or longitudinal reports unique to the facility. To perform effective work-process control, facilities will need more detailed data systems that can be augmented by the QIs. Whatever the internal reporting system, it should have a rigorous methodological infrastructure with controls for data reliability and statistical error.

Finally, widespread reporting of the QIs presents rich opportunities for natural experimentation. Building research and evaluation designs into QI applications can generate useful information about the validity of the indicators, their effects on provider behavior or consumer decision making, and their overall impact on quality of care.

References