Abstract
Comparative outcomes data are widely used to monitor quality of care in the cardiovascular area; little is available in the respiratory field. We applied validated methods to compare hospital outcomes for chronic obstructive pulmonary disease (COPD) exacerbation.
From the hospital information system, we selected all hospital admissions for COPD exacerbation in Rome (for 2001–2005). Vital status within 30 days was obtained from the municipality mortality register. Each hospital was compared to a pool of hospitals with the lowest adjusted mortality rate (the benchmark). Age, sex and several potential clinical predictors were covariates in logistic regression analysis.
12,756 exacerbated COPD patients were analysed (mean age 74 yrs, 71% males). Diabetes, hypertension, ischaemic heart disease, heart failure and arrhythmia were the most common coexisting conditions. The average crude mortality in the benchmark group was 3.8%; in the remaining population it was 7.5% (range 5.2–17.2%). In comparison with the benchmark, the relative risk of 30-day mortality varied widely across the hospitals (range 1.5–5.9%).
A large variability in 30-day mortality after COPD exacerbation exists even considering patients’ characteristics. Although these results do not detect mechanisms related to worse outcomes, they may be useful to stimulate providers to revision and improvement of COPD care management.
- Chronic obstructive pulmonary disease exacerbation
- health information systems
- health outcomes
- mortality
- quality of care
- report cards
In the last decades, there has been a proliferation of data on comparative performance of healthcare providers both in the USA and in Europe, on the assumption that measuring quality of care is a key component in improving care. The best-known example is the publication of the hospital report cards for cardiac surgery in New York State since the late 1980s, followed by similar programmes in other countries 1–3. The publication of hospital outcome data has become progressively more popular as an answer to society's increasing consensus on general “right to know”. However, there is still much debate on the actual impact. While it has been recognised that publicly releasing performance data stimulates quality improvement activity at the hospital level, the effect of public reporting on effectiveness, safety and patient-centredness remains uncertain 4, 5. Outcome data are proved to be useful for research and monitoring trends within an organisation. However, it has been underlined that, without further analysis, these data may wrongly penalise doctors and managers, and research efforts should rather focus on measures of adherence to clinical and managerial standards than on comparative outcomes 6.
While large experience exists in measuring and publishing comparative outcomes data in the cardiovascular area, interest is now growing in implementing this methodology in other fields. Chronic obstructive pulmonary disease (COPD) is one of the most common causes of hospital admissions in Italy: about 100,000 patients aged ≥65 yrs are hospitalised every year 7. The health and social economic burden of COPD is serious and increasing over time, with hospitalisation for acute exacerbations being the major component 8. In-hospital mortality is high and patients experience extended lengths of stay 7, 9. Current guidelines identify evidence-based management strategies to be implemented at hospital level to improve outcomes for exacerbated COPD patients, but little is known on their current application and on factors potentially influencing outcomes 10. As a unique and stimulating initiative in Europe, the UK national audit programme for acute exacerbation of COPD found wide variability in 90-day mortality unexplained by clinical factors, and demonstrated an association between high-quality specialised hospital care and better outcomes 11, 12.
Within the framework of a National Health Service programme on healthcare outcomes, standardised methodology to produce comparative hospital performance data for a large range of medical and surgical conditions has been developed in Italy in recent years 13. In the present study, we compare hospitals in terms of mortality for hospitalised patients with acute COPD exacerbation, using data from the regional electronic health registries.
METHODS
Source of data
Discharge abstracts, from both public and private hospitals, are routinely collected by the Hospital Information System of the Lazio Region, Italy, where Rome is located (about 2.7 million residents). They contain patient demographic data, admission and discharge dates, up to six discharge diagnoses (International Classification of Disease, 9th revision, Clinical Modification (ICD-9-CM)), up to six clinical procedures, modality of admission (emergency versus scheduled admission) and status at discharge (alive, dead or transferred to other hospital). The municipal registry maintains records on all official residents of Rome, including vital status and date and place of death. Individual codes allow identification of people in different datasets.
Selection of episodes of care
Overall, we identified 26,370 acute episodes of care in the period January 1, 2001 to September 30, 2005 for residents aged ≥35 yrs in Rome with acute exacerbation of COPD (ICD-9-CM codes: main diagnosis 490, 491, 492, 494 or 496). We also included episodes with main diagnosis of acute respiratory failure (ICD-9-CM codes 518.81 or 518.82) or dyspnoea and other respiratory symptoms (ICD-9-CM codes 786.0, 786.2 or 786.4) and secondary diagnosis of COPD. In case of multiple episodes in a 90-day period, we included only the first episode, assuming the subsequent admissions to be clinically related. All patients were initially assessed at the emergency department (ED). Those patients with secondary diagnosis of major trauma or major surgical operations during the index event were excluded. At the end of the selection procedures, there was a total of 12,756 episodes of care (index events), corresponding to 10,124 patients. Details on selection procedures and ICD-9-CM codes are in the online Appendix.
Individual characteristics
On the basis of ICD-9-CM codes, we identified three categories of “acute respiratory conditions in the index event” as a proxy of severity of the COPD exacerbation: 1) acute respiratory failure, 2) dyspnoea or other respiratory symptoms, and 3) other acute respiratory conditions including infections. Following the enhanced Elixhauser AHRQ-Web-ICD-9-CM coding algorithm, we defined chronic coexisting conditions (comorbidities) that can influence the prognosis 14. We defined comorbidities both in the index event and in the hospital admissions of the previous 4 yrs. In order to deal with the limit of administrative data in distinguishing present-on-admission diagnoses from other acute events potentially related to the care delivered, we adopted specific coding algorithms aimed at defining only chronic conditions, separately for the index and the previous hospital admissions. In the index hospitalisation coding algorithm, codes related to acute medical events that could be complications of care were not included in the definition of chronic comorbidities. As a proxy of severity, hospital utilisation for acute COPD exacerbation in the preceding 4 yrs was also examined. For details of selection procedures and ICD-9-CM codes, see the online Appendix.
Hospital characteristics
Patients were treated in 21 public hospitals admitting acute medical cases (20 in the city of Rome, one outside the city but in a nearby community). A total of 15 were public hospitals, three were teaching hospitals and three were privately funded. Apart from four of the hospitals, they all had an ED. Only six hospitals had a pneumological ward.
Outcome
Vital status at the end of 30 days after the admission date was evaluated with a linkage procedure with the municipal registry. The outcome was termed “30-day mortality”.
Statistical analysis
Age was subdivided into classes (35–64, 65–74, 75–84 and ≥85 yrs). Acute respiratory conditions in index events and hospitalisation for COPD in the previous 4 yrs were dummy variables (having or not having the condition). Similarly, for each comorbidity, we created a binary variable indicating the presence, separately at the index event or in the hospitalisations in the previous 4-yr period.
The analyses were conducted in steps, as follows.
1) We performed a logistic regression analysis (calculating odds ratios (ORs)) to develop the best predictive model for the outcome without considering hospitals. The initial variables were sex, age, acute respiratory conditions in the index event, COPD hospitalisation in the previous 4 yrs and chronic comorbidities. These variables were all included in the model and a backward stepwise procedure was used to discard those variables that were not associated with the outcome (pstay = 0.05, pentry = 0.10). Comorbidities reported in the hospital admissions of the previous 4 yrs were “forced” into the models (even if not statistically significant) when the corresponding ones registered in the index admission were significant predictors of the outcome. This procedure partially took into account the known potential biased relationship with outcomes of selected diagnoses registered in the index admissions 15.
2) In order to compare hospitals, we ran a random effect logistic model (to take into account the possible effect of clustering since we considered episodes of care) including the variables resulting from step 1 and a dummy variable for each hospital. We included in the reference group (the benchmark) the hospitals with the lowest OR. The procedure to define the benchmark was the following. First, 20 hospital dummies were added to the model and the corresponding adjusted ORs were estimated. At this step, the hospital with the highest number of patients was chosen as the reference category. Secondly, after ranking all hospitals by adjusted ORs, the five hospitals with the lowest adjusted ORs were selected as the reference group. This group was selected by an iterative procedure that at each step included one hospital in the reference group. The procedure stopped when the hospital to be included in the reference group was significantly different (p<0.10) from the benchmark defined in the previous step. A Chi-squared test was performed to quantify the global heterogeneneity between hospitals in mortality rates (p<0.001).
3) As a final step, in order to provide a meaningful summary of the results, the ORs were then transformed into relative risks (RRs) following the formula:
where p0 represents mortality (percentage) in the reference group (benchmark).
For each hospital, the risk-adjusted mortality rate was calculated by multiplying the average mortality in the benchmark by the RR.
Several sensitivity analyses were performed to evaluate the robustness of our results: 1) excluding all “acute respiratory conditions in the index event” from the multivariate models as potentially involved in the clinical pathway or emerged during the admission; 2) including all hospital admissions in the cohort (without the restriction to the first hospital admission in the case of multiple admissions in a 90-day period); 3) testing the effect modification by sex using an interaction term in the regression model and the likelihood ratio test under the hypothesis that the relationship between hospital and mortality varies between sexes; and 4) analysing a restricted cohort of patients with acute respiratory failure to improve the specificity of case definition.
Datasets were managed and analysed using SAS 8.1 (SAS Institute, Inc., Cary, NC, USA). All p-values reported are two-sided.
RESULTS
Table 1⇓ summarises the characteristics of the study population. Data for 12,756 episodes of acute COPD exacerbations were analysed. Most were males aged ≥75 yrs. About 45% reported at least one “acute respiratory conditions in the index event”. Diabetes, hypertension, ischaemic heart disease, heart failure/cor pulmonale and arrhythmia were the most common coexisting conditions. Prevalence of comorbidities in hospitalisations in the previous 4 yrs tended to be lower than in the index event, with the exception of other chronic heart disease, chronic digestive disease and chronic respiratory disease other than COPD. Statistically significant differences in various items were found between sexes.
Characteristics of the study population
Table 2⇓ shows the crude and adjusted association between several variables and 30-day mortality. Increasing age and acute respiratory conditions were significant predictors of the outcome, while hospitalisation for an acute COPD exacerbation in the previous 4 yrs was a protective factor. Among the comorbidities in the index event, some were strong predictors (heart failure/cor pulmonale, vascular disease including cerebrovascular, chronic renal disease, neurological and muscular disorders, psychiatric disease and cancer) while others had a protective effect (hypertension and thyroid disease). Similar results were obtained for comorbidities in the hospitalisations of the previous 4 yrs.
Association of individual characteristics with 30-day mortality
Table 3⇓ and figure 1⇓ show the comparative performance of the 21 hospitals. Five hospitals were in the pooled benchmark (two privately funded, three without ED, three without a pneumological ward), with a total of 1,689 episodes of care and a 30-day mortality of 3.8%. The average crude mortality in the remaining population was 7.5% (range 5.2–17.2%). In comparison with the benchmark, the adjusted 30-day mortality varied widely across the remaining 16 hospitals (ranging from RR = 1.5 in hospital number 6 (95% CI 0.9–2.4) to RR = 5.9 (95% CI 3.8–8.6) in hospital number 21). For nine hospitals, mortality was more than 2.5-fold higher than the benchmark. Table 3⇓ also shows the RRs and the ranking order obtained from the models with exclusion of “acute respiratory conditions in the index event including infections”: the benchmark and the RRs were substantially similar to the main analysis. The interaction term for sex was not statistically significant (p = 0.92). A similar ranking order was obtained from models including all hospital admissions in the cohort. The analysis from the subcohort of cases with acute respiratory failure (33% of the total, mortality 12.0%) showed a large variability across hospitals as well.
Risk-adjusted 30-day mortality rates (with 95% CI) across hospitals for patients with acute exacerbation of chronic obstructive pulmonary disease in Rome, Italy, in 2001–2005. Rates derived from random effect model 1. ·······: average mortality in the benchmark (3.8%); - - - - -: average mortality in the remaining population (7.5%).
30-day mortality for patients with acute exacerbation of chronic obstructive pulmonary disease (COPD): comparative performance of hospitals
DISCUSSION
We found a wide variability in 30-day mortality after acute COPD exacerbation among different hospitals in Rome. Hospitals with worse performance in the crude comparison were confirmed as “high outliers” in the adjusted analysis, suggesting that heterogeneity in the case mix did not fully explain the observed differences. Other unidentified factors, such as hospital-level variables, might play an important role. The high mortality after acute COPD exacerbation is in keeping with results from previous studies where short-term mortality ranged from 2 to 14% 9, 16.
While many professional associations and government institutions have recognised the importance of measuring quality of care, a debate is still ongoing about the role of process versus outcome indicators. On the one hand, guideline-based process measures are considered important means of assessing quality of care 6; on the other hand, several studies underline the potential of measuring and publically reporting comparative outcome measures 4, 5. In the UK, three main reasons to support the policy of publishing outcome data have been suggested: to stimulate action, to promote public trust, and to support patient choice 17. Even if rigorous evidence of the effects of public disclosure of comparative outcomes data is scanty, it has been found that the publication of health outcome data stimulates quality improvement activities at the hospital level 4. Our intent is to update this analysis periodically within the framework of a comprehensive “performance reporting system” in Lazio Region, based on a regional legislation 18. We aim to publish results and discuss them with clinicians and managers in order to promote activities of clinical and organisational audit and address problems in hospitals presenting critical results.
Despite their increasing use and acceptance, outcome report cards have their critics. The statistical methodology is fundamental: a poor validity of public report cards has a potentially devastating impact on an individual’s career or a hospital's reputation 19, 20. Moreover, excluding important prognostic factors from the models may result in misclassification or mislabelling of hospital performance 21. Recently, it was found that hospital rankings may change considerably on the basis of the risk-adjustment models used and that the classification of “outlier status” depends on the chosen threshold for statistical significance 22, 23. In our study, a less conservative approach, using p<0.1, could have implied more high and low outliers.
In the UK, all units admitting acute medical cases were involved in a multidisciplinary study that started in 2001 11, 12. It represents an interesting example of a prospective audit tool at national level for routine clinical evaluation of the process and outcome of acute care for hospitalised patients. This programme showed a wide variability in standards of care for in-patients with COPD, despite the publication of management guidelines. Units with more specialists and better care had lower mortality, smaller units had worse results and patients receiving specialist care were more likely to be given interventions of proven efficacy 11, 12. In Italy, admission to the pneumological ward was associated with better outcome in a study on COPD 24. However, adequate resources per se may not be sufficient markers of quality. We hypothesise that elements of variability across hospitals in our study could be both structurally related factors (i.e. the size, the complexity of ED services, the presence of a specialist respiratory ward, the numbers of specialists, the availability of invasive ventilation or noninvasive ventilation instruments) and organisation of care-related factors (i.e. specialist triage, integrated admission policy, access to proven-efficacy interventions like the specialist care). However, the aim of this study was to identify critical performance; exploring reasons for poor performances and testing possible different hypotheses represent the next steps.
When profiling hospital performance, it is necessary to control for pre-admission predictors of the outcome. Risk adjustment is utilised to control confounding factors due to severity and comorbidity 1, 2, 15. We followed a validated methodology to define the patients’ case mix 14. The coding comorbidity algorithm for the index event allowed analysis only of patient factors on admission or close to admission as case mix predictors, excluding events potentially on the causal pathway between hospital and outcome. Even if detailed clinical data on patient risk factors collected from chart reviews is considered the “gold standard”, several comparative studies demonstrated a good correlation between risk-adjusted outcomes obtained from clinical data versus administrative datasets 25, 26. The empirical approach we used to define severity and adjust for it allowed the identification and control of confounding factors according to their specific relationship with the outcome in the population. Any risk adjustment function, in fact, should be time and population specific 27.
After adjustment for difference in case mix, hospitals treating sicker patients are reassured that they will not be penalised 1, 2, 27. In general, hospitals are compared either with an average-mortality institution or with the average performance in the entire population. All hospitals need to be compared to the same benchmark. However, there are questions about which is the most appropriate benchmark 19. Moreover, the choice of benchmark may have potentially different important implications for patients, providers and policy makers. Historically, acute myocardial infarction (AMI) report cards used indirect standardisation to compare hospitals to an average performer 2, 21. However, using peer-group-defined benchmarks to compare hospitals sharing similar contextual and organisational characteristics might result in a better way to stimulate quality improvement among providers, while in the case of public reporting a comparison within peer-groups might be less relevant 19, 20.
The role of comorbidities in outcomes among COPD patients is not fully understood. Comorbidities such as cardiovascular disease and lung cancer are the leading causes of mortality in COPD patients, but the underlying mechanism of association is not clear 8, 28. Even the definition of comorbidities is problematic in COPD patients, as certain coexisting illnesses may be a consequence of the patient's underlying COPD. A greater number of comorbidities are associated with in-hospital mortality in COPD patients admitted to hospitals 8, 9, among them diabetes mellitus and congestive heart failure. Our findings confirmed previous results and add information on the underlying complex mechanisms.
We found that COPD hospitalisation in the previous 4 yrs is a protective factor. It seems a paradoxical result since other studies showed the opposite 9. However, it needs to be considered that information on previous COPD was available only for those patients surviving at the time of study inclusion, thus introducing a potential bias in the comparison between individuals with or without past COPD exacerbation. Otherwise, previous COPD hospitalisation could be considered a marker of more frequent contacts with specialist care and access to appropriate treatment like corticosteroids or long-acting bronchodilators, which are associated with lower mortality 10.
The strengths of this study are the population-based design, the numbers and the validated algorithm for variable definitions. This is the first large study in Europe comparing hospital performance in relation to mortality after acute COPD exacerbation by applying standardised methodology. The major limitation is the accuracy of ICD-9-CM coding. Unfortunately, we are not able to validate the diagnosis of COPD by evaluating clinical/functional variables and addressing the potential miscoding across hospitals. In studies like this, we identified COPD patients by using the main diagnosis; however, we cannot exclude inclusion of non-COPD patients. The sex distribution (41% females), slightly different from figures observed for national COPD prevalence (7.3% among males, 5.0% among females) 29, seems to support this hypothesis. However, factors influencing hospital admissions (i.e. severity of disease, adherence to therapy, access to preventive services/primary care) have been shown to vary according to sex for many conditions and may have played a role in our study. As a partial support for reliability of the adopted case definition, in a cohort study of 500 patients resident in Rome with a diagnosis of COPD based on clinical data, we found good correlation between severity of COPD and hospitalisation rate (data not published). As an attempt to minimise potential case misclassification, we only included patients who were admitted through the ED. We also made further sensitivity analyses to improve specificity of case definition, but the main result of this study was confirmed. Finally, administrative data did not allow insight into clinical or physiological factors. However, important clinical determinants of outcomes after COPD were found to explain only 15% of the variability in mortality across institutions 12, and we trust the known good performance of administrative data for studies on outcomes 26, 27. Residual confounding bias for unmeasured factors should be considered as a potential limitation of the study 30.
In conclusion, the wide variation of mortality after COPD among hospitals, only partially explained by the clinical case mix of admitted patients, is unacceptable. Public disclosure of these results should stimulate clinicians and managers to investigate the underlying mechanisms in order to improve the quality of care. The large number of patients admitted with COPD and the high mortality justify further studies to address which aspects of the organisation of care may be associated with worse outcomes in our country.
Statement of interest
None declared.
Acknowledgments
The authors are grateful to R. Macci and S. Magliolo (both Agency for Public Health, Lazio Region, Rome, Italy) for their support in finding the cited articles.
Footnotes
-
This article has supplementary material accessible from www.erj.ersjournals.com
- Received April 4, 2009.
- Accepted September 28, 2009.
- © ERS Journals Ltd
References
