Abstract
The 2007 Infectious Diseases Society of America (IDSA)/American Thoracic Society (ATS) guidelines proposed “minor” criteria to predict intensive care unit (ICU) admission in patients with community-acquired pneumonia. These criteria were based on expert opinion. Consequently, the authors of the guidelines asked investigators to determine whether the score could be simplified by excluding noncontributory variables.
Each IDSA/ATS minor criterion was validated using a random effects meta-analysis of seven studies. Variables present in <5% of cases or that were nonsignificantly associated with mortality/ICU admission were excluded. A simplified score excluding these variables was tested for prediction of mortality and ICU admission in an established database. Prediction was assessed using the area under the receiver operator characteristic curve (AUC).
Leukopenia (<4000 cells·mm−3), thrombocytopenia (<100 000 cells·mm−3) and hypothermia <36°C occurred in <5% of cases. A simplified score excluding these variables was performed similarly for prediction of mortality, AUC 0.77 (95% CI 0.73–0.81) versus 0.78 (95% CI 0.74–0.82) (p=0.9) and intensive care unit admission, AUC 0.85 (95% CI 0.82–0.87) versus 0.85 (95% CI 0.82–0.88) (p=0.9). Additional predictors suggested by the IDSA/ATS were associated with mortality and ICU admission, but only incorporating acidosis (pH <7.35) altered the AUC (0.82 (95% CI 0.78–0.86) (p=0.6) for mortality and 0.86 (95% CI 0.82–0.88) (p=0.8) for ICU admission). No improvements were statistically significant.
The IDSA/ATS criteria can be simplified by removing three infrequent variables.
Abstract
The IDSA/ATS criteria can be simplified by removing three infrequent variables http://ow.ly/s1dA1
Introduction
Community-acquired pneumonia (CAP) is a leading cause of death and a frequent indication for intensive care unit (ICU) admissions internationally [1]. Reducing mortality from CAP requires rapid identification of patients at risk and several scoring systems have been developed to identify severe CAP [2, 3].
In 2007 the Infectious Diseases Society of America (IDSA)/American Thoracic Society (ATS) guidelines recommended new criteria to define severe CAP and to guide ICU admission [1]. In addition to the universally accepted indications for ICU admission, requirement for mechanical ventilation or need for vasopressors, these guidelines suggested nine minor criteria with a recommendation that patients with three or more criteria be considered for ICU admission [1].
The IDSA/ATS criteria have now been validated for prediction of 30-day mortality, ICU admission and requirement for subsequent mechanical ventilation or vasopressor (MV/VS) use in several countries [4–10].
There are, however, several outstanding questions to address regarding the minor criteria. Unlike other severity scores, which were derived from independent predictors in multivariate analysis, the minor criteria were selected by expert opinion and their individual components have not been separately validated [1]. It has been suggested that some criteria may be stronger predictors than others, or that some criteria may be noncontributory to the overall score [1, 7, 8]. In addition, the 2007 guidelines specifically requested that a group of additional predictors, such as acidosis, hypoglycaemia and liver cirrhosis, be tested by adding to or substituting the existing criteria to determine whether they improve its accuracy [1].
A major problem with severity scores has been a failure to implement them in clinical practice [11, 12]. Complexity of scores makes implementation more difficult and, therefore, scores should be designed to contain the fewest components necessary to give accurate predictions [13]. Several scoring systems in other conditions have been successfully simplified to make them easier to use in clinical practice [2, 13].
Using a combination of meta-analysis and a large observational clinical study, this analysis aimed to improve the IDSA/ATS minor criteria by simplification: excluding infrequent and noncontributory variables, and testing additional variables to determine whether they improve the score. The study subsequently validated the resulting simplified IDSA/ATS criteria in an established observational database of patients with CAP.
Methods
The present study reports two analyses: first, an initial systematic review and meta-analysis evaluating the prognostic value of the individual components of the IDSA/ATS minor criteria. The second part of the analysis consists of validating a simplified IDSA/ATS minor criteria in a prospective observational database.
Systematic review and meta-analysis
The systematic review and meta-analysis was conducted and is reported according to MOOSE (meta-analysis of observational studies in epidemiology) guidelines [14].
Search criteria
A search of PubMed was performed using the following search terms: (“IDSA/ATS” or “IDSA” or “ATS” or “Minor criteria” or “major criteria” or “ICU” or “intensive care”) and (“Pneumonia” or “community-acquired pneumonia” or “CAP”).
The search included articles published between January 1980 and November 2012. No language criteria were applied. Full articles of all potentially appropriate abstracts were reviewed. Only peer-reviewed data were included; therefore, conference abstracts were excluded. The search was repeated in Embase to obtain any articles missed by the original search. The search strategy was supplemented by reviewing reference lists, bibliographies and the investigators’ files.
Data extraction
Two investigators independently reviewed abstracts to determine study eligibility. Irrelevant studies were excluded based on title and abstract review only. Data abstraction was conducted blind by two reviewers. Where appropriate, we contacted study authors to clarify inconsistencies or to obtain unpublished data.
Study inclusion and study quality assessment
Studies were considered eligible if they fulfilled the following criteria: original publications; inclusion of consecutive/unselected patients with CAP; radiographic confirmation of CAP and exclusion of non-CAP diagnoses, e.g. nonpneumonic exacerbation of chronic obstructive pulmonary disease; calculation of severity score based on admission data; and reported on one of the outcome measures in the study (mortality or ICU admission). As previously described, we used Hayden’s criteria to assess quality of studies [15].
Modification of the IDSA/ATS minor criteria
A priori, we determined that we would attempt to simplify the IDSA/ATS minor criteria by systematically excluding variables if any of the following criteria were met: variables not significantly associated with mortality (p>0.05) in the pooled analysis; variables not significantly associated with ICU admission (p>0.05) in the pooled analysis; and variables present in <5% of the study populations in either pooled analysis. A simplified score would therefore be developed for testing in an observational database.
The IDSA/ATS 2007 guidelines suggested additional criteria that should be tested to determine whether they improve the minor criteria: these were hypoglycaemia, hyperglycaemia, metabolic acidosis, elevated lactate, liver cirrhosis, acute alcohol ingestion or alcohol withdrawal and asplenia [1, 16]. The guidelines do not suggest a pre-specified cut-off for these variables and so cut-offs were identified based on the published literature (hypoglycaemia <4.4 mmol·L−1, hyperglycaemia >14 mol·L−1, pH <7.35 and hyponatraemia <130 mmol·L−1) [16–19]. Lactate was only measured routinely in one study hospital and so was excluded from this analysis due to missing data. Arterial blood gas results were assumed to be normal when they were not available. Variables significantly associated with mortality and requirement for MV/VS, and present in >5% of the study population were added to the IDSA criteria to create a modified score.
Validation of the modified score in an observational database
The simplified scoring system was validated in the Edinburgh pneumonia study database. This was a prospective observational study of patients with CAP conducted in Edinburgh, UK. The methodology of this study has been described previously [20]. As the IDSA/ATS criteria have been designed to guide ICU admission, we analysed a cohort of patients in which patients with “do not attempt resuscitation” orders and “not for ICU” orders were excluded, as previously described [4]. In addition, as the minor criteria are not useful in patients already receiving MV/VS support in the emergency department, this cohort excluded these patients (major criteria) [1].
Outcomes
In all analyses, the outcomes of interest were ICU admission, requirement for mechanical ventilation or vasopressor support (a surrogate of ICU admission) and mortality (30-day or in-hospital mortality depending on the study design).
Statistical analysis
Meta-analysis was conducted comparing the frequency of each IDSA/ATS minor criteria in the outcome group with the frequency in those without the outcome. Odds ratios were pooled using a Mantel–Haenszel random effects model. A random effects model was chosen due to the expected heterogeneity between studies. Statistical heterogeneity was assessed using Higgins’ I2 test [21]. In the observational database, sensitivity, specificity, positive likelihood ratio, negative likelihood ratio and area under the receiver operator characteristic curve (AUC) are presented. When interpreting likelihood ratios, it is generally agreed that a likelihood ratio >10 and a negative likelihood ratio <0.1 provide strong evidence to either rule in or rule out a diagnosis [22]. To identify the optimal cut-off point for discrimination we used Youden’s index. AUC values were compared using the method described by Hanley and McNeil [23]. Odds ratios were compared as described by Altman and Bland [24].
Analyses were conducted using SPSS version 21 for windows (SPSS inc, Chicago, IL, USA) and Review Manager version 5 (Cochrane Collaboration, Oxford, UK).
Results
The initial search retrieved 8827 publications. After exclusion of manuscripts based on abstract review alone, 30 manuscripts were potentially eligible for inclusion. After exclusion of studies that either did not report data or did not consider mortality or ICU admission as an outcome, seven studies were included in the final meta-analysis.
The size of included studies varied from 158 patients to 2413 patients. The ICU admission rate varied from 6.3% to 19.6%. All studies were observational; three were retrospective and four were prospective in design. All seven studies were designed specifically to evaluate the IDSA/ATS criteria. Three studies were regarded as high quality with a low risk of bias, two were of intermediate quality and two of lower quality. The characteristics of the included studies are shown in table 1.
Meta-analysis of individual components of the IDSA/ATS criteria
Individual components of the IDSA/ATS criteria were evaluated for their prediction of mortality and ICU admission. Three studies reported data for both mortality and ICU admission, two studies reported data only for mortality, and one study only reported data regarding ICU admission. One author responded to a request for additional data and, therefore, data for both mortality and ICU admission are included. Therefore, six studies contained valid data for the evaluation of mortality. These comprised 5686 patients. The most frequently positive criteria was multilobar shadowing (32.1% of patients) and the least frequent was thrombocytopenia (<100 000 cells·mm−3) (3.1% of patients). The highest odds ratio was for arterial oxygen tension (PaO2)/inspiratory oxygen fraction (FIO2) ratio ≤250 and the lowest odds ratio was for hypotension. All of the individual components were significantly associated with mortality (p<0.05). Significant (>50%) heterogeneity was present for all components except white blood cell count <4000 cells·mm−3 and thrombocytopenia. Heterogeneity was not resolved by exclusion of low quality studies or limiting to exclusively prospective studies.
For the prediction of ICU admission, there were five studies including 6240 patients. All of the variables were significantly associated with ICU admission (p<0.05) for all analyses. In these studies, the most frequently positive variable was also multilobar shadowing (30.5%), with the least frequent being white blood cell count <4000 cells·mm−3 (2.9%). The highest odds ratio was for hypothermia (<36°C) with the lowest odds ratio for thrombocytopenia. There was significant heterogeneity in analyses for respiratory rate, PaO2/FIO2 ratio ≤250, confusion, urea ≥20 mg·dL−1 and hypothermia. The meta-analysis results are summarised in table 2.
To determine whether the different criteria were of equal or unequal weight, odds ratios for each predictor were compared using interaction tests, which allow comparison of two estimates of effect [23]. In this case, interaction tests were used to determine whether there was a statistically significant increased risk of mortality or ICU admission with any one predictor over the others. Comparisons between the odds ratios demonstrated no statistically significant differences (p>0.05 for all comparisons), suggesting that the variables were not of unequal weight (table 2).
Evaluation of the additional IDSA/ATS criteria
There was a significant relationship between hypoglycaemia, hyponatraemia and acidosis with requirement for mechanical ventilation or vasopressor support. Hyponatraemia (8.5%) and metabolic acidosis (10.5%) were relatively common predictors while all other predictors were present in <5% of patients.
For prediction of 30-day mortality, hypo- and hyperglycaemia, hyponatraemia, metabolic acidosis and liver cirrhosis were all significantly associated with mortality, while there was no statistically significant association with asplenia and acute alcohol withdrawal, although these analyses were limited by low numbers of patients. These are shown in table 3.
Modification of the IDSA/ATS minor criteria
Based on table 3, white blood cell count, platelets and hypothermia contributed to <5% of cases in the analysis of both mortality and ICU admission. We hypothesised that exclusion of these infrequent predictors from the score would not impact its accuracy based on the AUC (table 4).
We compared the performance of the simplified and original minor criteria in the Edinburgh dataset. This dataset has been previously described. 1062 patients were included with a median age of 63 years (interquartile range 47–74 years); 513 (48.3%) patients were male. The 30-day mortality rate in this population was 4.5%, and 6.6% required mechanical ventilation or vasopressor support.
Comparing this simplified score to the original scoring system in the Edinburgh pneumonia dataset, the performance characteristics are shown in table 4. Simplification did not affect the overall performance of the score for predicting 30-day mortality (p=0.8), requirement for MV/VS (p=0.9) or ICU admission (p=0.8).
No significant improvements in the AUC were observed by incorporating any of the additional predictors (hypo- or hyperglycaemia, hyponatraemia, alcohol withdrawal, cirrhosis, or asplenia). The largest improvement in the AUC was observed with the addition of acidosis (table 5), which improved the AUC for mortality from 0.78 to 0.82; however, improvements were not statistically significant (p>0.05 for all comparisons).
In addition to showing equivalent AUC to the existing IDSA/ATS minor criteria, the simplified score was superior to the pneumonia severity index (PSI) (p=0.01), the CURB-65 (confusion, urea >7 mmol·L−1, respiratory rate ≥30 breaths per min, blood pressure <90 mmHg (systolic) or ≤60 mmHg (diastolic), age ≥65 years) score (p=0.008), CRB-65 (p=0.005) and the 2001 ATS minor criteria (p=0.001) for prediction of MV/VS. The AUC was equivalent to those of the SMART-COP (systolic blood pressure, multilobar chest radiography involvement, albumin, respiratory rate, tachycardia, confusion, oxygenation, pH) and España SCAP (Severe Community-Acquired Pneumonia) scores (p>0.05 for comparisons).
There were no statistically significant differences between any of these scores for prediction of mortality (p>0.05 for all comparisons). 29.6% of ICU admissions occurred between 24 and 72 h of admission, representing early deterioration in ward patients. The AUCs to identify this group of patients were original IDSA/ATS definition 0.84 (95% CI0.79–0.89), simplified IDSA/ATS score 0.83 (95% CI 0.78–0.88), and the simplified score including acidosis 0.84 (95% CI 0.79–0.89).
Selecting the optimal cut-off point for the IDSA/ATS criteria
Using likelihood ratios to determine the cut-off points at which the score was most likely to be clinically useful, the analysis was first performed for requirement for MV/VS. Each of the scores demonstrated a positive likelihood ratio of >10 at a cut-off of five criteria or more (fig. 1a), indicating a very high likelihood of the score being clinically useful to determine ICU admission. As a test to exclude the likelihood of requiring MV/VS, each set of criteria produced a likelihood ratio <0.1 using a cut-off of one of fewer criteria, indicating that these criteria would be likely to be useful to identify patients unlikely to require ICU admission (fig. 1b).
Performance characteristics of the Infectious Diseases Society of America (IDSA)/American Thoracic Society (ATS) criteria and modifications for predicting requirement for mechanical ventilation/vasopressor. a) Positive likelihood ratios for each score using a cut-off of ≥1, ≥2 and so on to define severe community-acquired pneumonia (CAP). b) Negative likelihood ratios for each score using the indicated cut-off or less to exclude severe CAP. The horizontal lines indicate the positive likelihood of 10 and negative likelihood of 0.1, which is typically held to give strong evidence to rule in or rule out a diagnosis [22]. c) The Youden index as a measure of overall discrimination for each cut-off. The cut-off with the highest index is the optimal cut-off in terms of sensitivity and specificity.
The analysis for ICU admission produced similar results, with negative likelihood ratios <0.1 at a cut-off of ≤1, and positive likelihood ratio's >10 associated with a cut-off of ≥5 for each of the scores. Similarly for mortality prediction, each of the scores gave a positive likelihood ratio >10 with a cut-off of ≥5. The negative likelihood ratio was 0.9 for the original IDSA/ATS definition using ≤1 criteria, and 0.8 for the modified IDSA/ATS with acidosis criteria.
Using the Youden index to identify the optimal cut-off point, three or more criteria were most discriminatory for both the original IDSA/ATS criteria and the modified version with acidosis. Two or more criteria were optimal for the simplified score (fig. 1c).
Discussion
This study has shown that the IDSA/ATS minor criteria can be simplified by removing three noncontributory variables without compromising its prognostic accuracy. Removing thrombocytopenia (<100 000 cells·mm−3), white blood cell count <4000 cells·mm−3 and hypothermia (<36°C), which all occur in <5% of CAP cases, resulted in no significant loss of diagnostic performance, but may make the score easier to use in clinical practice.
This study used both a systematic review and meta-analysis, and an observational study of patients with CAP to comprehensively investigate the IDSA/ATS minor criteria. Previous studies had not validated the individual components of the IDSA/ATS criteria and had not investigated the additional criteria, such as acidosis, hyperglycaemia and hyponatraemia, recommended by the guidelines authors. No previous study has investigated whether the criteria could be simplified by removing noncontributory variables [1].
Our meta-analysis has validated the individual components of the 2007 IDSA/ATS minor criteria, showing that each criterion was associated with increased mortality and requirement for ICU admission in CAP. The IDSA/ATS guideline committee had asked authors to investigate whether the effect of some components was greater than others, and should therefore receive more weight within the score, as is the case for some others scores, such as the PSI or SMART-COP [17, 25, 26]. Several individual studies have argued that the minor criteria are of unequal weight; for example, Liapikou et al. [7] found that mental confusion and leukopenia had the strongest association with mortality. They could not demonstrate an association between hypotension, thrombocytopenia or multilobar involvement and mortality [7]. Phua et al. [5] found that each minor criterion was predictive of mortality and that the presence of PaO2/FiO2 ≤250 mmHg and confusion had the strongest association with mortality. Furthermore, Guo et al. [8] showed that minor criteria of the IDSA/ATS criteria for severe CAP were of unequal weight in predicting mortality. They concluded that PaO2/FiO2 ≤250 mmHg, confusion and uraemia were most strongly associated with mortality, and that an association between leukopenia, hypothermia and hypotension and mortality could not be demonstrated [8]. Our analysis suggests that no predictors were significantly or consistently associated with a greater risk compared with the others, showing the value of a meta-analysis approach where individual study results are conflicting.
Of the additional criteria investigated, acidosis and hypoglycaemia were most strongly associated with mortality and MV/VS, but addition of these variables to the minor criteria produced little or no improvement in the AUC. Only the addition of acidosis showed some improvement in the AUC, but this did not reach statistical significance. Further studies should evaluate whether addition of this parameter may improve the score.
The IDSA/ATS criteria recommended consideration of ICU admission for patients with a score of ≥3, but no other guidelines or manuscripts have provided guidance on how the criteria may be used in clinical practice. Based on our score performance data, we can make some cautious recommendations about how this score might be used clinically. Our data suggest that patients with 0–1 criteria are at low risk of ICU admission and mortality, and may be suitable for outpatient or ward-level care. Patients with 2–4 criteria are at increased risk of ICU admission and mortality, but admitting all of this very large group of patients in level 2/level 3 care would be impractical in most healthcare settings. Therefore, in this group, additional predictors, biomarkers and clinical judgement should be taken into account. In patients with ≥5 criteria (using either the original or modified scores) the risk of ICU admission and mortality is very high, and this group should be strongly recommended for level 2/level 3 care, or have treatment restrictions in place if this is inappropriate.
These findings and the modified minor criteria described in this manuscript will require further independent validation studies. The IDSA/ATS guidelines will shortly be updated and policymakers may wish to consider these results in revising the minor criteria.
It is well recognised that simple scores composed of only a few variables can be equivalent to more complex scores, for example, the five-variable CURB-65 score is equivalent to the 20-component PSI for predicting 30-day mortality in CAP [2]. A major problem with severity scores has been in implementing them in busy hospital settings. For example, a survey of 536 physicians in Australia found that fewer than half of physicians used severity scores and that only ∼20% of physicians could accurately calculate the PSI or CURB-65 scores [27]. It is logical that simpler scores will be easier to calculate and, therefore, to use in clinical practice. An excellent example of a complex score being simplified is the pulmonary embolism severity index (PESI). This was developed to estimate the 30-day mortality in patients with acute pulmonary embolism. An initial complex risk stratification score containing 11 different variables, each carrying a different weight, was initially derived and validated [28, 29]. Recognising that this was not ideal for use in a busy hospital environment, Jiménez et al. [13] proposed a simplified version of the PESI score. They excluded noncontributory variables, resulting in a score of only six variables. They then validated their simplified score in >17 000 patients, showing that it performed similarly to the more complex score [13].
Similarly, our study has demonstrated that a simplified criteria consists of respiratory rate ≥30 breaths per min, PaO2/FIO2 ratio ≤250, multilobar infiltrates, confusion/disorientation, uraemia (blood urea nitrogen level ≥20 mg·dL−1) and systolic blood pressure <90 mmHg performs similarly to the existing IDSA/ATS minor criteria.
Limitations to this study are acknowledged. There was significant heterogeneity in estimates of effect sizes in the meta-analysis resulting from different study designs and differences in patient populations. It is well recognised, for example, that criteria for ICU admission in the USA are very different from those in Europe or the UK [26]. In addition, we were unable to evaluate lactate in addition to the IDSA/ATS criteria, as this was not measured in all study patients.
In conclusion, our study has simplified the IDSA/ATS criteria by removing three infrequent variables, resulting in a six-point score with the same prognostic accuracy. This simplified score may be easier to remember and to implement by clinicians in a busy hospital environment.
Footnotes
Earn CME accreditation by answering questions about this article. You will find these at the back of the printed copy of this issue or online at www.erj.ersjournals.com/site/misc/cmeinfo.xhtml
Conflict of interest: None declared.
- Received May 25, 2013.
- Accepted September 16, 2013.
- ©ERS 2014
References
