Abstract
Background Patients who present to an emergency department (ED) with respiratory symptoms are often conservatively triaged in favour of hospitalisation. We sought to determine if an inflammatory biomarker panel that identifies the host response better predicts hospitalisation in order to improve the precision of clinical decision making in the ED.
Methods From April 2020 to March 2021, plasma samples of 641 patients with symptoms of respiratory illness were collected from EDs in an international multicentre study: Canada (n=310), Italy (n=131) and Brazil (n=200). Patients were followed prospectively for 28 days. Subgroup analysis was conducted on confirmed coronavirus disease 2019 (COVID-19) patients (n=245). An inflammatory profile was determined using a rapid, 50-min, biomarker panel (RALI-Dx (Rapid Acute Lung Injury Diagnostic)), which measures interleukin (IL)-6, IL-8, IL-10, soluble tumour necrosis factor receptor 1 (sTNFR1) and soluble triggering receptor expressed on myeloid cells 1 (sTREM1).
Results RALI-Dx biomarkers were significantly elevated in patients who required hospitalisation across all three sites. A machine learning algorithm that was applied to predict hospitalisation using RALI-Dx biomarkers had a mean±sd area under the receiver operating characteristic curve of 76±6% (Canada), 84±4% (Italy) and 86±3% (Brazil). Model performance was 82±3% for COVID-19 patients and 87±7% for patients with a confirmed pneumonia diagnosis.
Conclusions The rapid diagnostic biomarker panel accurately identified the need for inpatient care in patients presenting with respiratory symptoms, including COVID-19. The RALI-Dx test is broadly and easily applicable across many jurisdictions, and represents an important diagnostic adjunct to advance ED decision-making protocols.
Abstract
The RALI-Dx assay is a biomarker-based approach to emergency department triage for patients with respiratory illness and is superior to conventional strategies. This diagnostic test will help to optimise the utilisation of scarce healthcare resources. https://bit.ly/3PVjnYp
Introduction
Symptoms of respiratory tract infection are a common cause of emergency department (ED) visits. During 2018–2019 in Canada, more than 200 000 ED visits were associated with respiratory illness [1]. This demand is traditionally increased in the seasonal flu months and has been further complicated in the coronavirus disease 2019 (COVID-19) pandemic [1], where ED visits for respiratory illness have dramatically increased to the point of repeatedly overwhelming healthcare systems worldwide. Current diagnostic tests that focus on identifying the underlying aetiological viral agent are useful, but are unable to assess the severity of disease and do not capture the host inflammatory response, thus failing to measure the biological indicators of patients that require inpatient treatment [2]. Therefore, there is a critical need to develop diagnostic tools that more accurately determine which symptomatic patients have an exaggerated underlying host inflammatory response and should be considered for inpatient treatment following ED presentation.
To assess whether a biomarker panel could provide insight into the host inflammatory response and subsequent need for hospitalisation, we tested the RALI-Dx (Rapid Acute Lung Injury Diagnostic) assay. This is a rapid (<50 min), multiplexed immunoassay that quantifies interleukin-6 (IL)-6, IL-8, IL-10, soluble tumour necrosis factor receptor 1 (sTNFR1) and soluble triggering receptor expressed on myeloid cells 1 (sTREM1). Individually, these immune activation markers have been correlated with a heightened host response to respiratory tract infections, including clinical deterioration and the need for mechanical ventilation in countries with high COVID-19 infection rates [3–11]. To further evaluate the inflammatory profile provided by RALI-Dx, we applied a novel machine learning approach, XGBoost, to the biomarker output to derive an algorithm that predicts the need for hospitalisation that would theoretically help to guide decision making in the ED.
This study presents the RALI-Dx biomarker results in patients who presented to an ED with symptoms of respiratory infection from 2020 to 2021 in three countries: Canada, Italy and Brazil. RALI-Dx biomarkers measured at clinical presentation were compared with disposition from the ED and illness severity (i.e. hospital admission, intensive care unit (ICU) admission and/or mortality) during the 28-day follow-up period. We then describe the development of a novel machine learning algorithm based on the RALI-Dx biomarker results generated in Canada followed by validation on patients in Italy and Brazil. Importantly, we also provide the accuracy of the algorithm in patients with confirmed COVID-19 infection across the three international cohorts.
Methods
Additional details regarding assays, model development and statistical methods are provided in the supplementary material.
Patient population and data source
We conducted a prospective international multicentre study of adult patients presenting to the ED. Inclusion criteria were symptoms of respiratory illness, age ≥18 years and provision of informed consent. The study comprised three cohorts with identical inclusion criteria: 1) Canada cohort (n=310) of patients presenting to a University Health Network (UHN; Toronto, ON, Canada) ED from April to October 2020 and an additional set of patients that tested positive for COVID-19 at UHN from November to December 2020; 2) Italy cohort (n=131) of patients presenting to the ED of Cittá della Salute e della Scienza di Torino Hospital-Molinette Site (Turin, Italy) from April to May 2020; and 3) Brazil cohort (n=200) of patients presenting to the ED of Hospital São Lucas-PUCRS (Porto Alegre, Brazil) from January to March 2021. Healthy healthcare workers at Cittá della Salute e della Scienza di Torino Hospital-Molinette Site provided samples that served as controls for this study.
Participant details
A whole-blood sample was collected via venipuncture as part of routine blood work during initial evaluation. Demographic details were collected alongside standard vital signs and hospitalisation metrics. Hospitalisation decisions were made according to each institution's standard procedures. All patients were followed for 28 days after their ED visit via medical records and/or phone call. Patients who withdrew from the study at any time were excluded from analysis. The COVID-19 status of each patient was confirmed using a nasopharyngeal reverse transcriptase-PCR swab for the presence of severe acute respiratory syndrome coronavirus 2.
All studies were reviewed and approved by the research ethics board of each institution (UHN: REB 20-5225; Turin: Comitato Etico CS2/139; Porto Alegre: CAAE 39181420.0.0000.5336). The study was registered at ClinicalTrials.gov with identifier number NCT04750369.
RALI-Dx assay
A five-plex immunoassay (RALI-Dx; SQI Diagnostics, Toronto, ON, Canada) was developed based on previous lung injury studies [12] and included the following protein markers: IL-6, IL-8, IL-10, sTNFR1 and sTREM1.
RALI-Dx model development
The RALI-Dx biomarker predictive algorithm model was developed using the XGBoost (Extreme Gradient Boosting) algorithm [13]. To ensure that the model was appropriately trained to determine the level of care required, all Canada cohort cases were independently reviewed and adjudicated by two clinicians who were blinded to the RALI-Dx results to determine completion of data and the need for outpatient versus inpatient care (i.e. hospitalised for >72 h and/or required significant clinical intervention, such as intravenous therapies and/or supplemental oxygen or mechanical ventilation). Any discrepancies between clinicians were resolved via paired consensus.
Statistical analysis
All analyses were conducted using Stata (StataCorp, College Station, TX, USA), Prism (GraphPad, La Jolla, CA, USA), SPSS Statistics (IBM, Armonk, NY, USA), Python Programming Language (Python Software, Wilmington, DE, USA) or R statistics (www.r-project.org).
Results
Patient characteristics
In the Canada cohort, 310 patients were recruited from the ED with symptoms of respiratory illness. Patients were predominantly male (58%), Caucasian (57%) and presented with shortness of breath (51%) (table 1). 51% of patients were admitted to hospital following their ED visit and 5% of the cohort required care in an ICU within the 4-week period after initial presentation (figure 1). The 28-day mortality rate was 3.5% (figure 1). Characteristics of the Italy (131 patients) and Brazil (200 patients) cohorts are presented in supplementary table S1 and supplementary figure S1. The levels of RALI-Dx panel biomarkers in each patient population are summarised in table 1 and supplementary table S1.
Patient characteristics at emergency department baseline in the Canada cohort
Schematic of patient outcomes following emergency department (ED) presentation in the Canada cohort. ICU: intensive care unit.
RALI-Dx biomarker results for patients with symptoms of respiratory tract infection
In healthy patients, the plasma levels of RALI-Dx biomarkers were below RALI-Dx detection limits for IL-6, IL-8 and IL-10, and at 637 pg·mL−1 and 174 pg·mL−1 for sTNFR1 and sTREM1, respectively (supplementary table S5). All RALI-Dx biomarkers were significantly elevated in the plasma of patients who were hospitalised with respiratory illness (figure 2a). Median sTNFR1 and sTREM1 plasma levels were ∼2-fold higher in hospitalised patients (figure 2a). Biomarker levels were lowest in patients who were discharged from the ED for outpatient follow-up care and highest in those requiring care in an ICU at any point during the 28-day follow-up period (figure 2a). Importantly, all RALI-Dx biomarkers were significantly elevated in patients who died during the 28-day follow-up period (figure 2b). Consistent with the Canada cohort, significantly elevated plasma levels of RALI-Dx biomarkers were also observed in patients who were hospitalised in Italy and Brazil (supplementary figure S2).
RALI-Dx biomarkers are associated with a) level of care provided (Kruskal–Wallis test) and b) 28-day mortality (Mann–Whitney U-test) in the Canada cohort. IL: interleukin; sTNFR1: soluble tumour necrosis factor receptor 1; sTREM1: soluble triggering receptor expressed on myeloid cells 1; ICU: intensive care unit. Box-and-whisker plots show median, interquartile range (IQR) and 1.5 times the IQR. *: p<0.05; **: p<0.01; ***: p<0.001; ****: p<0.0001.
Univariate logistic regression analysis of RALI-Dx biomarkers shows that each biomarker significantly predicts the decision to hospitalise (supplementary table S4); sTNFR1 and sTREM1 were strong univariate predictors with area under the receiver operating characteristic curve (AUROC) values of 77% and 70%, respectively (supplementary table S4). Similar trends were observed in the patient cohorts from Italy and Brazil (supplementary table S4). Univariate cut-offs based on Youden's J-statistic for each biomarker were: 15 pg·mL−1 (IL-6), >0 pg·mL−1 (IL-8), 7 pg·mL−1 (IL-10), 1339 pg·mL−1 (sTNFR1) and 316 pg·mL−1 (sTREM1) for the Canada cohort. Notably, for patients discharged from the ED with undetectable levels of IL-10 (<7 pg·mL−1), specificity was 100%.
A model for host inflammatory biomarker assessments in the ED
The Canada cohort was used to develop a RALI-Dx predictive model, using a tree-based machine learning algorithm (XGBoost). The algorithm determines the probability that a particular patient requires hospitalisation using numerous decision trees comprised of the individual RALI-Dx biomarkers. The model classifier used patients who required outpatient monitoring versus hospitalisation (i.e. inpatient care required). The Canada cohort was randomly partitioned 80:20 for training and testing, and 5-fold cross-validation was performed in the training dataset. The Italy and Brazil cohorts were then used as external test datasets for the RALI-Dx model. Modelling results indicated that all five biomarkers in the RALI-Dx panel were required by the XGBoost algorithm, with SHAP (Shapley Additive Explanations) values >0 for each marker.
As a generalised model to predict hospitalisation for any patient with symptoms of respiratory illness, the RALI-Dx model had an AUROC of 82% and 76% in the training and test (Canada) datasets, and an AUROC of 84% and 86% in the Italy and Brazil test datasets, respectively (table 2). For comparison, the commonly used clinical model based on CRB-65 (confusion, respiratory rate ≥30 breaths·min−1, systolic blood pressure <90 mmHg or diastolic blood pressure ≤60 mmHg, age ≥65 years) had significantly inferior performance of 70% (p=0.00010) and 61% (p=0.017) in the training and test (Canada) datasets, and 77% (p=0.024) and 66% (p<0.0001) in the Italy and Brazil test datasets, respectively (table 2).
Prediction of inpatient treatment using RALI-Dx biomarkers or CRB-65 (confusion, respiratory rate ≥30 breaths·min−1, systolic blood pressure <90 mmHg or diastolic blood pressure ≤60 mmHg, age ≥65 years) for all patients with respiratory symptoms
We investigated the performance of the RALI-Dx model to correctly predict the need for hospitalisation in critically ill patients. Based on biomarker levels in the ED, the RALI-Dx model correctly reported an increased probability of hospitalisation and correctly predicted the need for hospitalisation in 74% (39 out of 53) of patients who required care in an ICU during the 28-day follow-up period across the three cohorts. Moreover, for high-risk patients that died during follow-up, the accuracy of the RALI-Dx model to predict inpatient care upon ED presentation was 89% (39 out of 44) and was the same for patients that died within 7 days of ED presentation (89% (17 out of 19)).
RALI-Dx model applicability to COVID-19 and pneumonia
For patients with a confirmed COVID-19 diagnosis, the plasma levels of RALI-Dx biomarkers were significantly elevated in patients who required hospitalisation compared with those that were safely discharged from the ED (figure 3). There were no significant differences in severity (hospital admission, ICU admission, mechanical ventilation and death) in the COVID-19 patient cohorts (supplementary table S2).
RALI-Dx biomarkers are elevated in hospitalised COVID-19 patients from Canada, Italy and Brazil. Box-and-whisker plots (median, interquartile range (IQR) and 1.5 times the IQR) for COVID-19 patients (n=245) that were discharged or hospitalised following emergency department presentation for: interleukin (IL)-6, IL-8, IL-10, soluble tumour necrosis factor receptor 1 (sTNFR1) and soluble triggering receptor expressed on myeloid cells 1 (sTREM1). Mann–Whitney U-test. **: p<0.01; ****: p<0.0001.
We then evaluated the RALI-Dx biomarker algorithm to predict the need for inpatient care in patients with COVID-19. There were 245 confirmed COVID-19 patients included in the study across the Canada, Italy and Brazil cohorts (table 2). The AUROC of the RALI-Dx model for disposition decision in COVID-19 patients was 82%. This represents a significant 12% improvement over the clinical CRB-65 approach (70%) (p<0.0001) (table 2).
In the Canada cohort, there were n=50 patients with a confirmed community-acquired or COVID-19 pneumonia diagnosis. In this subgroup, the RALI-Dx model predicted the need for hospitalisation better than or equally well to known scoring systems for pneumonia patients, namely CRB-65 and the pneumonia severity index (PSI) (table 3). RALI-Dx and PSI significantly outperformed CRB-65 (p=0.009 (PSI versus CRB-65) and p=0.005 (RALI-Dx versus CRB-65)). Similar trends in predictive performance were observed for pneumonia patients that required care in an ICU (mean±sd AUROC: 72±11% (CRB-65), 72±10% (PSI) and 79±15% (RALI-Dx)) or that died during the 28-day follow-up period (mean±sd AUROC: 72±20% (CRB-65), 91±6% (PSI) and 89±8% (RALI-Dx)).
Prediction of hospitalisation using RALI-Dx, CRB-65 (confusion, respiratory rate ≥30 breaths·min−1, systolic blood pressure <90 mmHg or diastolic blood pressure ≤60 mmHg, age ≥65 years) or the pneumonia severity index (PSI) for patients with confirmed pneumonia diagnosis in the Canada cohort
Discussion
In this prospective, multisite, international observational study, we evaluated the ability of the RALI-Dx immunoassay to augment patient assessments in the ED. We developed a novel machine learning approach to analyse the RALI-Dx biomarker results and demonstrated a dramatic improvement over the current standard of care for ED assessments in all jurisdictions studied. For any patient that presented to an ED with symptoms of respiratory illness, the RALI-Dx model predicted the need for inpatient care with an AUROC of 82% and 76% in training and test datasets from Canada, and 84% and 86% in the Italy and Brazil test datasets, respectively. Notably, this model is also specifically applicable to patients with COVID-19 (AUROC 82%) and pneumonia diagnosis (AUROC 87%). It is important to note that this rapid assay is applicable in any respiratory illness regardless of underlying viral aetiology. This is of relevance to the current COVID-19 pandemic, any future variants and waves, and any future respiratory pandemics, as the assay is directed to the host response to the viral illness and not the identity of the infecting virus itself.
Respiratory illnesses are a common cause of ED visits and hospital admissions worldwide each year. One of the most widely used clinical assessment tools in the ED to assess severity of respiratory illness, the CRB-65 score, does not precisely assess the host inflammatory response. As a result, patients are conservatively triaged in favour of admission for inpatient monitoring, which potentially admits many more patients to the ward and ICU, and places undue stress on the healthcare system. This issue has been immensely exacerbated by the COVID-19 pandemic that has imposed a crushing burden on hospitals worldwide.
The host inflammatory response to respiratory illness, particularly the pulmonary inflammatory response, represents a common pathway leading to severe disease. This has been shown to been especially true during the COVID-19 pandemic, with latent class analysis identifying a distinct subclass of COVID-19-related acute respiratory distress syndrome (ARDS) having an inflammatory phenotype [14]. Thus, the ability to rapidly quantify multiple inflammatory biomarkers upon presentation is of great benefit to ED teams in deciding hospital admission, irrespective of the aetiology of pulmonary disease.
We and others have previously identified the RALI-Dx biomarkers as critical markers of lung injury during isolated lung assessments [12, 15, 16]. Importantly, the results of this study confirm that these early markers of acute lung injury and ARDS are well suited to assess the extent and spectrum of host responses to any respiratory infection. The immunoassay is comprised of IL-6, IL-8 and IL-10 (three ILs which serve as important mediators of inflammatory processes), and sTNFR1 and sTREM1 (which indicate TNF-α signalling and neutrophil and macrophage activation, respectively). During the COVID-19 pandemic, several additional reports have confirmed the prognostic value of these inflammatory biomarkers and thus support the plausibility of the RALI-Dx concept [3–11]. Observational research has shown that biomarker-based approaches can predict adverse outcomes and mortality in patients with respiratory tract infection [17]. Taken together, these biomarkers provide an objective biological assessment of the host immunological response profile. As such, a rapid diagnostic panel could convey timely, meaningful information to the ED clinician in order to enhance the accuracy of safe and expedited patient assessment and disposition.
In an effort to interpret complex clinical data, artificial intelligence and machine learning techniques have been applied to many aspects of healthcare. Recently, the XGBoost algorithm has been applied to clinical features to develop models of COVID-19 severity [18–21]. However, those findings have primarily been reported by single-centre or retrospective studies and do not include cytokines or inflammatory biomarkers, and are therefore unable to capture the extent of the host response. The RALI-Dx biomarker model was developed using a cohort of ED patients from Canada who presented with symptoms of respiratory infection during a period of low COVID-19 infection positivity rates. To test the validity and generalisability of this approach and cohort, the model was validated in external test sites in Italy and Brazil. With consistent performance results across all test datasets, we have shown that the RALI-Dx biomarker model is generalisable across respiratory illnesses and jurisdictions. Importantly, the accuracy of the model was maintained across geographical sites with diverse demographics (i.e. age and ethnicity), clinical practices, COVID-19 burden (Italy and Brazil sites were tested during a peak wave in their respective pandemics) and possible COVID-19 variants.
Existing clinical prognostic tools that are widely used in the ED to determine patient disposition, such as CRB-65 or PSI, generally indicate the safety of outpatient management by estimating mortality risk [22]; however, there are a number of limitations to these approaches that include: need for an a priori pneumonia diagnosis, generalisability to or specificity in different respiratory illnesses (i.e. COVID-19), number of input variables needed and/or scoring complexity, and reduced memorability [23–25]. These limitations were confirmed in our study as CRB-65 scores performed poorly compared with the RALI-Dx biomarker-based approach with respect to assessing the need for hospitalisation in a generalised population of ED patients with respiratory symptoms. Interestingly, CRB-65 performance in the external cohorts was lower in Brazil compared with Italy, and may be attributed to the age dependency and poor performance of CRB-65 in COVID-19 patients [26, 27]. For patients with a confirmed pneumonia diagnosis, RALI-Dx and PSI scoring approaches performed equally well in predicting hospitalisation, ICU care and death; however, the RALI-Dx assay offers a more pragmatic assessment approach as it only requires a single blood sample, provides results in <1 h and, most importantly, can be applied to suspected pneumonia patients prior to diagnosis confirmation.
While the final decision on hospital admission and length of stay can be affected by medical, functional, psychosocial factors, and patients’ and relatives’ preferences [28, 29], the RALI-Dx biomarker test provides a novel approach to patient disposition assessments that enhances precision of decision making over existing strategies. Importantly, the RALI-Dx biomarker test provides the probability that a patient requires hospitalisation.
There are limitations to this study. The observational study design enabled the description of inflammatory biomarkers and association with patient disposition, but does not allow for evaluation of potential cause–effect relationships between biomarkers and clinical outcomes. While the results of our study are encouraging, a randomised, prospective trial in a diverse patient population would be ideal to fully understand the true impact of biomarker algorithms on ED triage decisions. The results of this study demonstrate the generalisability of an inflammatory biomarker approach to patient prognostication for any respiratory illness; however, supplemental studies that address RALI-Dx performance in additional subpopulations may uncover the relative contribution of each biomarker for a given pathogen. Although the RALI-Dx inflammatory biomarkers alone were prognostic for the severity of respiratory illness, other biomarkers that were not assessed in this study (i.e. procalcitonin and proadrenomedullin), in addition to approaches that include a larger suite of available clinical parameters and artificial intelligence-guided ED assessment, are targets for future studies.
In conclusion, a predictive model using a rapid inflammation diagnostic immunoassay (RALI-Dx) represents a valuable tool to accurately assess the individual patient's host inflammatory response in patients presenting to an ED with respiratory illness. This pragmatic assay will augment the precision of clinical decision making: to admit patients who are likely to develop severe illness and safely discharge those on a milder course to recover outside of the hospital, thereby better managing limited healthcare resources. RALI-Dx measures the host inflammatory response in patients with any respiratory illness and, therefore, will be broadly applicable for patients with COVID-19, its variants, as well as any future respiratory pandemic.
Supplementary material
Supplementary Material
Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.
Supplementary material ERJ-00459-2022.Supplement
Shareable PDF
Supplementary Material
This one-page PDF can be shared freely online.
Shareable PDF ERJ-00459-2022.Shareable
Acknowledgements
The authors wish to thank the participants in this study, the University Health Network (UHN; Toronto, ON, Canada) ED staff, including Debra Davies and Kathy Bates, and the research coordinators for their efforts to make this work possible. We would like to thank our ED colleagues in Italy and Brazil for their work and collaboration, and Ornella Bosco, Barbara Vizio and Martina Schiavello from the Laboratory of Cellular and Molecular Biology, Department of Medical Sciences, University of Torino (Turin, Italy) for their technical support. We also wish to thank the Toronto Lung Transplant Program Biobank and the PRESERVE Biobank at UHN, the clinical laboratory at Mount Sinai Hospital (Toronto, ON, Canada) for sample collection and processing, and Rasheed Ghany at UHN for the development and management of the study database.
Footnotes
This article has an editorial commentary: https://doi.org/10.1183/13993003.01808-2022
This study is registered at ClinicalTrials.gov with identifier number NCT04750369.
Conflict of interest: S. Keshavjee serves as Chief Medical Officer of Traferox Technologies and receives personal fees from Lung Bioengineering, outside the submitted work. A.T. Sage, M. Cypel, L. del Sorbo, B. Wang, J. Valero and S. Keshavjee are inventors of a patent licensed to SQI Diagnostics. The inventors fully adhere to a number of policies in place at the University Health Network that ensure academic integrity and management of potential conflicts of interest between authors and industry partners. For the purposes of regulatory application, SQI Diagnostics provided funding to support Brazil data collection and the work performed by D.R. Marinowic, F.O. Friedrich, C.R.R. Schmitz, L.S.M. dos Santos, F.M. Barbe-Tuana and M.H. Jones, but remained arms-length and blinded from the data analysis included in this study. All other authors report no conflicts of interest.
Support statement: This study was funded in part by the Canadian Institutes of Health Research (FRN 440205) and SQI Diagnostics. Neither funder played any role in study design, data collection, analysis, interpretation of the data, writing the report or the decision to submit the paper for publication. All authors confirm access to all data collected throughout the study term. Funding information for this article has been deposited with the Crossref Funder Registry.
- Received March 8, 2022.
- Accepted July 25, 2022.
- Copyright ©The authors 2022.
This version is distributed under the terms of the Creative Commons Attribution Non-Commercial Licence 4.0. For commercial reproduction rights and permissions contact permissions{at}ersnet.org
References
