Skip to main content

Main menu

  • Home
  • Current issue
  • ERJ Early View
  • Past issues
  • Authors/reviewers
    • Instructions for authors
    • Submit a manuscript
    • Open access
    • COVID-19 submission information
    • Peer reviewer login
  • Alerts
  • Podcasts
  • Subscriptions
  • ERS Publications
    • European Respiratory Journal
    • ERJ Open Research
    • European Respiratory Review
    • Breathe
    • ERS Books
    • ERS publications home

User menu

  • Log in
  • Subscribe
  • Contact Us
  • My Cart

Search

  • Advanced search
  • ERS Publications
    • European Respiratory Journal
    • ERJ Open Research
    • European Respiratory Review
    • Breathe
    • ERS Books
    • ERS publications home

Login

European Respiratory Society

Advanced Search

  • Home
  • Current issue
  • ERJ Early View
  • Past issues
  • Authors/reviewers
    • Instructions for authors
    • Submit a manuscript
    • Open access
    • COVID-19 submission information
    • Peer reviewer login
  • Alerts
  • Podcasts
  • Subscriptions

Performance of Prediction Models for Covid-19: The Caudine Forks of the External Validation

Glen P. Martin, Matthew Sperrin, Giovanni Sotgiu
European Respiratory Journal 2020; DOI: 10.1183/13993003.03728-2020
Glen P. Martin
1Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Matthew Sperrin
1Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Giovanni Sotgiu
2Clinical Epidemiology and Medical Statistics Unit, Department of Medical, Surgical and Experimental Sciences, University of Sassari, Sassari, Italy
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Giovanni Sotgiu
  • Article
  • Info & Metrics
  • PDF
Loading

Abstract

COVID-19 and prediction modelling

Healthcare systems worldwide have observed significant changes to meet demands due to the coronavirus disease 2019 (Covid-19) pandemic. The uncertainty surrounding optimal treatment, the rapid public health urgency and clinical emergencies have caused a chaotic disruption of the cases and their related contacts at inpatient and outpatient settings. Developing more tailored healthcare plans based on the currently available scientific evidence, could help improve clinical efficacy, treatment outcomes, prognosis, and health efficiency.

Development and implementation of risk prediction models to aid risk stratification and resource allocation could improve the current scenario. Clinical prediction models (CPMs) aim to predict an individual's expected outcome value, or an individual's risk of an outcome being present (diagnostic) or happening in the future (prognostic), based on sets of identified predictor variables [1, 2]. A plethora of such models have been described during the first wave of the Covid-19 epidemic: a recent “living” systematic review identified (at the time of writing) 145 CPMs focused on Covid-19 patients [3].

Unfortunately, many of the existing Covid-19 CPMs have been identified to be at high risk of bias, due to poor reporting, over-estimation of predictive performance, and lack of external validation [3]. External validation, which is an important aspect during the development process of any CPM, can independently evaluate the model focusing on data independent to those data used to derive the model [1, 2]. Crucially, this step assesses the generalisability/transportability of the CPM into new populations before they are recommended for widespread clinical implementation.

To address this gap in the current literature, a paper by Gupta et al. [4], recently published in the European Respiratory Journal, aimed to externally validate 22 of the CPMs identified in the above-mentioned systematic review [3]. Using data from 411 adults who were admitted to the University College Hospital in London with clinically diagnosed Covid-19, it showed that all CPMs performed poorly in the new data [4]. Moreover, the Authors found that (within their data) baseline oxygen saturation on room air was the most predictive variable for in-hospital worsening, while age was the most predictive variable for in-hospital mortality. Astonishingly, none of the 22 CPMs included in the external validation demonstrated significantly higher clinical utility when compared with these variables alone [4]. Together, these results imply that none of the selected CPMs should be recommended for adoption in daily clinical routine.

Some caution is required in the interpretation of the findings of Gupta et al. [4]. First, this is a single site validation. Performance of CPMs may vary widely from site to site [5], so it could be argued that some of the Covid-19 models were simply “unlucky” on this particular site, and may perform better elsewhere. Therefore, wider multi-site external validation is urgently needed, potentially combined with meta-analysis of their predictive performance [6]. Second, new CPMs for Covid-19 are being developed all the time, and it is apparent that the quality of those models is now beginning to rise, based on new insights and methodological criticisms raised during this first evolution of the pandemic. For example, the ISARIC 4C prediction model [7] shows promising predictive performance results and has been developed on one of the largest dataset for inpatient Covid-19 admissions to-date, although, there remain some methodological concerns surrounding this model [8]. Thus, it is imperative that further external validation studies are conducted, so that emerging models can be evaluated.

Indeed, despite the widespread interest in developing new CPMs, it is key that the research community takes stock of the available evidence, before adding more models to the mix. For example, the important study by Gupta et al. [4] focussed on validating CPMs aimed to predict in-hospital clinical worsening or mortality among Covid-19 patients. Given that there are numerous other outcomes and different clinical settings (e.g., outpatient settings) where Covid-19 CPMs have been developed, future external validation studies should be a target for research in the near-future.

Nonetheless, the emerging scientific evidence indicates that none of the models can be recommended for clinical use and widespread adoption. Therefore, there remains one key outstanding question: how can we change this situation and develop CPMs for Covid-19 that can be recommended and largely implemented? One potential answer is that incentives need to change [9]. Specifically, having sufficient and qualitatively representative sample size is a crucial assumption when any CPM is planned to be developed [10–13]; however, this methodological necessity could be challenging in the context of an emerging pandemic where high-quality data is often scarce. In this situation, data sharing becomes paramount, but can be easily hampered by current research incentives [9]. Inevitably, incentives meant that the clinical need for Covid-19 prediction models in different contexts acted as a starting-signal for a race towards the “high-impact-publication” finish line. This is a multifaceted issue, but it does raise implications if there are future new diseases where the development of CPMs is required, but limited data are available. Even so, there are still opportunities for data sharing in the context of Covid-19 CPMs; Gupta et al. [4] nicely articulate this in the context of validating existing Covid-19 CPMs as follows: “future studies may seek to pool data from multiple centres in order to robustly evaluate the performance of existing and newly emerging models across heterogeneous populations”. If combined with meta-analytic methods [6, 14], such an approach will arguably be pivotal in addressing the clinical need for robust Covid-19 CPMs.

As the research community looks towards the “what now?” surrounding CPMs for Covid-19, it is important that we carefully utilise the available scientifically-sound evidence where possible. Specifically, it is entirely conceivable that emerging models (such as the ISARIC 4C prediction model [7]) will show adequate predictive performance results in data similar to that in which the model was developed, but external validation shows poor transferability of the models to new demographics (e.g. new countries) and statistical populations. In this situation, the community should build-upon such models, instead of developing de-novo models in distinct populations. For example, such existing Covid-19 models could be updated and refined using data in other populations, and, thereby, model transferability facilitated [15, 16]. A careful assessment and in-depth study of population characteristics can help find out the best approach to adapt a model to a new context, characterised by its own demographic, clinical, and epidemiological covariates. Repeatedly developing new models from scratch in distinct populations wastes prior information and risks overfitting. In contrast, model updating uses the existing models as a foundation, and builds upon this with the new data so that they are tailored to populations of interest.

Relatedly, there have been a multitude of CPMs developed for different aspects and contexts of Covid-19. Indeed, the existing models range from diagnostic CPMs aimed at predicting a virologically confirmed diagnosis of Covid-19, to prognostic CPMs oriented to the prediction of different clinical outcomes in those already diagnosed with Covid-19. However, many of these outcomes are inter-related and, in the context of facilitating decision-making for treatment/management of Covid-19 patients, looking at multiple outcomes simultaneously is often of greater clinical interest and offers more relevant insights. For example, different models have been developed to predict clinical worsening, longer length of hospital stay, or death (at various time points), whereas others have defined composite outcomes (e.g., death and clinical worsening) [3]. For decision-making, clinicians could be more interested in the risk variability of several of those outcomes co-occurring (e.g., estimating a probability of clinical worsening and longer hospital stay). Here, joint (rather than marginal) prediction is the primary focus but developing separate models for each outcome individually cannot enable this type of prediction [17]. Future work might wish to consider this approach and type of objectives in the context of Covid-19 CPMs.

Finally, an important consideration for all Covid-19 models developed to-date is that they are derived in the context of current care. However, current care continues to change rapidly as the pandemic unfolds. This means that outputs from Covid-19 CPMs need to be interpreted carefully: when, how, and where are question words to be posed to better interpret the potential inference and application of a model. Specifically, the predictions reveal the risk under the applied practices observed within the development dataset. They cannot be used to inform an individual's risk under various competing interventions [18]. A potential solution could be the exploration of counterfactual prediction [19, 20], in which the risk is estimated under fixed care regimes. This separates the baseline risk and the actions taken to mitigate the risk, thereby enabling users to answer “what-if” questions surrounding the impact of different interventions on Covid-19 risk in a particular setting [18]. Incorporating counterfactual prediction into the modelling might also increase the chances of model transferability across populations. Alternatively, embedding Covid-19 models within a dynamic framework [21] would allow the models to adapt rapidly to the changing clinical and temporal context. Such dynamic approaches to model updating help ensure that models maintain predictive performance through time, but it does require appropriate infrastructure to enable the real-time updates as new data are collected. Clearly, any form of model updating should only be undertaken on Covid-19 models that show lowest risk of bias, which are scarce to-date [3].

To summarise, the current scientific evidence suggests that none of the existing Covid-19 CPMs can be recommended for clinical use. We urgently recommend additional external validation studies, such as those by Gupta et al. [4]. Future work should seek to pool data across different populations and to apply model updating methods, where appropriate, to facilitate refinement of models across population variability.

Footnotes

  • Support statement: None

  • Conflict of interest: Dr. Martin has nothing to disclose.

  • Conflict of interest: Dr. Sperrin has nothing to disclose.

  • Conflict of interest: Dr. Sotgiu has nothing to disclose.

  • Received October 5, 2020.
  • Accepted October 6, 2020.
  • Copyright ©ERS 2020
http://creativecommons.org/licenses/by-nc/4.0/

This article is open access and distributed under the terms of the Creative Commons Attribution Non-Commercial Licence 4.0.

References

  1. ↵
    1. Steyerberg EW
    . Clinical Prediction Models. New York, Springer, 2009.
  2. ↵
    1. Riley RD,
    2. Windt D,
    3. Croft P, et al.
    Prognosis Research in Healthcare: Concepts, Methods, and Impact. Oxford University Press, 2019.
  3. ↵
    1. Wynants L,
    2. Van Calster B,
    3. Collins GS, et al.
    Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 2020; 369: m1328. doi:10.1136/bmj.m1328
    OpenUrlAbstract/FREE Full Text
  4. ↵
    1. Gupta RK,
    2. Marks M,
    3. Samuels THA, et al.
    Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: An observational cohort study. Eur Respir J 2020: 2003498. in press. https://doi.org/10.1101/2020.07.24.20149815. doi:10.1183/13993003.03498-2020
  5. ↵
    1. Riley RD,
    2. Ensor J,
    3. Snell KIE, et al.
    External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: Opportunities and challenges. BMJ 2016; 353: 27–30.
    OpenUrl
  6. ↵
    1. Snell KIE,
    2. Hua H,
    3. Debray TPA, et al.
    Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model. J Clin Epidemiol 2016; 69: 40–50. doi:10.1016/j.jclinepi.2015.05.009
    OpenUrlCrossRefPubMed
  7. ↵
    1. Knight SR,
    2. Ho A,
    3. Pius R, et al.
    Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ 2020; 370: m3339. doi:10.1136/bmj.m3339
    OpenUrlAbstract/FREE Full Text
  8. ↵
    1. Riley RD,
    2. Collins GS,
    3. van Smeden M, et al.
    Is the 4C Mortality Score fit for purpose? Some comments and concerns. BMJ 2020; 370: m3339. www.bmj.com/content/370/bmj.m3339/rr-3.
    OpenUrlAbstract/FREE Full Text
  9. ↵
    1. Sperrin M,
    2. Grant SW,
    3. Peek N
    . Prediction models for diagnosis and prognosis in Covid-19: All models are wrong but data sharing and better reporting could improve this. BMJ 2020; 369: m1464. www.bmj.com/content/369/bmj.m1464. doi:10.1136/bmj.m1464
    OpenUrlFREE Full Text
  10. ↵
    1. Riley RD,
    2. Snell KIE,
    3. Ensor J, et al.
    Minimum sample size for developing a multivariable prediction model: Part I - Continuous outcomes. Stat Med 2019; 38: 1262–1275. doi:10.1002/sim.7993
    OpenUrlCrossRefPubMed
    1. Riley RD,
    2. Snell KI,
    3. Ensor J, et al.
    Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes. Stat Med 2019; 38: 1276–1296. doi:10.1002/sim.7992
    OpenUrlCrossRefPubMed
    1. Riley RD,
    2. Ensor J,
    3. Snell KIE, et al.
    Calculating the sample size required for developing a clinical prediction model. BMJ 2020; 368: m441. doi:10.1136/bmj.m441
    OpenUrlFREE Full Text
  11. ↵
    1. van Smeden M,
    2. Moons KG,
    3. de Groot JA, et al.
    Sample size for binary logistic prediction models: Beyond events per variable criteria. Stat Methods Med Res 2019; 28: 2455–2474. doi:10.1177/0962280218784726
    OpenUrlCrossRefPubMed
  12. ↵
    1. Debray TP,
    2. Damen JA,
    3. Riley RD, et al.
    A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes. Stat Methods Med Res 2019; 28: 2768–2786. doi:10.1177/0962280218785504
    OpenUrlCrossRefPubMed
  13. ↵
    1. Janssen KJM,
    2. Moons KGM,
    3. Kalkman CJ, et al.
    Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 2008; 61: 76–86. doi:10.1016/j.jclinepi.2007.04.018
    OpenUrlCrossRefPubMedWeb of Science
  14. ↵
    1. Su T-L,
    2. Jaki T,
    3. Hickey GL, et al.
    A review of statistical updating methods for clinical prediction models. Stat Methods Med Res 2018; 27: 185–197. doi:10.1177/0962280215626466
    OpenUrlPubMed
  15. ↵
    1. Martin GP,
    2. Sperrin M,
    3. Snell KIE, et al.
    Clinical Prediction Models to Predict the Risk of Multiple Binary Outcomes: a comparison of approaches. arXiv 2020. www.arxiv.org/abs/2001.07624.
  16. ↵
    1. Sperrin M,
    2. Martin GP,
    3. Pate A, et al.
    Using marginal structural models to adjust for treatment drop-in when developing clinical prediction models. Stat Med 2018; 37: 4142–4154. doi:10.1002/sim.7913
    OpenUrlCrossRefPubMed
  17. ↵
    1. Hernán MA,
    2. Hsu J,
    3. Healy B
    . A Second Chance to Get Causal Inference Right: A Classification of Data Science Tasks. Chance 2019; 32: 42–49. doi:10.1080/09332480.2019.1579578
    OpenUrlCrossRef
  18. ↵
    1. van Geloven N,
    2. Swanson SA,
    3. Ramspek CL, et al.
    Prediction meets causal inference: the role of treatment in clinical prediction models. Eur J Epidemiol 2020; 35: 619–630. doi:10.1007/s10654-020-00636-1
    OpenUrl
  19. ↵
    1. Jenkins DA,
    2. Sperrin M,
    3. Martin GP, et al.
    Dynamic models to predict health outcomes: current status and methodological challenges. Diagn Progn Res 2018; 2: 23. doi:10.1186/s41512-018-0045-2
    OpenUrl
PreviousNext
Back to top
View this article with LENS
Vol 59 Issue 5 Table of Contents
European Respiratory Journal: 59 (5)
  • Table of Contents
  • Index by author
Email

Thank you for your interest in spreading the word on European Respiratory Society .

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Performance of Prediction Models for Covid-19: The Caudine Forks of the External Validation
(Your Name) has sent you a message from European Respiratory Society
(Your Name) thought you would like to see the European Respiratory Society web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Print
Citation Tools
Performance of Prediction Models for Covid-19: The Caudine Forks of the External Validation
Glen P. Martin, Matthew Sperrin, Giovanni Sotgiu
European Respiratory Journal Jan 2020, 2003728; DOI: 10.1183/13993003.03728-2020

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero

Share
Performance of Prediction Models for Covid-19: The Caudine Forks of the External Validation
Glen P. Martin, Matthew Sperrin, Giovanni Sotgiu
European Respiratory Journal Jan 2020, 2003728; DOI: 10.1183/13993003.03728-2020
del.icio.us logo Digg logo Reddit logo Technorati logo Twitter logo CiteULike logo Connotea logo Facebook logo Google logo Mendeley logo
Full Text (PDF)

Jump To

  • Article
    • Abstract
    • Footnotes
    • References
  • Info & Metrics
  • PDF
  • Tweet Widget
  • Facebook Like
  • Google Plus One

More in this TOC Section

  • Commemorating World Tuberculosis Day 2022
  • Is low level of vitamin D a marker of poor health or its cause?
  • Risky political game with climate change
Show more Editorial

Related Articles

Navigate

  • Home
  • Current issue
  • Archive

About the ERJ

  • Journal information
  • Editorial board
  • Reviewers
  • Press
  • Permissions and reprints
  • Advertising

The European Respiratory Society

  • Society home
  • myERS
  • Privacy policy
  • Accessibility

ERS publications

  • European Respiratory Journal
  • ERJ Open Research
  • European Respiratory Review
  • Breathe
  • ERS books online
  • ERS Bookshop

Help

  • Feedback

For authors

  • Instructions for authors
  • Publication ethics and malpractice
  • Submit a manuscript

For readers

  • Alerts
  • Subjects
  • Podcasts
  • RSS

Subscriptions

  • Accessing the ERS publications

Contact us

European Respiratory Society
442 Glossop Road
Sheffield S10 2PX
United Kingdom
Tel: +44 114 2672860
Email: journals@ersnet.org

ISSN

Print ISSN:  0903-1936
Online ISSN: 1399-3003

Copyright © 2022 by the European Respiratory Society