Skip to main content

Main menu

  • Home
  • Current issue
  • ERJ Early View
  • Past issues
  • For authors
    • Instructions for authors
    • Submit a manuscript
    • Author FAQs
    • Open access
    • COVID-19 submission information
  • Alerts
  • Podcasts
  • Subscriptions
  • ERS Publications
    • European Respiratory Journal
    • ERJ Open Research
    • European Respiratory Review
    • Breathe
    • ERS Books
    • ERS publications home

User menu

  • Log in
  • Subscribe
  • Contact Us
  • My Cart

Search

  • Advanced search
  • ERS Publications
    • European Respiratory Journal
    • ERJ Open Research
    • European Respiratory Review
    • Breathe
    • ERS Books
    • ERS publications home

Login

European Respiratory Society

Advanced Search

  • Home
  • Current issue
  • ERJ Early View
  • Past issues
  • For authors
    • Instructions for authors
    • Submit a manuscript
    • Author FAQs
    • Open access
    • COVID-19 submission information
  • Alerts
  • Podcasts
  • Subscriptions

A Fully Automatic Deep Learning System for COVID-19 Diagnostic and Prognostic Analysis

Shuo Wang, Yunfei Zha, Weimin Li, Qingxia Wu, Xiaohu Li, Meng Niu, Meiyun Wang, Xiaoming Qiu, Hongjun Li, He Yu, Wei Gong, Yan Bai, Li Li, Yongbei Zhu, Liusu Wang, Jie Tian
European Respiratory Journal 2020; DOI: 10.1183/13993003.00775-2020
Shuo Wang
1Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Medicine and Engineering, Beihang University, Beijing, China
12contribute equally
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Yunfei Zha
2Department of Radiology, Renmin Hospital of Wuhan University, Wuhan, China
12contribute equally
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Weimin Li
3Department of respiratory and critical care medicine, West China hospital of Sichuan University, Chengdu, China
12contribute equally
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Qingxia Wu
4College of Medicine and Biomedical Information Engineering, Northeastern University, Shenyang, China
12contribute equally
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Xiaohu Li
5Department of Radiology, the First Affiliated Hospital of Anhui Medical University, Hefei, China
12contribute equally
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Meng Niu
6Department of Interventional Radiology, the First Hospital of China Medical University, Shenyang, China
12contribute equally
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Meiyun Wang
7Department of Medical Imaging, Henan Provincial People's Hospital & the People's Hospital of Zhengzhou University, Zhengzhou, China
12contribute equally
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Xiaoming Qiu
8Department of Radiology, Huangshi Central Hospital, Affiliated Hospital of Hubei Polytechnic University, Edong Healthcare Group, Huangshi, China
12contribute equally
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Hongjun Li
9Department of Radiology, Beijing Youan Hospital of Capital Medical University, Beijing, China
12contribute equally
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
He Yu
3Department of respiratory and critical care medicine, West China hospital of Sichuan University, Chengdu, China
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Wei Gong
2Department of Radiology, Renmin Hospital of Wuhan University, Wuhan, China
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Yan Bai
7Department of Medical Imaging, Henan Provincial People's Hospital & the People's Hospital of Zhengzhou University, Zhengzhou, China
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Li Li
9Department of Radiology, Beijing Youan Hospital of Capital Medical University, Beijing, China
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Yongbei Zhu
1Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Medicine and Engineering, Beihang University, Beijing, China
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Liusu Wang
1Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Medicine and Engineering, Beihang University, Beijing, China
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jie Tian
1Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Medicine and Engineering, Beihang University, Beijing, China
10CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
11Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, School of Life Science and Technology, Xidian University, Xi'an, China
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • Info & Metrics
  • PDF
Loading

Abstract

Coronavirus disease 2019 (COVID-19) has spread globally, and medical resources become insufficient in many regions. Fast diagnosis of COVID-19, and finding high-risk patients with worse prognosis for early prevention and medical resources optimisation is important. Here, we proposed a fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis by routinely used computed tomography.

We retrospectively collected 5372 patients with computed tomography images from 7 cities or provinces. Firstly, 4106 patients with computed tomography images were used to pre-train the DL system, making it learn lung features. Afterwards, 1266 patients (924 with COVID-19, and 471 had follow-up for 5+ days; 342 with other pneumonia) from 6 cities or provinces were enrolled to train and externally validate the performance of the deep learning system.

In the 4 external validation sets, the deep learning system achieved good performance in identifying COVID-19 from other pneumonia (AUC=0.87 and 0.88) and viral pneumonia (AUC=0.86). Moreover, the deep learning system succeeded to stratify patients into high-risk and low-risk groups whose hospital-stay time have significant difference (p=0.013 and 0.014). Without human-assistance, the deep learning system automatically focused on abnormal areas that showed consistent characteristics with reported radiological findings.

Deep learning provides a convenient tool for fast screening COVID-19 and finding potential high-risk patients, which may be helpful for medical resource optimisation and early prevention before patients show severe symptoms.

Abstract

Fully automatic deep learning system provides a convenient method for COVID-19 diagnostic and prognostic analysis, which can help COVID-19 screening and finding potential high-risk patients with worse prognosis.

Introduction

In Dec. 2019, the novel coronavirus disease 2019 (COVID-19) occurred in Wuhan, China and became a global health emergency very fast with more than 170 000 people infected [1–3]. Due to its high infection rate, fast diagnosis and optimised medical resource assignment in epidemic areas are urgent. Accurate and fast diagnosis of COVID-19 can help isolating infected patients to slow the spread of this disease. On the other hand, in epidemic area, insufficient medical resources have become a big challenge [4]. Therefore, finding high-risk patients with worse prognosis for prior medical resources and special care is crucial in the treatment of COVID-19.

Currently, reverse transcription polymerase chain reaction (RT-PCR) is used as the gold truth for diagnosing COVID-19. However, the limited sensitivity of RT-PCR and the shortage of testing kits in epidemic areas increase the screening burden, and many infected people are thereby not isolated immediately [5, 6]. This accelerates the spread of COVID-19. On the other hand, due to the lack of medical resources, many infected patients cannot receive immediate treatment. In this situation, finding high-risk patients with worse prognosis for prior treatment and early prevention is important. Consequently, fast diagnosis, finding high-risk patients with worse prognosis are very helpful for the control and management of COVID-19.

In recent studies, radiological findings demonstrated that computed tomography (CT) has great diagnostic and prognostic value for COVID-19. For example, CT showed much higher sensitivity than RT-PCR in diagnosing COVID-19 [5, 6]. For patients with COVID-19, bilateral lung lesions consisting of ground-glass opacities (GGO) were frequently observed in CT images [6–8]. Even in asymptomatic patients, abnormalities and changes were observed in serial CT [9, 10]. As a common diagnostic tool, CT is easy and fast to acquire without adding much cost. Building a sensitive diagnostic tool using CT image can accelerate the diagnostic process and is complementary to RT-PCR. On the other hand, predicting personalised prognosis using CT image can identify the potential high-risk patients who are more likely to become severe and need urgent medical resources.

Deep learning (DL) as an artificial intelligence method, has shown promising results in assisting lung disease analysis using CT images [11–15]. Benefiting from the strong feature learning ability, DL can mine features that are related to clinical outcomes from CT images automatically. Features learned by DL models can reflect high-dimensional abstract mappings which are difficult for human to sense but are strongly associated with clinical outcomes. Different from the published DL models [16, 17], we aim to provide a fully automatic DL system for COVID-19 diagnostic and prognostic analysis. Without requiring any human-assisted annotation, this novel DL system is fast and robust in clinical use. Moreover, we collected a large multi-regional dataset for training and validating the proposed DL system, including 1266 patients (471 had follow-up) from six cities or provinces. Notably, different with many studies using transfer learning from natural images. We collected a large auxiliary dataset including 4106 patients with chest CT images and gene information to pre-train the DL system, aiming at making the DL system learn lung features that can reflect the association between micro-level lung functional abnormalities and chest CT images.

Methods

Study design and participants

The institutional review board of the seven hospitals (supplementary methods 1) approved this multi-regional retrospective study and waived the need to obtain informed consent from the patients. In this study, we collected two datasets: COVID-19 dataset (n=1266) and CT-epidermal growth factor receptor (CT-EGFR) dataset (n=4106). In the COVID-19 dataset, 1266 patients were finally included who met the following inclusion criteria: i) RT-PCR confirmed COVID-19; ii) lab-confirmed other types of pneumonia before Dec. 2019; iii) have non-contrast enhanced chest CT at diagnosis time. Since RT-PCR has a relatively high false-negative rate, we collected other types of pneumonia before Dec. 2019 when the COVID-19 did not show up to guarantee the diagnosis of typical pneumonia are correct. In the COVID-19 dataset, patients from Wuhan city and Henan province formed the training set; patients from Anhui province formed the external validation set 1; patients from Heilongjiang province formed the validation set 2; patients from Beijing formed the validation set 3; patients from Huangshi city formed the validation set 4 (fig. 1).

FIGURE 1
  • Download figure
  • Open in new tab
  • Download powerpoint
FIGURE 1

Datasets used in this study. A total of 5372 patients with CT images from 7 cities or provinces were enrolled in this study. The auxiliary training set includes 4106 patients with lung cancer and EGFR gene mutation status information, and is used to pre-train the COVID-19Net to learn lung features from CT images. The training set includes 709 patients from Wuhan city and Henan province. The external validation set 1 (226 patients) from Anhui province, and the external validation set 2 (161 patients) from Heilongjiang province are used to assess the diagnostic performance of the DL system. The external validation set 3 (53 patients with COVID-19) from Beijing, and the external validation set 4 (117 patients with COVID-19) from Huangshi city are used to evaluate the prognostic performance of the DL system.

In the CT-EGFR dataset, 4106 patients with lung cancer were finally included who met the following criteria: i) epidermal growth factor receptor (EGFR) gene sequencing was obtained; ii) non-contrast enhanced chest CT data obtained within 4 weeks before EGFR gene sequencing. The CT-EGFR dataset was used for auxiliary training of the DL system, making the DL system learn lung features automatically. CT scanning parameters about the COVID-19 and CT-EGFR datasets were available in supplementary methods S1.

For prognostic analysis, 471 patients with COVID-19 and regular follow-up for at least 5 days were used. We defined the prognostic end event as the hospital-stay time which is counted from the diagnosis of COVID-19 to the time when the patient is allowed to discharge hospital (supplementary methods S2). A short hospital-stay time corresponds to good prognosis, and a long hospital-stay time means worse prognosis. Patients with long hospital-stay time take longer time to recover, and are defined as high-risk patients in this study. These patients need prior medical resources and special care since they are more likely to become severe.

The training set was used to train the proposed DL system; the validation set 1 and 2 were used to evaluate the diagnostic performance of the DL system; and the validation set 3 and 4 were used for evaluating the prognostic performance of the DL system.

The fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis

The proposed DL system includes three parts: automatic lung segmentation, non-lung area suppression, and COVID-19 diagnostic and prognostic analysis. In this DL system, two DL networks were involved: DenseNet121-FPN for lung segmentation in chest CT image, and the proposed novel COVID-19Net for COVID-19 diagnostic and prognostic analysis. DL is a family of hierarchical neural networks that aim at learning the abstract mapping between raw data to the desired clinical outcome. The computational units in DL model are defined as layers and they are integrated to simulate the inference process of human brain. The main computational formulas are convolution, pooling, activation and batch normalisation as defined in the supplementary methods S3.

Automatic lung segmentation

Routinely used chest CT image includes some non-lung areas (muscle, heart, et al.) and blank space outside body. To focus on analysing lung area, we used a fully automatic DL model (DenseNet121-FPN) [18, 19] to segment lung areas in chest CT image. This model is pre-trained using ImageNet dataset, and fine-tuned on VESSEL12 dataset [20] (supplementary methods S4).

Through this automatic lung segmentation procedure, we acquired the lung mask in CT image. However, some inflammatory tissues attaching to lung wall may be excluded falsely by the DenseNet121-FPN model. To increase the robustness of the DL system, we used the cubic bounding box of the segmented lung mask to crop lung areas in CT image, and defined this cubic lung area as lung-ROI (fig. 2). In this lung-ROI, all inflammatory tissues and the whole lung were correctly reserved, and most areas outside of lung were eliminated.

FIGURE 2
  • Download figure
  • Open in new tab
  • Download powerpoint
FIGURE 2

Illustration of the proposed DL system. Given the chest CT scanning of a patient, the DL system predicts the probability of the patient has COVID-19 and the prognosis of this patient directly without any human-annotation. The DL system includes three parts: automatic lung segmentation (DenseNet121-FPN), non-lung area suppression, and COVID-19 diagnostic and prognostic analysis (COVID-19Net). To let the COVID-19Net learn lung features from large dataset, we used the auxiliary training process for pre-training, which trained the DL network to predict EGFR gene mutation status using CT images of 4106 patients. The dense connection in this figure means each convolutional layer is connected to all of its previous convolutional layers inside the same dense block. CT: computed tomography. EGFR: epidermal growth factor receptor.

Non-lung area suppression

After the above processing, some non-lung tissues or organs (e.g., spine, heart) inside the lung-ROI may also exist. Consequently, we proposed a non-lung area suppression operation to suppress the intensities of non-lung areas inside the lung-ROI (supplementary methods S4). Finally, the lung-ROI is standardised by z-score normalisation, and resized to the size of 48×240×360 for further process.

Deep learning model for COVID-19 diagnosis and prognosis

After non-lung area suppression operation, the standardised lung-ROI is sent into the COVID-19Net for diagnostic and prognostic analysis. In figure 2, we illustrated the topological structure of the proposed novel COVID-19Net (supplementary table S1). This DL model used DenseNet-like structure [18], consisting of four dense blocks, where each dense block is multiple stacks of convolution, batch normalisation, and ReLU activation layers. Inside each dense block, we used dense connection to consider multi-level image information. At the end of the last convolutional layer, we used global average pooling to generate the 64-dimensional DL features. Finally, the output neuron is fully connected to the DL features to predict the probability of the input patient has COVID-19.

To enable the COVID-19Net learn discriminative features associated with COVID-19, a large training set is needed. Consequently, we proposed a two-step transfer learning process. Firstly, we proposed an auxiliary training process using the large CT-EGFR dataset (4106 patients) as illustrated in figure 2. In this auxiliary training process, we trained the COVID-19Net to predict EGFR mutation status (EGFR-mutant or EGFR wild type) using the lung-ROI [11]. Benefitting from the large CT-EGFR dataset, the COVID-19Net learned CT features that can reflect the associations between micro-level lung functional abnormality and macro-level CT images.

In the second training process, we transferred the pre-trained COVID-19Net to the COVID-19 dataset to specifically mine lung characteristics associated with COVID-19. After iterative training process in the COVID-19 dataset (supplementary methods S5), the COVID-19Net can predict a probability of the input patient being infected with COVID-19; this probability was defined as DL score in this study.

To explore the prognostic value of the DL features, we extracted the 64-dimensional DL feature from the COVID-19Net for prognostic analysis. Firstly, we combined the 64-dimensional DL feature and clinical features (age, sex, and comorbidity) to construct a combined feature vector. Afterwards, we used stepwise method to select prognostic features. These selected features were then used to build a multivariate Cox proportional hazard (CPH) model [21] to predict the hazard of the patient needing a long hospital-stay time to recover.

Visualisation of lung features learned by the DL system

Through the two-step transfer learning technique, the DL system learned lung features from CT images of 4815 patients. To further understand the inference process of the DL system, we used DL visualisation algorithm to analyse features learned by the COVID-19Net from two perspectives: 1) visualising the DL-discovered suspicious lung area that contribute most for identifying COVID-19 for the DL system; 2) visualising the feature patterns extracted by hierarchical convolutional layers in the COVID-19Net (supplementary methods S6, S7).

Statistical analysis

Area under the receiver operating characteristic (ROC) curve (AUC), accuracy, sensitivity, specificity, F1-score, calibration curves and Hosmer-Lemeshow test were used to assess the performance of the DL system in diagnosing COVID-19. Kaplan-Meier analysis and log-rank test were used to evaluate the performance of the DL system for prognostic analysis. The implementation of the DL system used the Keras 2.3.1 toolkit and Python 3.7 (https://github.com/wangshuocas/COVID-19).

Results

Clinical characteristics of patients in the COVID-19 dataset were presented in table 1. This dataset was collected from six cities or provinces including Wuhan city in China.

View this table:
  • View inline
  • View popup
TABLE 1

Clinical characteristics of patients

Diagnostic performance of the DL system

Table 2 and figure 3 illustrated the diagnostic performance of the DL system. In the training set, the DL system showed good diagnostic performance (AUC=0.90, sensitivity=78.93%, specificity=89.93%). This performance was further confirmed in the two external validation sets (AUC=0.87 and 0.88, sensitivity=80.39% and 79.35%, specificity=76.61% and 81.16%). The DL score revealed a significant difference between COVID-19 and other pneumonia groups in the three datasets (p<0.0001). The good performance in the validation sets indicated that the DL system generalised well on diagnosing COVID-19 of unseen new patients. Meanwhile, we illustrated the ROC curves of the DL system in the three datasets in figure 3a, and the calibration curves of the DL system in the two validation sets in figure 3b. The good calibration in figure 3b indicated that the DL system did not systematically under-predict or over-predict the COVID-19 probability because the Hosmer-Lemeshow test yielded a non-significant statistic to the perfect model (p=0.133 and p=0.229 in the two validation sets). Benefiting from the auxiliary training process in the large CT-EGFR dataset, the generalisation ability of the DL system improved largely compared with the DL system without auxiliary training (supplementary table S2).

View this table:
  • View inline
  • View popup
TABLE 2

Diagnostic performance of the DL system

FIGURE 3
  • Download figure
  • Open in new tab
  • Download powerpoint
FIGURE 3

Diagnostic performance of the DL system. a) ROC curves of the DL system in the training set and the two independent external validation sets. Validation 2-viral is a stratified analysis using the patients with COVID-19 and viral pneumonia in the validation set 2. b) Calibration curves of the DL system in the two external validation sets. c) AUC and distribution of the training set and the two external validation sets.

In other types of pneumonia, viral pneumonia has similar radiological characteristics to COVID-19, and therefore is more difficult to identify. Consequently, we performed a stratified analysis in the validation set 2. Table 1 indicated that the DL system also achieved good results in distinguish COVID-19 to other viral pneumonia (AUC=0.86).

Prognostic value of the DL features

In the COVID-19 dataset, 471 patients had follow-up for 5+ days. Through the stepwise prognostic feature selection, 3 features were selected (supplementary table S3). These selected prognostic features were fed into the multivariate CPH model to predict a hazard value for each patient. We used median value of the hazards in the training set as cut-off value to stratify patients into high-risk and low-risk groups. This cut-off value was also applied to the validation set 3 and 4. Kaplan-Meier analysis in the supplementary figure S1 demonstrated that patients in high-risk and low-risk groups had significant difference in hospital-stay time in the three datasets (p<0.0001, p=0.013, and p=0.014, log-rank test). These results suggested that the DL features have potential prognostic value for COVID-19.

Suspicious lung area discovered by the DL system

Through DL visualisation algorithm [22, 23], we are able to visualise the lung area that draws most attention to the DL system. These DL-discovered suspicious lung areas usually demonstrated abnormal characteristics consistent with radiologists’ findings. Figure 4 illustrated DL-discovered suspicious lung areas of eight patients with COVID-19. From this figure, we can see that although the input lung-ROI to the DL system includes some non-lung tissues such as muscle and bones, the DL system can always focus on areas inside lung for prediction instead of being disturbed by other tissues.

FIGURE 4
  • Download figure
  • Open in new tab
  • Download powerpoint
FIGURE 4

DL-discovered suspicious lung area. a–h) are CT images of eight patients with COVID-19. The first and the third rows are CT images of the patients (these CT images are processed by the DL system). The second and the fourth rows are heat maps of the DL-discovered suspicious lung area. In the heat map, areas with bright red color are more important than dark blue areas.

Moreover, the DL-discovered suspicious lung areas showed high overlap with the actual inflammatory areas. In figure 4a–d, we can see that, although we did not involve any human-annotation in the DL system, the DL system focused on the GGO area automatically for inference. This is consistent with radiologists’ experiences that many COVID-19 illustrated GGO features [6, 9]. In figure 4e–h, the DL-discovered suspicious lung areas distributed on bilateral lung, and mainly focused on lesions with consolidation, GGO, diffuse or mixture patterns. When comparing these DL-discovered suspicious lung areas with actual abnormal lung areas, we found a high overlap and consistent.

Although we did not use human annotation (e.g., human annotated ROI) to tell the DL system where to watch, the DL system is capable of discovering the abnormal and important lung areas automatically. This phenomenon could come from the advantage of using the large CT-EGFR dataset and the large COVID-19 dataset for training.

DL feature visualisation

Since DL is an end-to-end prediction model that learns abstract mappings between lung CT image and COVID-19 directly, it is helpful to explain the inference process of the DL system. The most important component of DL model is convolutional filter. Therefore, we visualised the 3-dimensional feature patterns extracted by hierarchical convolutional layers in figure 5. The shallow convolutional layer learned low-level simple features such as spindle edges (fig. 5a) and wave-like edges (fig. 5b). A deeper convolutional layer learned more complex and detailed features (fig. 5c). When going deeper, the feature pattern became more abstract and lack visual characteristics (fig. 5d) for our eyes. However, these high-level feature patterns are more related to COVID-19 information.

FIGURE 5
  • Download figure
  • Open in new tab
  • Download powerpoint
FIGURE 5

DL feature visualisation. a–d) are four 3-dimensional (3D) convolutional filters from different convolutional layers. e) is the distribution of patients in the 64-dimensional DL feature space. For display convenience, the 64-dimensional DL feature space is reduced to 2-dimensional by principle component analysis algorithm.

At the end of the DL model, the outputs of convolutional filters were compressed into a 64-dimensional vector, which was defined as DL feature. In figure 5e, we reduced the 64-dimensional DL feature into two-dimensional space to see the DL feature distribution in the two classes (COVID-19 versus other types of pneumonia). This figure demonstrated that the two classes distributed separately in the DL feature space, which means the DL features are discriminative to identify COVID-19 from other types of pneumonia.

Discussion

In this study, we proposed a novel fully automatic DL system using raw chest CT image to help COVID-19 diagnostic and prognostic analysis. To let the DL system mine lung features automatically without involving any time-consuming human annotation, we used a two-step transfer learning strategy. Firstly, we collected 4106 lung cancer patients with both CT image and EGFR gene sequencing. Through training in this large CT-EGFR dataset, the DL system learned hierarchical lung features that can reflect the associations between chest CT image and micro-level lung functional abnormality. Afterwards, we collected a large multi-regional COVID-19 dataset (n=1266) from 6 cities or provinces to train and validate the diagnostic and prognostic performance of the DL system.

The good diagnostic and prognostic performance of the DL system illustrates that DL could be helpful in the epidemic control of COVID-19 without adding much cost. Given a suspected patient, CT scanning can be acquired within minutes. Afterwards, this DL system can be applied to predict the probability of the patient has COVID-19. If the patient is diagnosed as COVID-19, the DL system also predicts his/her prognostic situation simultaneously, which can be used to find potential high-risk patients who need urgent medical resources and special care. More importantly, this DL system is fast and does not require human-assisted image annotation, which increases its clinical value and become more robust. For a typical chest CT scan of a patient, the DL system takes less than ten seconds for prognostic and diagnostic prediction.

During building and training the DL system, we did not involve any human annotation to tell the system where the inflammatory area was. However, the DL system managed to automatically discover the important features that are strongly associated with COVID-19. In figure 4, we visualised the DL-discovered suspicious lung areas that were used by the DL system for inference. These DL-discovered suspicious lung areas have high overlap with the actual inflammatory areas that are used by radiologists for diagnosis. In previous studies, some radiological features such as GGO, crazy-paving pattern, and bilateral involvement are reported to be important for diagnosing CVOID-19 [7]. In the DL-discovered suspicious lung areas, we also observed these radiological features. This demonstrates that the high-dimensional features mined by the DL system can probably reflect these reported radiological finding.

Recently, deep learning methods with different processes and models were reported to diagnose COVID-19 using CT images. These methods can be classified into three types: 1) Using manually or automatically segmented lesions for diagnosis. Wang et al. [16] used manually annotated lesions as ROI, and a modified ResNet34 model combined with decision tree and AdaBoost classifiers was used to diagnose COVID-19. To avoid time-consuming lesion annotation by radiologists, automatic lesion segmentation models [17, 24] were used in further studies. Afterwards, 3-dimensional CNN models such as 3DResNet were used to diagnose COVID-19 using the lesion images. 2) Using 2-dimensional lung image slices to train deep learning model. Since lesions can distribute in many locations in lungs, and automatic lesion segmentation may not guarantee very high precision. More studies used the whole lung image slices for analysis. In [25], a feature pyramid network using ResNet50 as backbone was used to analyse 2-dimensional image slices of the whole lung area. Similarly, Jin et al. [26] used DeepLabv1, Li et al. [27] used U-Net to segment lung from CT images, and then used 2-dimensional ResNet model to analyse image slices of lung area. 3) Using 3-dimensional deep learning model to analyse whole lung in CT images. To consider 3-dimensional information of the whole lung, Zheng et al. used 3DResNet model to analyse the 3-dimensional lung area in CT image [28].

Compared with these studies, our study has three main differences: 1) we used the 3-dimensional bounding box of the whole lung as ROI instead of only use lesions or segmented lung fields. Since lesion segmentation may not guarantee a very high accuracy, inaccurate lesion segmentation may cause information loss. Compared with segmenting lung lesion, lung segmentation is easier, and analysing the whole lung can mine more information. On the other hand, different with the methods using only the segmented lung areas [25, 27], we used the 3-dimensional bounding box of lung as ROI. In supplementary figure S2, we illustrated the lung segmentation results. In most situations, the lung segmentation method generated good results. However, for some patients with severe symptoms and consolidation lesions, the performance of the lung segmentation method may be affected. Consequently, we used the 3-dimensional bounding box of the segmented lung mask as ROI, which ensures the lung-ROI covering the complete lung area. Combined with the non-lung area suppression strategy, the lung-ROI can reserve complete lung area, and suppress images outside lung area. 2) We used a large auxiliary dataset including chest CT images of 4106 patients to pre-train the proposed COVID-19Net, making it learn lung features. Many existed studies used deep learning models pre-trained in ImageNet dataset, this may increase the generalisation ability of the deep learning model. However, natural images in the ImageNet dataset have large difference to chest CT images. Consequently, using chest CT dataset for auxiliary training (pre-training) enables the deep learning model learn features that are more specific to chest CT images. 3) Most studies used small dataset and randomly selected data for validation. To assess the generalisation ability of the deep learning model, we used large dataset and two independent validation sets from different regions.

Despite the good performance of the DL system, this study has several limitations. First, there are other prognostic end events such as death or admission to intensive care unit, and they were not considered in this study. Second, the management of severe and mild COVID-19 are different, thereby, explore prognosis of COVID-19 in these two groups separately should be helpful. On the other hand, CT images of different slice thickness were included in this study. In the future work, we will use generative adversarial network to convert CT images of different slice thickness into CT images with a unified slice thickness, which may further improve the diagnostic performance of the DL system.

Acknowledgments

This paper is supported by the National Natural Science Foundation of China under Grant Nos. 81930053, 81227901 and 81871332, the National Key R&D Program of China under Grant No. 2017YFA0205200, Novel Coronavirus Pneumonia Emergency Key Project of Science and Technology of Hubei Province under Grant No. 2020FCA015. We thank all the collaborative hospitals for data collection, especially thank Shuang Yan from Cancer Hospital Chinese Academy of Medical Sciences.

Footnotes

  • Contributors: YZ and JT conceived and designed the study. SW implemented the DL system and wrote the paper. QW, YZ, and LW contributed to the data process and analysis. MN, HY, WG, YB, XQ, LL, XL, MW, HL, and WL contributed to data collection.

  • This article has supplementary material available from erj.ersjournals.com

  • Funder: National Key R&D Program of China; Grant: 2017YFA0205200; Novel Coronavirus Pneumonia Emergency Key Project of Science and Technology of Hubei Province; Grant: 2020FCA015; National Natural Science Foundation of China; DOI: http://dx.doi.org/10.13039/501100001809; Grant: 81227901, 81871332, 81930053.

  • Conflict of interest: Dr. S. Wang has nothing to disclose.

  • Conflict of interest: Dr. Zha has nothing to disclose.

  • Conflict of interest: Dr. H. Li has nothing to disclose.

  • Conflict of interest: Dr. Wu has nothing to disclose.

  • Conflict of interest: Dr. X. Li has nothing to disclose.

  • Conflict of interest: Dr. Niu has nothing to disclose.

  • Conflict of interest: Dr. M. Wang has nothing to disclose.

  • Conflict of interest: Dr. Qiu has nothing to disclose.

  • Conflict of interest: Dr. W. Li has nothing to disclose.

  • Conflict of interest: Dr. Yu has nothing to disclose.

  • Conflict of interest: Dr. Gong has nothing to disclose.

  • Conflict of interest: Dr. Bai has nothing to disclose.

  • Conflict of interest: Dr. L. Li has nothing to disclose.

  • Conflict of interest: Dr. Zhu has nothing to disclose.

  • Conflict of interest: Dr. L. Wang has nothing to disclose.

  • Conflict of interest: Dr. Tian has nothing to disclose.

  • Received March 19, 2020.
  • Accepted May 16, 2020.
  • Copyright ©ERS 2020
http://creativecommons.org/licenses/by-nc/4.0/

This article is open access and distributed under the terms of the Creative Commons Attribution Non-Commercial Licence 4.0.

References

  1. ↵
    1. Wang D,
    2. Hu B,
    3. Hu C, et al.
    Clinical characteristics of 138 hospitalized patients with 2019 novel coronavirus–infected pneumonia in Wuhan, China. JAMA 2020.
    1. Li Q,
    2. Guan X,
    3. Wu P, et al.
    Early transmission dynamics in Wuhan, China, of novel coronavirus–infected pneumonia. N Engl J Med 2020.
  2. ↵
    1. Yang X,
    2. Yu Y,
    3. Xu J, et al.
    Clinical course and outcomes of critically ill patients with SARS-CoV-2 pneumonia in Wuhan, China: a single-centered, retrospective, observational study. Lancet Respir Med 2020; 8: 475–481. doi:10.1016/S2213-2600(20)30079-5
    OpenUrl
  3. ↵
    1. Ji Y,
    2. Ma Z,
    3. Peppelenbosch MP, et al.
    Potential association between COVID-19 mortality and health- care resource availability. Lancet Glob Health 2020; 8: e480. doi:10.1016/S2214-109X(20)30068-1
    OpenUrlPubMed
  4. ↵
    1. Xie X,
    2. Zhong Z,
    3. Zhao W, et al.
    Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing. Radiology 2020: 200343. doi:10.1148/radiol.2020200343
  5. ↵
    1. Ai T,
    2. Yang Z,
    3. Hou H, et al.
    Correlation of Chest CT and RT-PCR Testing in Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases. Radiology; 0: 200642.
  6. ↵
    1. Zhang S,
    2. Li H,
    3. Huang S, et al.
    High-resolution CT features of 17 cases of Corona Virus Disease 2019 in Sichuan province, China. Eur Respir J 2020; 55. doi:10.1183/13993003.00334-2020
  7. ↵
    1. Wang L,
    2. Gao Y-h,
    3. Zhang G-J
    . The clinical dynamics of 18 cases of COVID-19 outside of Wuhan, China. Eur Respir J 2020.
  8. ↵
    1. Shi H,
    2. Han X,
    3. Jiang N, et al.
    Radiological findings from 81 patients with COVID-19 pneumonia in Wuhan, China: a descriptive study.  Lancet Infect Dis 2020.
  9. ↵
    1. Lee EY,
    2. Ng M-Y,
    3. Khong P-L
    . COVID-19 pneumonia: what has CT taught us? Lancet Infect Dis 2020; 20: 384–385. doi:10.1016/S1473-3099(20)30134-1
    OpenUrlPubMed
  10. ↵
    1. Wang S,
    2. Shi J,
    3. Ye Z, et al.
    Predicting EGFR mutation status in lung adenocarcinoma on computed tomography image using deep learning. Eur Respir J 2019; 53: 1800986. doi:10.1183/13993003.00986-2018
    OpenUrlAbstract/FREE Full Text
    1. Walsh SL,
    2. Calandriello L,
    3. Silva M, et al.
    Deep learning for classifying fibrotic lung disease on high-resolution computed tomography: a case-cohort study. Lancet Respir Med 2018; 6: 837–845. doi:10.1016/S2213-2600(18)30286-8
    OpenUrl
    1. Walsh SL,
    2. Humphries SM,
    3. Wells AU, et al.
    Imaging research in fibrotic lung disease; applying deep learning to unsolved problems. Lancet Respir Med 2020.
    1. Angelini E,
    2. Dahan S,
    3. Shah A
    . Unravelling machine learning: insights in respiratory medicine. Eur Respir J 2019; 54: 1901216. doi:10.1183/13993003.01216-2019
    OpenUrlFREE Full Text
  11. ↵
    1. Wang S,
    2. Zhou M,
    3. Liu Z, et al.
    Central focused convolutional neural networks: Developing a data-driven model for lung nodule segmentation. Med Image Anal 2017; 40: 172–183. doi:10.1016/j.media.2017.06.014
    OpenUrl
  12. ↵
    1. Wang S,
    2. Kang B,
    3. Ma J, et al.
    A deep learning algorithm using CT images to screen for Corona Virus Disease (COVID-19). medRxiv 2020. 2020.2002.2014.20023028.
  13. ↵
    1. Xu X,
    2. Jiang X,
    3. Ma C, et al.
    Deep learning system to screen coronavirus disease 2019 pneumonia. arXiv preprint arXiv: 200209334 2020.
  14. ↵
    1. Huang G,
    2. Liu Z,
    3. Van Der Maaten L
    , et al. Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, 2017; pp. 4700–4708.
  15. ↵
    1. Lin T-Y,
    2. Dollár P,
    3. Girshick R
    , et al. Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, 2017; pp. 2117–2125.
  16. ↵
    1. Rudyanto RD,
    2. Kerkstra S,
    3. Van Rikxoort EM, et al.
    Comparing algorithms for automated vessel segmentation in computed tomography scans of the lung: the VESSEL12 study. Med Image Anal 2014; 18: 1217–1232. doi:10.1016/j.media.2014.07.003
    OpenUrl
  17. ↵
    1. Wang S,
    2. Liu Z,
    3. Rong Y, et al.
    Deep learning provides a new computed tomography-based prognostic biomarker for recurrence prediction in high-grade serous ovarian cancer. Radiother Oncol 2019; 132: 171–177. doi:10.1016/j.radonc.2018.10.019
    OpenUrl
  18. ↵
    1. Selvaraju RR,
    2. Cogswell M,
    3. Das A
    , et al. Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, 2017, 2017; pp. 618–626.
  19. ↵
    1. Kotikalapudi R
    . Contributors. keras-vis. GitHub, 2017.
  20. ↵
    1. Jin S,
    2. Wang B,
    3. Xu H, et al.
    AI-assisted CT imaging analysis for COVID-19 screening: Building and deploying a medical AI system in four weeks. medRxiv 2020. 2020.2003.2019.20039354.
  21. ↵
    1. Song Y,
    2. Zheng S,
    3. Li L, et al.
    Deep learning Enables Accurate Diagnosis of Novel Coronavirus (COVID-19) with CT images. medRxiv 2020. 2020.2002.2023.20026930.
  22. ↵
    1. Jin C,
    2. Chen W,
    3. Cao Y, et al.
    Development and Evaluation of an AI System for COVID-19 Diagnosis. medRxiv 2020. 2020.2003.2020.20039834.
  23. ↵
    1. Li L,
    2. Qin L,
    3. Xu Z, et al.
    Artificial Intelligence Distinguishes COVID-19 from Community Acquired Pneumonia on Chest CT. Radiology 2020: 200905. doi:10.1148/radiol.2020200905
  24. ↵
    1. Zheng C,
    2. Deng X,
    3. Fu Q, et al.
    Deep Learning-based Detection for COVID-19 from Chest CT using Weak Label. medRxiv 2020. 2020.2003.2012.20027185.
View Abstract
PreviousNext
Back to top
View this article with LENS
Vol 57 Issue 1 Table of Contents
European Respiratory Journal: 57 (1)
  • Table of Contents
  • Index by author
Email

Thank you for your interest in spreading the word on European Respiratory Society .

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
A Fully Automatic Deep Learning System for COVID-19 Diagnostic and Prognostic Analysis
(Your Name) has sent you a message from European Respiratory Society
(Your Name) thought you would like to see the European Respiratory Society web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Print
Alerts
Sign In to Email Alerts with your Email Address
Citation Tools
A Fully Automatic Deep Learning System for COVID-19 Diagnostic and Prognostic Analysis
Shuo Wang, Yunfei Zha, Weimin Li, Qingxia Wu, Xiaohu Li, Meng Niu, Meiyun Wang, Xiaoming Qiu, Hongjun Li, He Yu, Wei Gong, Yan Bai, Li Li, Yongbei Zhu, Liusu Wang, Jie Tian
European Respiratory Journal Jan 2020, 2000775; DOI: 10.1183/13993003.00775-2020

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero

Share
A Fully Automatic Deep Learning System for COVID-19 Diagnostic and Prognostic Analysis
Shuo Wang, Yunfei Zha, Weimin Li, Qingxia Wu, Xiaohu Li, Meng Niu, Meiyun Wang, Xiaoming Qiu, Hongjun Li, He Yu, Wei Gong, Yan Bai, Li Li, Yongbei Zhu, Liusu Wang, Jie Tian
European Respiratory Journal Jan 2020, 2000775; DOI: 10.1183/13993003.00775-2020
del.icio.us logo Digg logo Reddit logo Technorati logo Twitter logo CiteULike logo Connotea logo Facebook logo Google logo Mendeley logo
Full Text (PDF)

Jump To

  • Article
    • Abstract
    • Abstract
    • Introduction
    • Methods
    • Results
    • Discussion
    • Acknowledgments
    • Footnotes
    • References
  • Figures & Data
  • Info & Metrics
  • PDF

Subjects

  • Respiratory infections and tuberculosis
  • Tweet Widget
  • Facebook Like
  • Google Plus One

More in this TOC Section

  • Serum and Pulmonary Uric Acid in Pulmonary Arterial Hypertension
  • A-to-I editing of miR-200b-3p in airway cells is associated with moderate-to-severe asthma
  • Functional lower airways genomic profiling of the microbiome to capture active microbial metabolism
Show more Original article

Related Articles

Navigate

  • Home
  • Current issue
  • Archive

About the ERJ

  • Journal information
  • Editorial board
  • Reviewers
  • CME
  • Press
  • Permissions and reprints
  • Advertising

The European Respiratory Society

  • Society home
  • myERS
  • Privacy policy
  • Accessibility

ERS publications

  • European Respiratory Journal
  • ERJ Open Research
  • European Respiratory Review
  • Breathe
  • ERS books online
  • ERS Bookshop

Help

  • Feedback

For authors

  • Instructions for authors
  • Submit a manuscript
  • ERS author centre

For readers

  • Alerts
  • Subjects
  • Podcasts
  • RSS

Subscriptions

  • Accessing the ERS publications

Contact us

European Respiratory Society
442 Glossop Road
Sheffield S10 2PX
United Kingdom
Tel: +44 114 2672860
Email: journals@ersnet.org

ISSN

Print ISSN:  0903-1936
Online ISSN: 1399-3003

Copyright © 2021 by the European Respiratory Society