Human reading versus computer-automated reading of chest radiographs in a tuberculosis screening programme in Romania

To the Editor:

One of the interventions in tuberculosis (TB) control is to screen people at high risk for TB with chest radiography [1]. Chest radiography in TB screening programmes are usually read by a radiographer or a pulmonologist specialised in TB. In recent years, computer-aided detection (CAD) software has become available for automated reading of CXRs and identifying people with presumptive TB [2, 3] and for TB screening [4, 5]. A systematic review published in 2016 concluded that the evidence of diagnostic accuracy of CAD was limited by the small number of studies, co-authored by owners of the only CAD software on the market at that time, and not generalisable to low TB and HIV settings [6]. The application of CAD software for TB detection has to our knowledge not been assessed in Europe.

From August, 2018, till September, 2019, TB screening was carried out in Romania in prisons, homeless persons, drug users and Roma population with a mobile digital radiography unit equipped with CAD software (CAD4TB version 6, Delft Imaging, ‘s-Hertogenbosch, the Netherlands). The CAD4TB software analyses the unobscured lung fields of a posterior–anterior chest radiography for the presence of abnormalities and provides a score between 0 and 100 [7]. The screening activity was part of the Early DETECTion of tuberculosis consortium (E-DETECT TB) project that applies evidence-based interventions to ensure early diagnosis and treatment of TB in vulnerable populations in European Union countries [8]. A total of 5003 individuals were radiologically screened (all in posterior-anterior projection) in the Romanian project; 5000 had a valid CAD4TB score. All CXRs were read by a pulmonologist (D. Gainaru), who had more than 20 years’ experience in reading CXRs in the TB screening programme. The management of people with presumptive TB or other CXR abnormalities was based on Romanian TB guidelines [9]. Ten individuals were diagnosed with TB, all with positive Mycobacterium tuberculosis cultures or positive Xpert MTB/RIF (Cepheid, Sunnyvale, CA, USA) results. Detected TB prevalence was 200 per 100 000 persons screened (95% confidence interval (CI) 96–368 per 100 000). Nine had a CAD4TB score >60 and one had a CAD4TB score of 59.

We designed a study to compare the CAD4TB reading with human reading as a tool to rule in people with radiographical abnormalities who require further evaluation for active TB. We included all 258 chest radiographs with a CAD4TB score >60, and randomly selected 742 chest radiographs (out of 4742, 15.6%) from the remaining chest radiographs with a score ≤60. The selected chest radiographs, except one that could not be accessed (total: 999), were read by two TB specialists (S. Keizer, I. Radulescu). Chest radiography with discordant results between the two readers, was read by a third TB specialist (M. Zamfirescu). All three readers (two pulmonologists and one TB public health physician) had more than 20 years' experience in reading chest radiography in TB screening programmes. Readers were blinded for age of the screened persons, and the CAD4TB score of the chest radiography. The three readers were asked to classify the chest radiography findings according to a classification previously used [10] and modified for this study, i.e. 1) normal; 2) highly suggestive for TB (cavities or extensive infiltrative disease, that warrant immediate action, i.e. respiratory isolation of the patient and urgent sputum examination); 3) possibly suggestive for TB (including TB sequelae and other signs of previous TB). Immediate respiratory isolation not required. Additional investigation to confirm or rule out TB, e.g. by asking for a previous episode of TB treatment, comparison of chest radiography with previous ones, sputum examination or otherwise; 4) chest radiography abnormality not suggestive for TB, including radiological findings suggestive for other pulmonary diseases (silicosis, pneumothorax), heart diseases (cardiomegaly), fractures, scoliosis, but also non-disease related chest radiography findings such as azygos lobe, a bullet, or a pacemaker. We combined the results of the chest radiographs highly and possibly suggestive for TB into one category, i.e. “presumptive TB”. Approval was provided by the ethical committee of Marius Nasta Institute of Pneumonology (Nr. 20276).

The results of the two readers were concordant for 798 (80%) of chest radiography (560 “normal”, 159 “presumptive TB” and 79 “other chest radiography abnormalities”), The Kappa score was 0.62 for the three categories (moderate agreement). The 201 (20%) chest radiographs with discordant results were read by the third reader, who was blinded for the results of the first two readers. The identical result reported by two of the three readers was taken as an agreed composite results, i.e. 46 “normal”, 32 “presumptive TB” and 102 “other chest radiography abnormalities”. In 21 chest radiographs, one of the readers classified the chest radiography as “normal”, one as “presumptive TB” and the third as “other chest radiography abnormalities”. These results were classified as “other chest radiography abnormalities”, since two out of three noticed an abnormality, and only one suggested that the chest radiography was suggestive for TB.

The chest radiography of the ten TB patients were all classified by both readers as “presumptive TB” and therefore not read by the third reader. Eight of these chest radiographs were classified by both readers as highly suspect for TB, one chest radiograph was classified by one reader as highly suspect and by the other as possibly suggestive for TB, and one chest radiograph was classified as possibly suggestive for TB by both readers. Figure 1 shows a linear association between the CAD4TB scores and the human readers classifying chest radiography as presumptive TB. All chest radiographs with a CAD4TB score >70 were scored abnormal, most of them as presumptive TB.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 1

Comparison of human reading results with computer-aided detection of chest radiography in a sample of the chest radiographs of a screening project for active tuberculosis in Romania with extrapolation to the actual screened population. CAD4TB: computer-aided detection for tuberculosis (Delft Imaging, ‘s-Hertogenbosch, the Netherlands). ^#: all chest radiographs with CAD4TB score >60, except one with a CAD4TB score of 64 that could not be accessed, were read, as well as 15.6% of chest radiographs with a score ≤60 (n=742); ^¶: extrapolation was done for each CAD4TB category (of ten values) by multiplying the actual number of screened people in the E-DETECT TB project with CAD4TB results in these categories with the proportions of classification by the human readers in the respective categories in the study population.

We tested the algorithm that only chest radiography above a threshold would need human reading, and based on the decision of the clinician additional TB examinations. Based on the sample size, we estimated (by extrapolation) the number of people that would be ruled in by CAD4TB and human reading. A CAD4TB score >60 would rule in 5.2% (n=258) of the chest radiographs for human reading, who would identify 164 individuals with presumptive TB, including nine out of ten patients diagnosed in the project (sensitivity 90%; 95% CI 60–98%). This algorithm would have avoided human reading of 94.8% (n=4742) of chest radiography, and the identification and subsequent examination of 178 individuals with presumptive TB, including one TB patient (CAD4TB score 59). If a cut-off of 50 was used, 16.3% (n=815) would be ruled-in for human reading, 275 individuals would be identified with presumptive TB, including all ten individuals who were diagnosed with TB in the project (sensitivity 100%). It would have avoided human reading of 83.7% (n=4185) of chest radiography and examination of 67 individuals with presumptive TB (none of them had TB).

We purposely asked readers in this study to rule in chest radiography with a low suspicion for TB, in order to increase sensitivity. This has resulted in a lower specificity, possibly a lower Kappa score and overestimation of people with presumptive TB (total 6.8%). This high proportion was partly due to the high percentage (2.7%) of clients with self-reported previous TB.

We conclude that computer-aided detection software provides an opportunity to identify people with presumptive TB in screening, allowing a reduction in the proportion of chest radiography that needs human reading (to less than 20% in our project) and avoiding unnecessary TB examinations while maintaining high sensitivity.

Shareable PDF