The global extensively drug-resistant tuberculosis (TB) response plan calls for implementation of rapid tests to screen patients at risk of drug-resistant TB. Currently, two line probe assays exist, the INNO-LiPA®Rif.TB assay (Innogenetics, Ghent, Belgium) and the GenoType® MTBDR assay (Hain LifeScience GmbH, Nehren, Germany). While LiPA studies have been reviewed, the accuracy of GenoType assays has not been systematically reviewed.
The present authors carried out a systematic review and used meta-analysis methods appropriate for diagnostic accuracy. After the literature searches, 14 comparisons for rifampicin and 15 comparisons for isoniazid were identified in 10 articles that used GenoType MTBDR assays. Accuracy results were summarised in forest plots and pooled using bivariate random-effects regression.
The pooled sensitivity (98.1%, 95% confidence interval (CI) 95.9–99.1) and specificity (98.7%, 95% CI 97.3–99.4) estimates for rifampicin resistance were very high and consistent across all subgroups, assay versions and specimen types. The accuracy for isoniazid was variable, with lower sensitivity (84.3%, 95% CI 76.6–89.8) and more inconsistent than specificity (99.5%, 95% CI 97.5–99.9).
GenoType MDTBR assays demonstrate excellent accuracy for rifampicin resistance, even when used on clinical specimens. While specificity is excellent for isoniazid, sensitivity estimates were modest and variable. Together with data from demonstration projects, the meta-analysis provides evidence for policy making and clinical practice.
- Diagnostic accuracy
- drug resistance
- line probe assay
- multidrug-resistant tuberculosis
- sensitivity and specificity
Tuberculosis (TB) is a major global health problem 1. The emergence of multidrug-resistant (MDR)-TB and, more recently, of extensively drug-resistant (XDR)-TB, are widely considered to be serious threats to global TB control 2–4. Conventional drug-susceptibility testing (DST) has limitations. Solid media-based techniques, such as Löwenstein–Jensen and Middlebrook 7H10/11 using the proportion, absolute concentration and resistance ratio methods, take up to 8–12 weeks 5. Liquid media-based methods, such as the BACTEC® (BD Diagnostics, Sparks, MD, USA), MGIT® (BD Diagnostics) and BacT/ALERT® (bioMérieux SA, Marcy l'Etoile, France) systems, are faster and sensitive, but more expensive and complex 6.
The World Health Organization (WHO) and partners have proposed a global XDR-TB response plan, which calls for wide-scale implementation of rapid methods to screen patients at risk of MDR-TB 7. Rapid tests can provide results within days (even without culture, i.e. directly on specimens) and thus enable prompt and appropriate treatment, decrease morbidity and mortality, and interrupt transmission. Line probe assays (based on reverse-hybridisation DNA strip technology) could potentially address this urgent need 5.
Currently, two commercial line probe assays exist, the INNO-LiPA® Rif.TB (Innogenetics, Ghent, Belgium) and GenoType® MTBDR (Hain LifeScience GmbH, Nehren, Germany). The LiPA test can simultaneously detect Mycobacterium tuberculosis and the presence of a mutation in the rpoB gene, which confers resistance to rifampicin 8. A recent meta-analysis summarised the results obtained for the INNO-LiPA Rif.TB test and showed that LiPA had high sensitivity and specificity when Mycobacterium tuberculosis isolates from culture were used. The majority of studies had sensitivity of ≥95%, and nearly all were 100% specific 9. The results, however, were less accurate when the test was directly applied on clinical specimens (i.e. sputum), and fewer data were available 9.
The GenoType MTBDR assay, introduced in 2004, includes three steps: DNA extraction, multiplex PCR amplification, and reverse hybridisation 10. The MTBDR assay has an additional advantage over the LiPA because it can detect both rifampicin and isoniazid resistance. The MTBDR assay identifies mutations in the rpoB gene as well as mutations in the katG gene for high-level isoniazid resistance 10. The MTBDRplus, the second-generation assay, also detects mutations in the inhA gene that confers resistance to low-levels of isoniazid 10. A systematic review and meta-analysis was performed to determine the accuracy of GenoType MTBDR assays for diagnosing MDR-TB in clinical specimens and culture isolates. In addition to estimating the overall accuracy, the quality of studies was evaluated and factors were explored that may be responsible for heterogeneity among studies.
The present authors systematically searched the literature using predetermined inclusion criteria: use of either the first or second-generation GenoType MTBDR assay for diagnosing drug-resistant TB, comparison of the assay result with conventional DST as a reference standard, information to calculate sensitivity and specificity and a minimum sample size of 20 to avoid potential selection bias in small studies 14.
Three databases (PubMed, EMBASE and BIOSIS) were searched for relevant English language citations. Search terms included “tuberculosis”, “Mycobacterium tuberculosis”, “Hain LifeScience”, “line probe assay”, “GenoType MTBDR” and “molecular diagnostic techniques”. The search was restricted to the time period January 2004 to March 2008, since the first generation MTBDR assay was introduced in October 2004, and the second-generation MTBDRplus assay became available in February 2007. Reference lists from included studies were also searched. In addition, laboratory experts and the test manufacturer (Hain LifeScience GmbH) were contacted for additional studies. Conference abstracts were included when sufficient data were reported.
Figure 1⇓ shows the study selection process. After screening titles and abstracts, 22 articles were eligible for full text review after excluding articles that were not TB-related, did not focus on MDR-TB or used other line probe assays. Of these, 12 articles that used other GenoType assays or did not contain enough information to calculate sensitivity and specificity were further excluded from data extraction, and ten studies (including one conference abstract 15) that used MTBDR assays for the detection of drug resistance were included in the present meta-analysis 15–24. Several studies made more than one comparison, in which case each comparison was considered separately. Thus, the total number of comparisons in the final analysis was 14 for detection of rifampicin resistance and 15 for detection of isoniazid resistance.
A data extraction form was created and piloted with a subset of eligible studies. Based on the experience gained in the pilot data extraction, the data extraction form was improved and finalised. The final set of studies was assessed with the standardised form by one reviewer (D.I. Ling) and cross-checked by a second (A.A. Zwerling). Any differences between reviewers were resolved by consensus. Since discrepant analysis (where discordant results between line probe assay and reference standard results are resolved, post-hoc, using clinical or other test data) may be a potential source of bias, unresolved data were preferentially included when available 25.
Assessment of study quality
Using the Quality Assessment of Studies of Diagnostic Accuracy included in Systematic Reviews (QUADAS) criteria 26, the quality characteristics that are important for diagnostic accuracy studies were assessed: 1) comparison of the index test with an appropriate reference standard; 2) blinded interpretation of the test result with reference standard results and vice versa; 3) complete verification of test results with the reference standard; 4) recruitment of patients either consecutively or randomly; and 5) study design (i.e. cross-sectional versus case–control design).
Sensitivity and specificity values were calculated for the GenoType tests investigated in each study, along with their 95% confidence intervals, and displayed as forest plots. Sensitivity is the proportion of resistant results that are correctly classified by the MTBDR assay, while specificity is the proportion of susceptible results that are correctly identified as susceptible by the commercial kit.
Each study in the meta-analysis contributed a pair of numbers: sensitivity (true-positivity rate) and specificity (one minus false-positivity rate). Since these measures tend to be correlated and vary with the thresholds (cut-off points) used across individual studies, a summary receiver operating characteristic (SROC) curve analysis was performed in order to explore the effect of thresholds on the results 11, 27. The SROC curve displays the sensitivity and specificity estimates from each study within the receiver operating characteristic space. A regression curve is fitted through the distribution of pairs of sensitivity and specificity. A shoulder-like curve indicates that heterogeneity between studies may be due to the threshold effect (i.e. variation in cut-offs across studies) and that a common diagnostic odds ratio (DOR) exists that does not change with the threshold 27–29. A nonshoulder-like curve shows that sensitivity and specificity are not correlated. In addition, the area under the curve (AUC) also estimates the overall diagnostic accuracy. An AUC of 50% would indicate poor discriminatory ability, while an AUC of 100% means that the test discriminates perfectly between resistant and susceptible strains 27–29.
In addition to the SROC analyses, bivariate random effects regression analyses 13 were performed in STATA/IC 10.0 (Stata Corporation, Texas, USA, 2007) using the program “metandi” 30 to generate pooled accuracy estimates of sensitivity, specificity, positive likelihood ratio (LR+), negative likelihood ratio (LR-), and DOR. The LR+ measures how much more frequent a positive result (i.e. resistant) is found in resistant versus susceptible strains. Conversely, the LR- measures how much more likely a negative result (i.e. susceptible) is found in resistant versus susceptible strains. The DOR, or the odds of a positive result in resistant strains compared with the odds of a positive result in susceptible strains, combines both likelihood ratios and is a global measure of test performance 31. The DOR is calculated by LR+/LR− or (sensitivity/(1-specificity))/((1-sensitivity)/specificity) 31.
As described by Reitsma et al. 13, the bivariate regression method assumes that the sensitivity values from individual studies (after logit transformation) within a meta-analysis are approximately normally distributed around a mean value with a certain amount of variability around this mean 13. This is a random effects approach. This variation in underlying sensitivity estimates between studies can be related to undetected differences in study population, differences in implicit threshold (cut-off), or unnoticed variations in the index test protocol. These considerations also apply to specificity estimates. The potential presence of a (negative) correlation between sensitivity and specificity within studies is addressed by explicitly incorporating this correlation into the analysis. The combination of two normally distributed outcomes, the logit-transformed sensitivity and specificity values, while acknowledging the possible correlation between them, leads to the bivariate normal distribution 13. The bivariate approach overcomes the problems associated with simple pooling (i.e. weighted average) of sensitivity and specificity estimates 12, 13.
Heterogeneity is usually a concern with meta-analyses and refers to a high degree of variability in accuracy estimates across studies 32. Heterogeneity could be due to variability in thresholds, prevalence of drug-resistance, populations studied, variations in assay methods and reference standard tests and differences in study quality. When significant heterogeneity is present, summary estimates from meta-analyses are not meaningful. Heterogeneity was detected using Chi-squared and I-squared tests 33. Further reasons for the heterogeneity were investigated by pre-specified subgroup (stratified) analysis 32. In the subgroup analysis, the data were stratified according to the type of sample tested (clinical specimen versus culture isolate) and test version used (MTBDR versus MTBDRplus) to determine if accuracy varied across subgroups. Accuracy for rifampicin resistance was estimated separately from accuracy for isoniazid resistance.
Characteristics of included studies
As shown in figure 1⇑, 10 articles were included, with 14 comparisons for the detection of rifampicin resistance and 15 comparisons for the detection of isoniazid resistance, for a total of 3,349 specimens (mean (range) 116 (36–470)). Tables 1⇓–⇓3⇓ show the characteristics of the 10 studies in the meta-analysis. More studies assessed the performance of the MTBDR assay, because the MTBDRplus has only been available since 2007. The majority of the studies detected drug resistance on clinical specimens, including both smear-positive and smear-negative sputum, other respiratory samples and nonrespiratory samples. In addition, most laboratories used the BACTEC 460 or MGIT 960 (both from BD Diagnostics) systems for conventional DST. Unresolved data, in which discrepant analysis was not performed, were available in eight out of the 10 studies.
Most of the studies were cross-sectional in design (table 3⇑). Only two studies used consecutive or random sampling, while seven studies recruited patients using convenience sampling methods. One review of 31 meta-analyses on several diseases found higher accuracy measures associated with studies that used nonconsecutive sampling methods 34. All studies reported complete verification of MTBDR results with conventional DST as the reference standard. Past evidence has shown that investigators do not report all the study components in their publications 34, 35. In the present analysis, four studies did not report blinded interpretation of either the MTBDR or conventional DST results. Not blinding investigators to reference standard results when interpreting index test results has been shown to overestimate accuracy 36. In addition, most studies did not report whether patients were pre-treatment cases or already on treatment.
Accuracy for rifampicin resistance
Figure 2⇓ and table 4⇓ show the accuracy measures from all the rifampicin comparisons in a forest plot. Both sensitivity and specificity were highly consistent across the studies. Pooled sensitivity and specificity for rifampicin resistance was very high at 98.1% and 98.7%, respectively. The Chi-squared and I-squared tests for heterogeneity in the summary results suggested no significant heterogeneity across studies. Thus, summary measures of the test's diagnostic ability for rifampicin resistance can adequately describe the data. Table 5⇓ shows the pooled accuracy measures using the bivariate random effects regression method. As seen in table 5⇓, pooled sensitivity and specificity for rifampicin resistance was very high, and this was also reflected in the pooled likelihood ratios and DOR. The high accuracy is confirmed by the SROC plot shown in figure 3⇓. The area under the SROC curve was 99%, indicating near perfect discriminatory ability. Although the overall accuracy was consistently high across studies, subgroup analysis was performed by assay version and by specimen type. The accuracy was consistently high across all subgroups, specimen types and assay versions (table 5⇓).
Accuracy for isoniazid resistance
Figure 4⇓ and table 6⇓ show the accuracy measures from all the isoniazid comparisons in a forest plot. Sensitivity estimates were highly heterogeneous across studies. In contrast, specificity estimates were fairly consistent across the studies. The tests for heterogeneity suggested significant variability across studies for sensitivity. The pooled sensitivity estimate should therefore be interpreted with caution. Figure 5⇓ shows the SROC plot for isoniazid studies. The plot shows high specificity for most studies, but variable and modest sensitivity, ranging 57–100%.
Overall, analyses of isoniazid studies clearly showed a high degree of variability in sensitivity estimates. This heterogeneity may result from differences in test methods, population and study characteristics 27. Thus, subgroup analysis was performed to stratify data into relatively more homogeneous strata 32. The current authors stratified the studies by test version as well as by sample type. As shown in table 7⇓, specificity did not vary across subgroups, but sensitivity was higher when only MTBDRplus studies were pooled. Specimen type did not seem to affect accuracy.
In the meta-analysis, the comprehensive literature search identified 14 comparisons for rifampicin resistance and 15 comparisons for isoniazid resistance in 10 articles that reported the use of MTBDR assays. Sensitivity and specificity estimates for the diagnosis of rifampicin-resistant TB were excellent and consistent, across all subgroups, assay versions and specimen types. Thus, the GenoType assay has very high accuracy for rifampicin resistance. This is reflected in the high LR+ estimate and the very low LR- estimate, suggesting an excellent ability to both rule in and rule out rifampicin resistance.
The accuracy for isoniazid-resistant TB was variable, with sensitivity lower and more inconsistent than specificity. While specificity did not vary across subgroups, sensitivity was higher when only MTBDRplus studies were pooled; the highest accuracy for isoniazid resistance was obtained with the MTBDRplus assay, with the sensitivity improving to nearly 90%.
A notable advantage of molecular tests is their rapid turn-around time, which may have implications for patient management and transmission of drug-resistant TB. The turn-around time for the MTBDR assays ranged from 6 h to 2 days, substantially faster than conventional DST. The latter was reported from a high-volume laboratory in South Africa 17. Another key advantage is the direct use of line probe assays on clinical specimens; this precludes the need to wait for cultures to grow.
Strengths and limitations of the meta-analysis
The meta-analysis had several strengths. First, a standard protocol was used for carrying out the systematic review 11, including a comprehensive search strategy. Moreover, two reviewers independently carried out various stages of the systematic review process, including article selection and data extraction. Lastly, rigorous methods were used for data analysis, including bivariate random effects models, SROC analyses and methods for exploring heterogeneity.
The meta-analysis was limited by the relatively small number of available studies and the types of outcomes reported in the studies. Most studies only presented data on sensitivity and specificity. An obvious limitation is the lack of data on whether or not line probe assays have a clinical impact on patient management and treatment outcomes, and how much value they contribute beyond conventional tests. Data are also lacking on cost-effectiveness and feasibility in routine programme settings. Furthermore, there is little current evidence on how line probe assays may fit into existing diagnostic and treatment algorithms.
Despite using subgroup analysis, considerable heterogeneity remained unexplained in the isoniazid sensitivity results; further work is necessary to determine why isoniazid sensitivity values vary across settings. Geographic and genetic variations in the distribution of drug-resistant strains of M. tuberculosis might partially explain the present finding. For example, the prevalence of mutations in the inhA and katG genes seems to vary widely in different geographic locations 17. Furthermore, there were inadequate data to stratify by smear status, as smear-negative patients are the group most likely to benefit from the use of molecular assays.
Finally, the present authors excluded studies published in languages other than English. This, combined with potential publication bias, may have resulted in an overly optimistic estimate of the accuracy of the GenoType MTBDR assays. Currently available statistical approaches for publication bias (e.g. funnel plots and regression tests) are not recommended for diagnostic meta-analysis 37, and it is therefore difficult to rule out potential publication bias in the meta-analysis. In addition, the rates of indeterminate MTBDR results ranged 1.4–19.2% across studies. Since these were often not included in the calculations for sensitivity and specificity, the reported accuracy estimates may be inflated to some degree.
GenoType MDTBR assays demonstrate excellent accuracy for rifampicin resistance, which is a proxy for MDR-TB. This suggests good utility as a rapid screening tool, especially in settings with high rates of MDR-TB or HIV (where appropriate infection control is a major concern). While specificity is excellent for isoniazid, sensitivity estimates are modest and highly variable. With the latest MTBDRplus version of the assay, sensitivity for isoniazid resistance improves to ∼90%. This could be further improved in future generation assays once more data on mutations conferring isoniazid resistance becomes available. In addition, while some studies included smear-negative or nonrespiratory specimens, the results as presented did not allow for separate calculations of sensitivity and specificity estimates. Further studies are needed to compare the accuracy of the MTBDR assays in smear-positive versus smear-negative patients and pulmonary versus extrapulmonary cases.
An important issue that remains is the affordability of molecular assays and the associated laboratory infrastructure needs in resource-constrained settings. Commercial molecular tests, with prices typically higher than conventional tests, are popular in resource-rich settings. However, the most resource-constrained countries tend to have the highest burden of MDR-TB cases and are least likely to benefit from expensive technologies because of high costs and lack of appropriate laboratory capacity. Given that line probe assays are nucleic acid amplification assays, the need for proper laboratory design, laboratory standard operating procedures and quality control to avoid cross-contamination is paramount 38.
Realising this need, several groups, including the World Health Organization, the Foundation for Innovative New Diagnostics, and the Stop Tuberculosis Partnership's New Diagnostics Working Group, have launched initiatives to improve global laboratory capacity and to make new diagnostics affordable and accessible 39, 40. For example, the Foundation for Innovative New Diagnostics has negotiated reduced pricing for the public and nonprofit sectors in low-resource countries, resulting in a substantial reduction of the cost of the MTBDRplus assay relative to conventional liquid culture and drug-susceptibility testing 17. In addition, field demonstration projects are ongoing to evaluate the feasibility, cost and impact of these assays in routine programme conditions 17, 40. Together with meta-analysis data, these real-world field studies will provide the evidence necessary for policy making and clinical practice. In fact, in June 2008, the World Health Organization announced a new policy statement, endorsing the use of line probe assays for the rapid screening of patients at risk of multidrug-resistant tuberculosis 41. The recommended use of line probe assays is currently limited to culture isolates and direct testing of smear-positive sputum specimens. Line probe assays are not recommended as a complete replacement for conventional culture and drug-susceptibility testing.
The present study was supported by funding from the World Health Organization, Geneva, Switzerland.
Statement of interest
A statement of interest for M. Pai can be found at www.erj.ersjournals.com/misc/statements.shtml
The authors would like to thank R. O'Brien (Foundation for Innovative New Diagnostics (FIND), Geneva, Switzerland), H. Albert (FIND) and D. Hain (Nehren, Germany) for providing lists of relevant studies. The authors are also grateful to K. Weyer (World Health Organization, Geneva, Switzerland) for helpful input and feedback on the current manuscript.
- Received April 23, 2008.
- Accepted June 27, 2008.
- © ERS Journals Ltd