GenoType MTBDR assays for the diagnosis of multidrug-resistant tuberculosis: a meta-analysis

Search strategy

The present authors systematically searched the literature using predetermined inclusion criteria: use of either the first or second-generation GenoType MTBDR assay for diagnosing drug-resistant TB, comparison of the assay result with conventional DST as a reference standard, information to calculate sensitivity and specificity and a minimum sample size of 20 to avoid potential selection bias in small studies 14.

Three databases (PubMed, EMBASE and BIOSIS) were searched for relevant English language citations. Search terms included “tuberculosis”, “Mycobacterium tuberculosis”, “Hain LifeScience”, “line probe assay”, “GenoType MTBDR” and “molecular diagnostic techniques”. The search was restricted to the time period January 2004 to March 2008, since the first generation MTBDR assay was introduced in October 2004, and the second-generation MTBDRplus assay became available in February 2007. Reference lists from included studies were also searched. In addition, laboratory experts and the test manufacturer (Hain LifeScience GmbH) were contacted for additional studies. Conference abstracts were included when sufficient data were reported.

Study selection

Figure 1⇓ shows the study selection process. After screening titles and abstracts, 22 articles were eligible for full text review after excluding articles that were not TB-related, did not focus on MDR-TB or used other line probe assays. Of these, 12 articles that used other GenoType assays or did not contain enough information to calculate sensitivity and specificity were further excluded from data extraction, and ten studies (including one conference abstract 15) that used MTBDR assays for the detection of drug resistance were included in the present meta-analysis 15–24. Several studies made more than one comparison, in which case each comparison was considered separately. Thus, the total number of comparisons in the final analysis was 14 for detection of rifampicin resistance and 15 for detection of isoniazid resistance.

• Download figure
• Open in new tab
• Download powerpoint

Fig. 1—

Study selection process. TB: tuberculosis; MDR: mutidrug-resistant. GenoType MTBDR is manufactured by Hain LifeScience GmbH (Nehren, Germany).

Data extraction

A data extraction form was created and piloted with a subset of eligible studies. Based on the experience gained in the pilot data extraction, the data extraction form was improved and finalised. The final set of studies was assessed with the standardised form by one reviewer (D.I. Ling) and cross-checked by a second (A.A. Zwerling). Any differences between reviewers were resolved by consensus. Since discrepant analysis (where discordant results between line probe assay and reference standard results are resolved, post-hoc, using clinical or other test data) may be a potential source of bias, unresolved data were preferentially included when available 25.

Assessment of study quality

Using the Quality Assessment of Studies of Diagnostic Accuracy included in Systematic Reviews (QUADAS) criteria 26, the quality characteristics that are important for diagnostic accuracy studies were assessed: 1) comparison of the index test with an appropriate reference standard; 2) blinded interpretation of the test result with reference standard results and vice versa; 3) complete verification of test results with the reference standard; 4) recruitment of patients either consecutively or randomly; and 5) study design (i.e. cross-sectional versus case–control design).

Meta-analysis methods

Sensitivity and specificity values were calculated for the GenoType tests investigated in each study, along with their 95% confidence intervals, and displayed as forest plots. Sensitivity is the proportion of resistant results that are correctly classified by the MTBDR assay, while specificity is the proportion of susceptible results that are correctly identified as susceptible by the commercial kit.

Each study in the meta-analysis contributed a pair of numbers: sensitivity (true-positivity rate) and specificity (one minus false-positivity rate). Since these measures tend to be correlated and vary with the thresholds (cut-off points) used across individual studies, a summary receiver operating characteristic (SROC) curve analysis was performed in order to explore the effect of thresholds on the results 11, 27. The SROC curve displays the sensitivity and specificity estimates from each study within the receiver operating characteristic space. A regression curve is fitted through the distribution of pairs of sensitivity and specificity. A shoulder-like curve indicates that heterogeneity between studies may be due to the threshold effect (i.e. variation in cut-offs across studies) and that a common diagnostic odds ratio (DOR) exists that does not change with the threshold 27–29. A nonshoulder-like curve shows that sensitivity and specificity are not correlated. In addition, the area under the curve (AUC) also estimates the overall diagnostic accuracy. An AUC of 50% would indicate poor discriminatory ability, while an AUC of 100% means that the test discriminates perfectly between resistant and susceptible strains 27–29.

In addition to the SROC analyses, bivariate random effects regression analyses 13 were performed in STATA/IC 10.0 (Stata Corporation, Texas, USA, 2007) using the program “metandi” 30 to generate pooled accuracy estimates of sensitivity, specificity, positive likelihood ratio (LR+), negative likelihood ratio (LR-), and DOR. The LR+ measures how much more frequent a positive result (i.e. resistant) is found in resistant versus susceptible strains. Conversely, the LR- measures how much more likely a negative result (i.e. susceptible) is found in resistant versus susceptible strains. The DOR, or the odds of a positive result in resistant strains compared with the odds of a positive result in susceptible strains, combines both likelihood ratios and is a global measure of test performance 31. The DOR is calculated by LR+/LR− or (sensitivity/(1-specificity))/((1-sensitivity)/specificity) 31.

As described by Reitsma et al. 13, the bivariate regression method assumes that the sensitivity values from individual studies (after logit transformation) within a meta-analysis are approximately normally distributed around a mean value with a certain amount of variability around this mean 13. This is a random effects approach. This variation in underlying sensitivity estimates between studies can be related to undetected differences in study population, differences in implicit threshold (cut-off), or unnoticed variations in the index test protocol. These considerations also apply to specificity estimates. The potential presence of a (negative) correlation between sensitivity and specificity within studies is addressed by explicitly incorporating this correlation into the analysis. The combination of two normally distributed outcomes, the logit-transformed sensitivity and specificity values, while acknowledging the possible correlation between them, leads to the bivariate normal distribution 13. The bivariate approach overcomes the problems associated with simple pooling (i.e. weighted average) of sensitivity and specificity estimates 12, 13.

Subgroup analysis

Heterogeneity is usually a concern with meta-analyses and refers to a high degree of variability in accuracy estimates across studies 32. Heterogeneity could be due to variability in thresholds, prevalence of drug-resistance, populations studied, variations in assay methods and reference standard tests and differences in study quality. When significant heterogeneity is present, summary estimates from meta-analyses are not meaningful. Heterogeneity was detected using Chi-squared and I-squared tests 33. Further reasons for the heterogeneity were investigated by pre-specified subgroup (stratified) analysis 32. In the subgroup analysis, the data were stratified according to the type of sample tested (clinical specimen versus culture isolate) and test version used (MTBDR versus MTBDRplus) to determine if accuracy varied across subgroups. Accuracy for rifampicin resistance was estimated separately from accuracy for isoniazid resistance.

Characteristics of included studies

As shown in figure 1⇑, 10 articles were included, with 14 comparisons for the detection of rifampicin resistance and 15 comparisons for the detection of isoniazid resistance, for a total of 3,349 specimens (mean (range) 116 (36–470)). Tables 1⇓–⇓3⇓ show the characteristics of the 10 studies in the meta-analysis. More studies assessed the performance of the MTBDR assay, because the MTBDRplus has only been available since 2007. The majority of the studies detected drug resistance on clinical specimens, including both smear-positive and smear-negative sputum, other respiratory samples and nonrespiratory samples. In addition, most laboratories used the BACTEC 460 or MGIT 960 (both from BD Diagnostics) systems for conventional DST. Unresolved data, in which discrepant analysis was not performed, were available in eight out of the 10 studies.

Most of the studies were cross-sectional in design (table 3⇑). Only two studies used consecutive or random sampling, while seven studies recruited patients using convenience sampling methods. One review of 31 meta-analyses on several diseases found higher accuracy measures associated with studies that used nonconsecutive sampling methods 34. All studies reported complete verification of MTBDR results with conventional DST as the reference standard. Past evidence has shown that investigators do not report all the study components in their publications 34, 35. In the present analysis, four studies did not report blinded interpretation of either the MTBDR or conventional DST results. Not blinding investigators to reference standard results when interpreting index test results has been shown to overestimate accuracy 36. In addition, most studies did not report whether patients were pre-treatment cases or already on treatment.

Accuracy for rifampicin resistance

Figure 2⇓ and table 4⇓ show the accuracy measures from all the rifampicin comparisons in a forest plot. Both sensitivity and specificity were highly consistent across the studies. Pooled sensitivity and specificity for rifampicin resistance was very high at 98.1% and 98.7%, respectively. The Chi-squared and I-squared tests for heterogeneity in the summary results suggested no significant heterogeneity across studies. Thus, summary measures of the test's diagnostic ability for rifampicin resistance can adequately describe the data. Table 5⇓ shows the pooled accuracy measures using the bivariate random effects regression method. As seen in table 5⇓, pooled sensitivity and specificity for rifampicin resistance was very high, and this was also reflected in the pooled likelihood ratios and DOR. The high accuracy is confirmed by the SROC plot shown in figure 3⇓. The area under the SROC curve was 99%, indicating near perfect discriminatory ability. Although the overall accuracy was consistently high across studies, subgroup analysis was performed by assay version and by specimen type. The accuracy was consistently high across all subgroups, specimen types and assay versions (table 5⇓).

• Download figure
• Open in new tab
• Download powerpoint

Fig. 2—

Forest plot of sensitivity (a) and specificity (b) estimates for rifampicin resistance (all 14 studies, regardless of specimen type or assay version). ▪: point estimates of sensitivity and specificity from each study (proportionate to size of the study); –––––: 95% confidence intervals. See also table 4⇓.

• Download figure
• Open in new tab
• Download powerpoint

Fig. 3—

Summary reciever operating characteristic (SROC) plot for rifampicin resistance (all 14 studies, regardless of specimen type or assay version). ▪: each study in the meta-analysis size proportional to size of study; –––––: regression line that summarises the overall diagnostic accuracy. Area under the curve (AUC) 0.9949; se of AUC 0.0023; point of the SROC curve where the sensitivity and specificity are equal (Q*) 0.9722; se of Q* 0.0073.

Accuracy for isoniazid resistance

Figure 4⇓ and table 6⇓ show the accuracy measures from all the isoniazid comparisons in a forest plot. Sensitivity estimates were highly heterogeneous across studies. In contrast, specificity estimates were fairly consistent across the studies. The tests for heterogeneity suggested significant variability across studies for sensitivity. The pooled sensitivity estimate should therefore be interpreted with caution. Figure 5⇓ shows the SROC plot for isoniazid studies. The plot shows high specificity for most studies, but variable and modest sensitivity, ranging 57–100%.

• Download figure
• Open in new tab
• Download powerpoint

Fig. 4—

Forest plot of a) sensitivity and b) specificity estimates for isoniazid resistance (all 15 studies, regardless of specimen type or assay version). ▪: point estimates of sensitivity and specificity from each study (proportional to size of the study). –––––: 95% confidence intervals. See also table 6⇓.

• Download figure
• Open in new tab
• Download powerpoint

Fig. 5—

Summary reciever operating characteristic (SROC) plot for isoniazid resistance (all 15 studies, regardless of specimen type or assay version). ▪: each study in the meta-analysis size proportional to size of study; –––––: regression line that summarises the overall diagnostic accuracy. Area under the curve (AUC) 0.9727; se of AUC 0.0282; point on the SROC curve where the sensitivity and specificity are equal (Q*) 0.9243; se of Q* 0.0473.

Overall, analyses of isoniazid studies clearly showed a high degree of variability in sensitivity estimates. This heterogeneity may result from differences in test methods, population and study characteristics 27. Thus, subgroup analysis was performed to stratify data into relatively more homogeneous strata 32. The current authors stratified the studies by test version as well as by sample type. As shown in table 7⇓, specificity did not vary across subgroups, but sensitivity was higher when only MTBDRplus studies were pooled. Specimen type did not seem to affect accuracy.

Principal findings

In the meta-analysis, the comprehensive literature search identified 14 comparisons for rifampicin resistance and 15 comparisons for isoniazid resistance in 10 articles that reported the use of MTBDR assays. Sensitivity and specificity estimates for the diagnosis of rifampicin-resistant TB were excellent and consistent, across all subgroups, assay versions and specimen types. Thus, the GenoType assay has very high accuracy for rifampicin resistance. This is reflected in the high LR+ estimate and the very low LR- estimate, suggesting an excellent ability to both rule in and rule out rifampicin resistance.

The accuracy for isoniazid-resistant TB was variable, with sensitivity lower and more inconsistent than specificity. While specificity did not vary across subgroups, sensitivity was higher when only MTBDRplus studies were pooled; the highest accuracy for isoniazid resistance was obtained with the MTBDRplus assay, with the sensitivity improving to nearly 90%.

A notable advantage of molecular tests is their rapid turn-around time, which may have implications for patient management and transmission of drug-resistant TB. The turn-around time for the MTBDR assays ranged from 6 h to 2 days, substantially faster than conventional DST. The latter was reported from a high-volume laboratory in South Africa 17. Another key advantage is the direct use of line probe assays on clinical specimens; this precludes the need to wait for cultures to grow.

Strengths and limitations of the meta-analysis

The meta-analysis had several strengths. First, a standard protocol was used for carrying out the systematic review 11, including a comprehensive search strategy. Moreover, two reviewers independently carried out various stages of the systematic review process, including article selection and data extraction. Lastly, rigorous methods were used for data analysis, including bivariate random effects models, SROC analyses and methods for exploring heterogeneity.

The meta-analysis was limited by the relatively small number of available studies and the types of outcomes reported in the studies. Most studies only presented data on sensitivity and specificity. An obvious limitation is the lack of data on whether or not line probe assays have a clinical impact on patient management and treatment outcomes, and how much value they contribute beyond conventional tests. Data are also lacking on cost-effectiveness and feasibility in routine programme settings. Furthermore, there is little current evidence on how line probe assays may fit into existing diagnostic and treatment algorithms.

Despite using subgroup analysis, considerable heterogeneity remained unexplained in the isoniazid sensitivity results; further work is necessary to determine why isoniazid sensitivity values vary across settings. Geographic and genetic variations in the distribution of drug-resistant strains of M. tuberculosis might partially explain the present finding. For example, the prevalence of mutations in the inhA and katG genes seems to vary widely in different geographic locations 17. Furthermore, there were inadequate data to stratify by smear status, as smear-negative patients are the group most likely to benefit from the use of molecular assays.

Finally, the present authors excluded studies published in languages other than English. This, combined with potential publication bias, may have resulted in an overly optimistic estimate of the accuracy of the GenoType MTBDR assays. Currently available statistical approaches for publication bias (e.g. funnel plots and regression tests) are not recommended for diagnostic meta-analysis 37, and it is therefore difficult to rule out potential publication bias in the meta-analysis. In addition, the rates of indeterminate MTBDR results ranged 1.4–19.2% across studies. Since these were often not included in the calculations for sensitivity and specificity, the reported accuracy estimates may be inflated to some degree.

Conclusions

GenoType MDTBR assays demonstrate excellent accuracy for rifampicin resistance, which is a proxy for MDR-TB. This suggests good utility as a rapid screening tool, especially in settings with high rates of MDR-TB or HIV (where appropriate infection control is a major concern). While specificity is excellent for isoniazid, sensitivity estimates are modest and highly variable. With the latest MTBDRplus version of the assay, sensitivity for isoniazid resistance improves to ∼90%. This could be further improved in future generation assays once more data on mutations conferring isoniazid resistance becomes available. In addition, while some studies included smear-negative or nonrespiratory specimens, the results as presented did not allow for separate calculations of sensitivity and specificity estimates. Further studies are needed to compare the accuracy of the MTBDR assays in smear-positive versus smear-negative patients and pulmonary versus extrapulmonary cases.

An important issue that remains is the affordability of molecular assays and the associated laboratory infrastructure needs in resource-constrained settings. Commercial molecular tests, with prices typically higher than conventional tests, are popular in resource-rich settings. However, the most resource-constrained countries tend to have the highest burden of MDR-TB cases and are least likely to benefit from expensive technologies because of high costs and lack of appropriate laboratory capacity. Given that line probe assays are nucleic acid amplification assays, the need for proper laboratory design, laboratory standard operating procedures and quality control to avoid cross-contamination is paramount 38.

Realising this need, several groups, including the World Health Organization, the Foundation for Innovative New Diagnostics, and the Stop Tuberculosis Partnership's New Diagnostics Working Group, have launched initiatives to improve global laboratory capacity and to make new diagnostics affordable and accessible 39, 40. For example, the Foundation for Innovative New Diagnostics has negotiated reduced pricing for the public and nonprofit sectors in low-resource countries, resulting in a substantial reduction of the cost of the MTBDRplus assay relative to conventional liquid culture and drug-susceptibility testing 17. In addition, field demonstration projects are ongoing to evaluate the feasibility, cost and impact of these assays in routine programme conditions 17, 40. Together with meta-analysis data, these real-world field studies will provide the evidence necessary for policy making and clinical practice. In fact, in June 2008, the World Health Organization announced a new policy statement, endorsing the use of line probe assays for the rapid screening of patients at risk of multidrug-resistant tuberculosis 41. The recommended use of line probe assays is currently limited to culture isolates and direct testing of smear-positive sputum specimens. Line probe assays are not recommended as a complete replacement for conventional culture and drug-susceptibility testing.