A pulmonologist's guide to understanding network meta-analysis

The basic premise

A common scenario is that drugs A and B have been compared to placebo in several trials; but no or very few trials have compared A to B directly (i.e. head-to-head studies). If we subtract the effect of (A versus placebo) from (B versus placebo), we can conclude the effect of A versus B. This effect is called an indirect estimate. If there was an actual head-to-head study that provided a direct estimate, then we can combine the direct estimate with the indirect estimate to produce a network meta-analysis estimate. Thus, a “network” of multiple interventions can be created. This network provides relative effectiveness estimates comparing all the possible permutations of the comparisons between interventions. Network meta-analysis also allows ranking of the interventions to help decision makers choose the most effective therapies [1].

Example

We are interested in determining which treatment to use for a patient with pulmonary arterial hypertension. A network meta-analysis is available and it included 31 randomised controlled trials with 6565 patients [2].

The systematic review conducted for the network meta-analysis aimed to assess the comparative efficacy and tolerability of pharmacological interventions for pulmonary arterial hypertension and searched five databases for relevant publications (between January 1, 1990 and March 3, 2016). The two main outcomes were mortality and clinical worsening. The available trials reporting each outcome can be depicted in a network graph (for example, for mortality see figure 1. In these graphs each intervention (also called node) is shown as a circle and each direct comparison as a solid line.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 1

Network diagram of comparisons of treatments for the outcome “mortality”. Solid lines indicate direct comparisons between treatments. Thickness of lines is proportional to the number of available trials. Each node (circle) represents a different treatment, and its size is proportional to the number of patients in which it is directly examined. ERA: endothelin receptor antagonist; PDE5i: phosphodiesterase-5 inhibitor. Produced from data reported in the study by Jain et al. [2].

Mortality

Mortality was ranked lowest for riociguat (risk ratio (RR) 0.36, 95% CI 0.07–1.81; compared with placebo), followed by endothelin receptor antagonist (ERA) (RR 0.70, 95% CI 0.45–1.07; compared with placebo), and highest with selexipag (RR 1.52, 95% CI 0.86–2.70; compared with placebo). In this network, no trials directly compared ERA to selexipag (i.e. looking at the figure, no line connects the two interventions). However, using their effect on placebo (the common comparator), analysis produced a RR of 0.46 (95% CI 0.22−0.93); suggesting significantly reduced mortality with ERA compared to selexipag (figure 2).

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 2

Comparative effectiveness of pharmacological treatments for pulmonary arterial hypertension to reduce clinical wosening and mortality. Comparisons should be read from left (active agent) to right (comparator agent or placebo). Lower numbers indicate a more favourable outcome (reduction of the relative risk of clinical worsening or mortality); darker shading indicates statistical significance. ERA: endothelin receptor antagonist; PDE5i: phosphodiesterase-5 inhibitor.

Assumptions

The following three assumptions have to be met for a network meta-analysis to be valid [3].

Analysis and results

Network meta-analyses rank interventions in terms of efficacy and report these rankings as a rank (i.e. drug A ranks first as most effective), a probability (drug A has 80% probability of being most effective) or as a surface under the cumulative ranking area, or SUCRA (e.g. SUCRA for drug A is 80%). For each treatment the probability that it has a certain rank can be derived from the distributions of all treatment effects based on analysis of combined direct and indirect evidence. The treatments can then be ranked by SUCRA [8]. Network meta-analyses also present “pairwise” results (i.e. comparison of each possible permutation between every drug and all others in the network). These comparisons are commonly presented as odds ratios or relative risks in a table that is called a staircase or league table (figure 2). These estimates are ideally presented as direct, indirect and network meta-analysis (combined) estimates to allow readers determine consistency and quality of the evidence. The analysis is done either following a Bayesian or frequentist approach. The choice between the two approaches depends on the training and experience of the statistical team conducting the network meta-analysis. No one approach of the two is preferred [3]. For the most part, the two approaches yield similar results. The exception to this is when the Bayesian analysis includes informative priors, which means that assumptions are made about the magnitude of heterogeneity or of the network meta-analysis effect size using outside data (data from other studies or literature aside from the trials being analysed in the network meta-analysis).

How to read and apply network meta-analysis results

Readers of network meta-analysis should first verify that a rigorous systematic review has been performed before trusting estimates reported in the analysis. Readers should then look at risk of bias tables in the review to get an overall sense of how well performed the individual studies were. Next, they should scrutinise the three assumptions and make their judgement about whether these assumptions have been met [5]. If so, they can consider whether the rankings make clinical sense by evaluating each individual pairwise comparison and determine whether important differences exist between interventions. Readers should also determine the network has sufficient direct evidence from head-to-head comparisons (versus being solely based on indirect comparisons). If only indirect evidence is available for comparison of two interventions, results must be interpreted with caution. Sparse direct evidence can make the analysis unstable and less reliable. Sophisticated frameworks such as GRADE (Grading of Recommendations, Assessment, Development and Evaluation) allow a formal evaluation of the quality of evidence (also called certainty in estimates) [9, 10].

Limitations of network meta-analysis

The results of network meta-analyses that do not meet the assumptions of homogeneity of direct evidence and/or transitivity (e.g. because of different target populations among included studies) can be potentially misleading.

Ranking probabilities need to be interpreted with caution and they can be misleading. For example, a drug may be ranked most effective but the difference in efficacy between this drug and the second-ranked drug could be trivial. Such probabilities are difficult to interpret clinically, and should be interpreted in the context of the magnitude of difference between each pair of interventions being compared.

The evidence from indirect comparisons is often not reported separately in network meta-analyses. A systematic review of 121 published network meta-analyses found that 89% of studies did not report indirect evidence [11]. It is important to check direct versus indirect evidence for each comparison, rather than depending on global tests of inconsistency [3].

36% of network meta-analyses, especially industry-sponsored studies, have been found not to present the full matrix of head-to-head treatment comparisons [12]. Readers of network meta-analyses need to be aware that such studies may not include all treatment comparisons for a certain condition, e.g. when there is a narrow research question.

Current and future applications of network meta-analysis in respiratory medicine

The major application of network meta-analysis is for the evaluation of comparative effectiveness of interventions that have not been assessed in head-to-head trials, e.g. different combinations of long-acting inhaled therapy (beta-agonists, anticholinergics and steroids) for chronic obstructive pulmonary disease [13]. Network meta-analysis has also been used in situations where the duration of an intervention varies, but data on different treatment length are not available from direct comparisons, e.g. isoniazid for preventive tuberculosis treatment for 6, 9 or 12 months [14]. There is a lack of comparative effectiveness studies that have compared pharmacological and non-pharmacological treatments in a wide spectrum of respiratory and sleep disorders, where network meta-analysis can potentially provide some indirect evidence.

Network meta-analysis can be used to calculate the required sample size for trials and to identify missing evidence on important comparisons. It can be expected that network meta-analysis will increasingly play a role in informing research priorities (which treatments should be compared against each other) [15, 16]. Network meta-analysis will also become increasingly important to inform policy-makers who often need to make decisions about interventions for clinical conditions for which a number of interventions are available, but only few have been studied in head-to-head comparisons.