Elsevier

The Lancet

Volume 352, Issue 9128, 22 August 1998, Pages 609-613
The Lancet

Articles
Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?

https://doi.org/10.1016/S0140-6736(98)01085-XGet rights and content

Summary

Background

Few meta-analyses of randomised trials assess the quality of the studies included. Yet there is increasing evidence that trial quality can affect estimates of intervention efficacy. We investigated whether different methods of quality assessment provide different estimates of intervention efficacy evaluated in randomised controlled trials (RCTs).

Methods

We randomly selected 11 meta-analyses that involved 127 RCTs on the efficacy of interventions used for circulatory and digestive diseases, mental health, and pregnancy and childbirth. We replicated all the meta-analyses using published data from the primary studies. The quality of reporting of all 127 clinical trials was assessed by means of component and scale approaches. To explore the effects of quality on the quantitative results, we examined the effects of different methods of incorporating quality scores (sensitivity analysis and quality weights) on the results of the meta-analyses.

Findings

The quality of trials was low. Masked assessments provided significantly higher scores than unmasked assessments (mean 2·74 [SD 1·10] vs 2·55 [1·20]). Low-quality trials (score ·2), compared with high-quality trials (score >2), were associated with an increased estimate of benefit of 34% (ratio of odds ratios [ROR] 0·66 [95% CI 0·52–0·83]). Trials that used inadequate allocation concealment, compared with those that used adequate methods, were also associated with an increased estimate of benefit (37%; ROR=0·63 [0·45–0·88]). The average treatment benefit was 39% (odds ratio [OR] 0·61 [0·57–0·65]) for all trials, 52% (OR 0·48 [0·43–0·54]) for low-quality trials, and 29% (OR 0·71 [0·65–0·77]) for high-quality trials. Use of all the trial scores as quality weights reduced the effects to 35% (OR 0·65 [0·59–0·71]) and resulted in the least statistical heterogeneity.

Interpretation

Studies of low methodological quality in which the estimate of quality is incorporated into the meta-analyses can alter the interpretation of the benefit of intervention, whether a scale or component approach is used in the assessment of trial quality.

Introduction

The conduct of a meta-analysis is retrospective1 and is therefore susceptible to several sources of bias.2 Meta-analyses of randomised controlled trials (RCTs) include studies of variable methodological quality. Features of RCTs that confer the least biased estimates of treatment effect have been intensively studied lately. Differences in quality across trials may indicate that the results of some trials are more biased than others. Meta-analysts need to take this information into consideration to reduce or avoid bias whenever possible. Similarly, there are few data to guide reviewers as to whether any method of quality assessment provides a more biased estimate than any other. In this study, we addressed whether the method of quality assessment of RCTs by a validated scale approach rather than one involving individual components influences estimates of intervention efficacy.

Section snippets

Selection of meta-analyses

We randomly (random numbers table) selected 12 meta-analyses from our larger database of 491 meta-analyses of RCTs. Three inclusion criteria were used: that the report was published in English; that there was no formal incorporation of quality scores in the quantitative analysis; and that the outcomes were presented as binary data, reported as an overall quantitative summary result. Meta-analyses were excluded if the report did not provide references for the included trials. Nine of the

Trials

The 127 RCTs included in the 11 meta-analyses involved 10 492 patients. The 11 meta-analyses were published between 1988 and 1995 in ten journals or the Cochrane Database of Systematic Reviews. The trials on which they were based were published between 1960 and 1995, in 57 journals and three books. One study was unpublished. The majority of outcomes (15/22 [68%]) included can be defined as objective (eg, histological remission, major amputation, overall mortality, conception rate, smoking

Discussion

Assessment of the quality of reports of RCTs included in a meta-analysis adds another layer of complexity to the reviewing process. Our results suggest, however, that incorporation of an estimate of the quality of RCTs is important. We found a clinically important and statistically significant 30–50% exaggeration of treatment efficacy when results of lower-quality trials were pooled. Inflated estimates of treatment efficacy were found whether the trial quality assessments were made by a scale

References (30)

  • M Ramirez-Lassepas et al.

    Medical treatment of transient ischemic attacks: does it influence mortality?

    Stroke

    (1988)
  • AW Lensing et al.

    Treatment of deep venous thrombosis with low-molecular-weight heparins: a meta-analysis

    Arch Intern Med

    (1995)
  • TM Loosemore et al.

    A meta-analysis of randomized placebo control trials in Fontaine stages III and IV peripheral occlusive arterial disease

    Int Angiol

    (1994)
  • JJ Mari et al.

    An overview of family interventions and relapse on schizophrenia: meta-analysis of research findings

    Psychol Med

    (1994)
  • AJ Loonen et al.

    Continuation and maintenance therapy with antidepressive agents: meta-analysis of research

    Pharmaceutisch Weekblad

    (1991)
  • Cited by (0)

    View full text