The importance of real-life research in respiratory medicine: manifesto of the Respiratory Effectiveness Group

Background

Randomised controlled trials (RCTs) are universally considered as the gold standard for evaluating the efficacy of treatments. Their main strength is that, through randomisation, they avoid any major imbalance between compared groups: therefore, observed outcome differences between groups at the end of the trial are most likely related to treatment effects. Since they are inherently prospective by design, they also permit stability throughout the study to ensure that all conditions remain optimal to test the hypothesis of interest. These include high-quality follow-up, reinforced adherence, etc. Consequently, these studies can reach the highest level of internal validity, provided that all quality standards are followed, such as those defined by CONSORT guidelines [1].

However, this should not lead to a generalisation of the applicability of findings, i.e. to credit these studies with a high level of external validity: by design, an RCT recruits a specific, well-defined population. Since human beings are biologically heterogeneous, they do not react uniformly to stimuli such as pharmacological agents. Thus, the results observed in a given population should be applied only in this population and not extrapolated to others without complementary testing [2].

A major difficulty here is that, in essence, subjects participating in a randomised trial may be significantly different from those who decline to participate or would not be invited based on the investigator's decision (selection bias). For this reason, a pragmatic statement could be that the populations and/or context/setting of an RCT may often not be fully representative of what could happen in real life.

This limitation makes other (complementary) designs important, with the potential to determine whether results observed in RCTs can be applied to wider populations, or whether other hypotheses should be considered and tested [3]. This is where real-life (or real-world) research (RLR) can contribute significantly by observing treatment effects in unselected patients. Thereby, RLR has the potential to 1) examine the possibility of extrapolating the results of RCTs to different patient populations, confirming or refuting effectiveness, and 2) examine the effects of the intervention in unselected populations. But the presence of bias could lead to misleading conclusions; for comparative effectiveness designs, the most frequent type of bias is the presence of more or less apparent differences between groups, which can be difficult to detect [4].

To benefit fully from what RLR can provide, high-quality protocols need to be developed and the interpretation of results needs to be cautious. As for RCTs, quality standards have been proposed and work is ongoing in this field to address and refine crucial methodological issues.

RLR is relevant for a wide range of stakeholders involved in the use, conduct, review and/or quality appraisal of therapeutic research: patients, clinicians, researchers, reviewers, policy makers, institutions, guideline developers, etc. [5].

The Respiratory Effectiveness Group (REG) was created in 2013 to promote high-quality real-life respiratory research. REG is involved in establishing and communicating quality standards for RLR, promoting RLR projects, providing leadership and examples of excellence in RLR, offering ethical review for RLR projects, engaging all stakeholders interested in RLR and facilitating collaborative networking in this field.

We define

RLR is research that includes the widest possible range of the target patients population, cared for in naturalistic conditions, with an intensity of follow-up that does not exceed what is provided in routine care [2].

Within the research framework proposed by the REG (figure 1), RLR corresponds to the top right of the graph (patients diagnosed as (condition), open design, naturalistic setting) with the “real-life” character decreasing towards the bottom left corner (highly selected patients, randomised blinded design, highly controlled setting).

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 1

The effectiveness research framework. Reproduced with permission of the publisher [2].

Tools other than the REG framework have been proposed to determine the extent to which a study is on the highly controlled versus fully naturalistic side of research, i.e. where it sits within the efficacy-to-effectiveness continuum [6].

We know

When performed appropriately [1], traditional RCTs are a source of highly reliable evidence on treatment effects, with high internal validity.

However, especially when performed for registration purposes, they rely on populations selected to maximise the chance of demonstrating efficacy. Investigators tend to avoid recruiting patients with too many comorbidities or concomitant treatments, both to comply with inclusion and exclusion criteria, control variables and to limit the risk of adverse effects that represent a significant burden for patients, and clinical and research resources. As a result, possible modifiers of treatment effects cannot be studied, which limits the generalisability of findings (limited external validity). For instance, many asthma RCTs do not include current smokers or obese patients, as both of these conditions are known to modify treatment effects [7, 8]. In such RCTs, patients receive detailed instructions and are closely followed to ensure proper inhaler use and adherence. They benefit from easier and quicker access to care, and often form stronger relationships with healthcare providers.

Altogether, several studies found that populations recruited in asthma or COPD treatment trials are representative of only a very small minority of these patient populations, not exceeding 15% in many studies [9–17].

RLR may be observational (prospective or retrospective, e.g. using medico-administrative databases) or interventional (pragmatic trials). Importantly, for each of these options strict quality criteria have been developed to allow readers to determine the strength of the produced evidence. These include the CONSORT statement extension for pragmatic trials [18], STROBE statement for observational studies in epidemiology [19], EMA-ENCePP checklist for pharmacoepidemiology and pharmacovigilance studies [20], UNLOCK initiative on quality criteria and minimal datasets requirements for observational studies [21], PRISMA for meta-analyses [22] and StaRI standards for implementation trials [23]. Recently, the REG has developed, tested and released the REal Life EVidence AssessmeNt Tool (RELEVANT) to assess the quality of real-life comparative effectiveness research [24, 25]. This tool relies on previously established quality standards proposed by REG and others [4, 26, 27] and is expected to facilitate both study design and appraisal.

Accordingly, the GRADE (Grading of Recommendations Assessment, Development and Evaluation) system acknowledges the possibility of upgrading the level of evidence provided by an observational study when all relevant quality criteria are satisfied, and the evidence is deemed “overwhelming”. Conversely, failure to meet these criteria will result in a downgrading of the evidence provided [28].

RLR cannot replace traditional RCTs but can be useful and even crucial to complement them: RLR studies of sufficient quality (high external validity) can help verifying and extending the applicability of RCT findings [29]. Although definitive conclusions cannot usually be drawn from results of observational studies (due to the possibility of residual confounding), they can be confirmatory and/or hypothesis-generating regarding, for example, treatment effectiveness in subgroups of interest, safety in a wide range of populations and cost-effectiveness in various healthcare systems. When possible, pragmatic trials could offer a stronger study design.

As such, RLR needs to be performed and accounted for by all stakeholders involved in issues relating to therapy, including patients, clinicians, researchers, reviewers, policy makers, institutions and guideline developers. Indeed, this requirement is acknowledged in some guidelines documents, such as the one produced by the Global Initiative for Chronic Obstructive Lung Disease on COPD management, which underlines the fact that a lack of external validity may compromise the applicability of trial results to a broader population [30]; similarly, the European Respiratory Society recently underlined the need to use evidence beyond RCTs when elaborating guidelines [31].

We advocate for

• The appraisal and inclusion of high-quality real-life studies in the development of clinical practice guidelines.

  •  o While theoretically any study can be used to inform guidelines, real-life studies are frequently excluded; this may lead to very low number of studies/subjects used and consequent agnostic positions that do not help in clinical practice, potentially compromising the applicability of guideline recommendations into practice.

  •  o RLR may improve the relevance of the guideline to the clinician.

  •  o RLR may contribute significantly to the identification of the best treatment options by identifying responsive populations, and adding value and knowledge to the evidence provided by RCTs.

• The use of implementation real life studies of interventions in different countries and settings to assess their impact in different populations, environments and healthcare systems.

• The use of real-life studies to complement RCTs for registration of new treatment modalities.

  •  o Regulators are now starting to understand that different types of real-life studies complement evidence for treatment modalities, in particular with regards to how their use can be optimised in practice. These real-life studies should be used to support decisions relating to the registration of new treatment modalities. The application of real-life studies in this context needs to be better established and utilised.

• The establishment of formal investigation plans/programmes of research/development and approval processes/post-marketing surveillance to include real-life studies.

  •  o In the life cycle of a product, real-life studies offer added value and can be helpful, particularly in post-authorisation phases. Long-term investigational plans including real-life studies produce added value for all stakeholders.

• Increased education and awareness of the scope, role and requirements of high-quality real-life studies in respiratory research.

  •  o There is some confusion around RLR and the term is used in variable ways. The limits and opportunities of different methodologies should be part of the education of both researchers, research administrators and practitioners.

• Continuing methodological research to expand and improve the quality and interpretability of real-life studies towards clinical practice.

  •  o The limits of real-life approaches have not yet been reached and innovation is welcome in incorporating new types of evidence into our knowledge base.

• The establishment of well-designed global registries around common respiratory conditions to prospectively collect key disease information.

  •  o Registries currently offer a cost-effective approach to longitudinal epidemiology with the potential for a much greater role in the future.

• The accelerated incorporation of RLR into technological advances in the form of digital outcomes, eHealth and mHealth in the research/registration/guideline process, in parallel to advances in analytical tools to evaluate big data.

  •  o The rate of technological advancement appears more rapid than our ability to evaluate and regulate their impact; the opportunity, however, of gaining (and using) knowledge through novel technological modalities has boundless possibilities, which need to be explored within a high-quality and structured framework of RLR.

We conclude

• RLR is now clearly positioned within the therapeutic research framework established by, for example, the REG.

• RLR (high external validity) is necessary to complement results of traditional RCTs (high internal validity) regarding treatment effects.

• RLR may be confirmatory, extend findings of RCTs in wider populations and settings, and generate hypotheses on specific subgroups or situations of interest.

• RLR may also provide evidence where RCTs cannot be performed for reasons relating to feasibility, ethics and affordability.

• As such, RLR should always be considered a potentially important contributor to fill knowledge gaps.

• High-quality RLR can and should be incorporated, by patients, healthcare professionals, guidelines developers and policy makers, to guide decision processes.

• Tools to rigorously assess the quality of RLR such as RELEVANT (developed by the REG) are now available and should be used systematically.

• Finally, researchers should always identify the most appropriate study design to answer a well-defined research question, be it classical RCTs, pragmatic RCTs, observational studies or implementation research designs.

Shareable PDF