Letting the right one in: evaluating the generalisability of clinical trials

Well-designed randomised controlled trials (RCTs) are the cornerstone of clinical research; they are the “gold standard” in comparing the efficacy and safety of two or more treatments in the fairest way. RCTs are essential in gaining regulatory approvals for new drugs; the US Food and Drug Administration (FDA) and the European Medicines Agency (EMA) used supporting RCTs in 90% of their approvals from 1999 to 2014 [1, 2]. The results from RCTs also provide the basis for recommendations made by guidelines, which inform evidence-based clinical practice.

A number of factors need to be considered when assessing the external validity, or generalisability, of RCTs, i.e. the degree to which the results apply to patients treated in routine clinical practice. These include, but are not limited to, the study population, outcome measures, trial duration, and the trial setting [3]. The inevitable conflict that arises with the requirement to achieve strong internal validity in the design of RCTs, in particular to meet stringent study inclusion and exclusion criteria to reduce bias, has meant that the generalisability of study findings to typical patient populations with the disease of interest can be poor [4].

The extent to which this limitation may apply to asthma has been shown by analyses which identified that the vast majority of adults with current asthma on treatment, in either primary or secondary care, would not have been eligible for inclusion in the major RCTs on which the international asthma guidelines are based [5, 6]. Highly selected populations were studied in these RCTs to achieve strong internal validity by confirming diagnostic labels, for example, excluding smokers to avoid including patients with chronic obstructive pulmonary disease (COPD), but thereby excluding around a third of patients with asthma, an important high-risk subgroup with a differential response to treatments such as inhaled steroids [7]. Similarly, the requirement to only include patients with substantive bronchodilator reversibility excludes most patients with asthma, as this characteristic is only present in a minority of patients with asthma on the basis of a single test [8]. The clinical implication is that there is not an adequate evidence base for the management of the majority of adults with asthma, with the inevitable risk of inappropriate overtreatment and/or undertreatment.

This issue has come into sharp focus with increased understanding of the heterogeneity in asthma, with recognition of clinically relevant phenotypes, such as type-2 high, eosinophilic patients. This has led to treatment options targeted at specific type-2 phenotypes [9], based on phenotype-specific clinical trials [10, 11]. Importantly, there is now a portfolio of monoclonal antibody therapies directed against specific type-2 targets, in which efficacy has been demonstrated in RCTs. Not least due to the high expense of these medications, it is necessary to scrutinise the external validity of the trials, to determine which specific populations should be preferentially considered for funding and prescription of these novel agents.

This has been addressed in the current issue of the European Respiratory Journal, in which Brown et al. [12] assess the eligibility of patients in their tertiary care severe asthma cohort, the Wessex Severe Asthma Cohort (WSAC), to enter RCTs of novel treatments for asthma. Their patients underwent detailed clinical, physiological and biological characterisation, and had confirmed severe asthma as per the American Thoracic Society/European Respiratory Society 2014 definition [13]. The authors report that 9.8% of their 342 patients with severe asthma would have been eligible for trials of biological treatments for severe asthma. A finding that a minority of patients were eligible may not have been surprising, as most of these trials targeting a specific part of the type-2 pathway required the presence of type-2 biology, a characteristic not required to be included in the WSAC cohort. Indeed, previous studies of mepolizumab have shown that applying type-2 targeted treatments to an unselected severe asthma cohort can mask the beneficial effects of the drug [14], which are clearly seen when tested in the right population [10]. However, the figure of around only one in 10 patients being eligible is extremely low and suggests poor generalisability of the trial's findings to a population with refractory severe asthma. Importantly, the authors report that the majority of patients who met the study's biomarker requirements were excluded due to criteria measuring airflow obstruction and/or variability. For example, only 21.1% of patients with severe eosinophilic asthma would have been eligible for inclusion in the phase III licensing trials of interleukin (IL)-5/5R targeted therapies. Thus, the vast majority of patients were excluded from trial participation by criteria designed to reconfirm diagnostic labels, rather than by biomarker criteria that identify the type-2 characteristics, and importantly predict responsiveness to treatment.

The funding and clinical availability of these new biological drugs is another consideration, and it is of interest that 50% of the WSAC cohort who are eligible for the anti-IL-5 monoclonal antibodies mepolizumab and reslizumab according to the UK National Institute for Health Care and Excellence (NICE) criteria would not have been eligible for the clinical trials which formed part of the NICE technology appraisals. NICE and other commissioning bodies have to apply their own methods to generalise the findings from RCTs conducted in narrow trial populations to a wider disease population.

The impact of patient selection criteria was evident from the 37 studies evaluated by Brown et al. [12], all of which evaluated drugs in severe asthma, in which 12 different criteria were used to define airflow obstruction. Some studies allowed one of a number of criteria to be used to demonstrate this and/or bronchodilator reversibility, but this only modestly increased trial eligibility. Differing criteria for defining a characteristic makes it increasingly difficult to generalise and apply RCT findings not only to a real-world patient, but also to other clinical trials.

Another question that needs to be asked is “what is the rationale for, and implications of the use of specific selection criteria?” In the case of type-2 targeted biological treatments, the desired effect is a reduction in airway inflammation and a subsequent reduction in exacerbation risk. Forced expiratory volume in 1 s (FEV₁), FEV₁/forced vital capacity ratio or bronchodilator reversibility have not been shown to predict a response to type-2 targeted treatments, and in some cases are negatively associated with response to treatment [15]. By contrast, markers of type-2 biology such as the peripheral blood eosinophil count or the exhaled nitric oxide fraction have been shown to predict response [16]. Diagnostic labels may also unnecessarily restrict the external validity of monoclonal antibody therapies for type-2 high inflammation. For example, mepolizumab has been shown to reduce exacerbations in asthma [10] and, to a lesser extent, in COPD [17] in patients with raised blood eosinophil counts regardless of the diagnostic label or lung function.

More broadly, the identification of individual causal mechanisms within a patient that have specific treatments is the basis of the “treatable traits” approach to the management of airways disease [18]. If the requirement for diagnostic labels which do not influence the treatment efficacy was removed, the focus could instead be on the biological trait of interest (figure 1).

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 1

External validity of randomised controlled trial results for targeted therapy is limited by a “diagnostic labelling” approach to participant eligibility criteria.

Agusti et al. [18] suggest a number of treatable traits in airways disease, including pulmonary traits such as airflow limitation and eosinophilic airway inflammation, extrapulmonary traits such as chronic rhinosinusitis and vocal cord dysfunction, and behavioural/lifestyle risk factors such as adherence, inhaler technique and exposure to sensitising agents. The challenge now is to determine not only the clinical utility of recognising and treating specific treatable traits, but also a practical system whereby this approach might be employed in clinical practice, both in primary and secondary care [19]. It will also be necessary to determine, for each trait, their prevalence in specific populations, their ease of diagnosis and the investigations required, the burden of disease, and the efficacy, safety and cost of treatment. These issues have been considered in a recent review of the Australasian Severe Asthma Web-based Database, in which McDonald et al. [20] explored the prevalence of 24 traits in two populations, and their impact on exacerbation risk. This approach allows the traits to be triaged by their prevalence, relative risks, and the impact of treatments on these risks [21].

The identification of type-2 phenotypes for which specific treatments are targeted, regardless of diagnostic labels, is arguably the best example of the utility and importance of the treatable traits approach in airways disease [22]. It will be important to learn from the experience obtained with the research programmes of the type-2 biologic agents in the assessment of other potential treatable traits in airways disease. This includes the internal and external validity of the RCTs on which their use in clinical practice, and funding, will be based.