Assessment of generalisability in trials of health interventions: suggested framework and systematic review

Can the intervention be delivered elsewhere?

Several factors affect whether an intervention can be delivered and received in other sites. Firstly, an intervention must be feasible. Providers will vary in their capacity to implement an intervention,^w3 as will institutions in being suitable places for an intervention.^w4 The presence of local “champions” may influence feasibility in a particular site.4 Some interventions require the existence of other health services4—for example, services for treating sexually transmitted infections require microbiology laboratories to target the right patients. Interventions may also require adequacy in other sectors such as transport. Feasibility has a cost dimension: an unaffordable intervention lacks general feasibility.

Secondly, an intervention must achieve adequate coverage. This may depend on the overall comprehensiveness of health systems or on whether providers can reach people in other ways—for example, through outreach. Adequate coverage may be more difficult in some sites or sub-populations.

Finally, an intervention generally must be acceptable to be effective. Acceptability refers to participants' assessment of their experience of an intervention and will influence whether recipients adhere to treatment plans, act on health advice, or return for follow-up.4 For example, condom promotion has proved acceptable and subsequently effective in urban Tanzania but not in rural regions.^w5 Acceptability will vary between populations as it depends on cultural norms and can have economic dimensions. For example, HIV voluntary counselling and testing services that require clients to attend clinics twice (first for testing and then for results) may be acceptable in high income settings but not low income settings because transport or opportunity costs are too great.^w3

Factors relating to delivery of an intervention are best documented by embedding an evaluation of process in trials.5 The study collects quantitative and qualitative data on planning, delivery, and uptake and how context affects them.

Does the intervention meet recipients' needs?

To be effective an intervention must meet recipients' needs—that is, the recipients must have capacity to benefit from an intervention. Thus potential recipients of an intervention should have similar needs to those of the original study participants. Trial participants may be untypical of the general population even in the study site, let alone in other sites. Trials tend to under-represent certain groups, such as minority ethnic and low income groups, women, and older people, whose needs may differ from those of people included in trials.6 Trials should therefore describe the sociodemographic profile of participants and report the extent to which they are representative of the target population.

If the needs of future potential recipients differ from those of the study participants, interventions may not work in a new population or have to be adapted. For example, provision of antiretroviral drugs in low income countries, or to certain sub-populations may have to be accompanied by support to promote adherence in order to achieve similar outcomes to those achieved among trial participants.^w6

This is also true of public health interventions. The extent to which a factor contributes to the incidence of a particular disease, and therefore needs intervention, varies across populations. For example, treating ulcerative sexually transmitted infections may have a significant effect on HIV incidence in an HIV epidemic localised within high risk groups but not in a more generalised epidemic.^w1 Assessing whether an intervention has met recipients' needs, or will meet those of future recipients, requires investigators to be explicit about the causal pathways through which an intervention is expected to act and to measure relevant pathway variables.

Current assessment of generalisability

We reviewed whether trials of HIV prevention targeting homosexually active men explored generalisability or factors affecting this. We obtained and examined all available evaluation reports of eight interventions that a recent systematic review reported to have rigorously evaluated outcomes.7 Two reviewers independently assessed whether the studies had empirically examined local factors affecting feasibility, coverage, and acceptability; evaluated process; assessed needs; and assessed the potential generalisability of interventions.

Six of the eight trials had integral process evaluations,8^–14 16 18 but only three of these collected quantitative and qualitative data on the planning, delivery, and receipt of the intervention (table 1).9 12 18 Only one process evaluation stated that consideration of generalisability was an aim.10 Six trials gave some information about participants' ethnicity (usually the proportion described as white).8 9 15^–18 Seven trials provided data on educational level.8 9 12 14^–17 None commented on the extent to which study samples were representative of the populations being targeted.

Only those trials incorporating process evaluations identified contextual factors influencing the feasibility, coverage, and acceptability of their intervention (table 2). Elford et al, for example, reported that recruitment and retention of peer educators to provide HIV prevention in gyms was extremely difficult because of educators' low confidence.10

Only one study reported on needs (table 2).19 Although other studies reported baseline sexual behaviour8 9 12 14^–17 or sexual health related attitudes or knowledge8 12 16 of the target population or participants, the purpose was to check for baseline differences between intervention and comparison groups rather than to describe normative need.

Most of the studies speculated about the potential generalisability of their intervention to other sites but did not consider this empirically. Rosser et al, for example, wondered whether their intervention might prove more effective among populations with more risky sexual behaviour.17 The trials that examined contextual barriers and facilitators to delivering the intervention could make more considered assessments of generalisability. Two reports referred to sociological theory to hypothesise what contextual factors might have influenced the effect of the intervention in the study site compared with other sites.10 12 However, these trials both reported on interventions previously reported as effective in other contexts^w7 that were largely ineffective in their own sites. Therefore, rather than consider the scope for transferring the interventions to new sites, they (reasonably) considered the contextual reasons for failure of transfer.

Systematic evaluation

To make informed decisions about whether they should implement interventions, providers require more information than simply whether interventions are effective in original study sites. They need information on context and needs. However, most of the studies we looked at did not empirically examine generalisability. Phase III trials should be judged not only in terms of the designs and methods they use to examine outcomes3 but also how they assess generalisability. To enable this trials should:

• Include process evaluations as integral elements5

• Develop evidence based theories about how intervention processes are influenced by context^w8 and how processes might differ if interventions are implemented in other sites^w9

• Report the extent to which their participants are representative of the population being targeted6

• Describe the prevalence of the needs being met by the intervention, informed by clear hypotheses about the intervention's mechanism.

We believe that these elements are essential to comply with the existing CONSORT requirement to report on “clinical characteristics” of participants if clinical is interpreted as meaning need for health intervention.

The most useful information on the potential for, as well as the barriers to, transfer of interventions comes from studies that compare an intervention in one site with similar interventions provided elsewhere, as in the study by Elford et al.10 Future phase III research might build on such work by setting out to examine interventions implemented across diverse contexts in multisite studies. These would examine differential effects by site and explore contextual determinants of success to generate hypotheses for future research and guidelines for the implementation of interventions outside trials.^w9 This approach is compatible with a phased approach to intervention trials. Assessing generalisability in phase III should inform choice of sites for phase IV replicability research.20 However, such multi-site evaluations are unlikely unless funding for such work is increased.

Finally, systematic reviews should consider generalisability. Currently, many do not examine intervention process or context and do not comment on the potential for and limits to intervention effects being generalised to other settings and populations.^w10