Validity

The value of a population survey depends on maximising both precision (lack of random error) and validity (lack of systematic error).5 Systematic error includes selection bias (for example, through poor response rates), information bias (for example, through misclassification of “exposure” or disease), and confounding.6 We initially focus on issues of information bias (particularly misclassification of disease), but we also briefly consider issues of random error and selection bias. We do not consider confounding since this is likely to be of similar relevance for all of the survey instruments under consideration.

The validity of a survey instrument, with regard to the classification of disease, depends on its intended use and the measure of effect (either relative risk or risk difference) that is being used.7 In comparisons of differences in asthma prevalence within and between populations, Youden's index (sensitivity + specificity − 1) is the best single measure of validity.5 In contrast, in case-control studies of prevalence or other “aetiological” investigations in which the relative risk is the main effect measure of interest, the positive predictive value of a test is most relevant,7 but we do not consider this situation here.

It is important to emphasise that, when assessing which measure is most valid for population comparisons, this assessment must itself be based on random population surveys rather than on studies of selected clinical populations. Although BHR has validated well against asthma in clinical studies,8 this is partly a result of the case mix in such studies in which patients with reasonably severe asthma are compared with non-asthmatic subjects who have been screened for possible asthma risk factors such as atopy or a family history of asthma.9 BHR does not fare so well in general population surveys since these include many mild or borderline asthmatic subjects, as well as non-asthmatics who may have various “risk factors” for asthma.5

A recent review by Pekkanen and Pearce7 found that there have only been two general population surveys10 ,11that have independently compared BHR testing and standard symptom questionnaires with a standardised approach to physician diagnosed asthma. In both studies, although BHR had a greater specificity for asthma, it had a low sensitivity and questions on “wheeze” had a higher Youden's index value than BHR alone or in combination with symptoms. For example, Jenkins et al 10 studied population samples of 91 adults aged 28–44 years and 168 children aged 13–14 years, and compared the results from a symptom questionnaire and from a hypertonic saline challenge with the diagnoses of current asthma based on a blinded history taken by a trained physician. Self-reported symptoms had a higher Youden's index than BHR in both children (0.66 versus 0.43) and adults (0.76 versus 0.29), mainly because of the better sensitivity of symptom questionnaires (0.85 versus 0.54 in children, 0.80 versus 0.37 in adults). Combining symptoms with BHR increased the specificity, especially in children, but caused a strong decline in sensitivity, thereby decreasing Youden's index (to 0.41 in children and 0.36 in adults) to a lower level than the use of symptoms or BHR alone.

In addition to avoiding information bias in population surveys, it is also important to avoid selection bias and to maximise precision. Thus, the value of a population survey depends not only on using the most valid survey instrument, but also on the size of the survey and its response rate. Small surveys are more prone to random error, and surveys with poor response rates may be subject to selection bias—for example, whether asthmatic subjects are more likely to participate than non-asthmatic subjects. Symptom questionnaires have considerable practical advantages over BHR testing as they can be administered to larger numbers of participants with higher response rates.

Thus, current evidence suggests that BHR testing has no greater (and may even have lesser) validity than symptom questionnaires for measuring the difference in asthma prevalence between populations (or subpopulations) with the same language and similar symptom recognition and reporting.7 Furthermore, its use may reduce study sizes (thereby reducing precision) and response rates (thereby increasing selection bias).

Comparability

An alternative argument for using BHR testing in prevalence surveys is that it gives a more valid comparison between populations since it is “objective”, whereas responses to questions on symptoms can depend on a wide variety of psychological, social, and cultural characteristics including health care practices, as well as on the translation of the questionnaire. Thus, BHR may not be more valid when comparing populations or subpopulations which share the same language, culture, health care system, and perceptions and labelling of asthma symptoms, but it may provide more comparable (and hence more valid) information when comparing populations which do not share these characteristics. However, standardising the performance of BHR testing is a major problem, especially in international comparisons.12 Comparisons among children are especially difficult, and it has been concluded that BHR tests cannot be compared between children of different ages and sizes.13 The problem of low response rates in studies involving BHR testing is also of concern for international comparisons. For example, in the ECRHS study the response rates for the phase II testing (including BHR testing) were relatively low14 and differences between populations could, in part, have been the result of differences in response rates. An alternative approach to standardising comparisons between populations is the use of video questionnaires such as the ISAAC asthma video questionnaire.15 This appears to have similar or greater validity than written questionnaires, and is likely to avoid the problems of comparability of information from written questionnaires across populations.

What is the role of BHR testing?

What then is the role of BHR testing in asthma prevalence surveys? Is there any reason to use BHR testing at all? It might be argued that BHR is of interest and worthy of study in itself. However, it is clinical asthma that is the fundamental clinical and public health problem, and this should be the principal focus of population surveys. Nevertheless, we consider that there is an important role for BHR testing as a supplementary component of prevalence studies, but that it should be used sparingly and the findings must be interpreted carefully. BHR testing cannot provide validation of the existence of differences in the prevalence of asthma between populations. However, BHR is one possible mechanism by which asthma can occur, and BHR testing can therefore be used to assess whether the observed differences in prevalence are occurring through this mechanism. BHR testing is therefore most useful in terms of interpreting, rather than validating, the findings of symptom prevalence questionnaires and/or clinical examinations. Thus, when performing a prevalence survey, a good way to combine the best qualities of the symptom questionnaires and BHR testing is to perform a large phase I questionnaire survey (written or video) first and then to perform more intensive phase II examinations on a subsample to ascertain whether or not the observed differences in prevalence are caused by mechanisms involving BHR.

Conclusions

It is commonplace for epidemiologists to note that “BHR is not the same thing as asthma”, but nevertheless to use it routinely in population surveys since it is assumed to have greater “objectivity” than symptom questionnaires. In particular, it is assumed that BHR testing reduces information bias by avoiding the problems of subjective symptom recall that may occur with symptom questionnaires. However, when the focus is on estimating differences in the prevalence of asthma between populations who share the same language, symptom perception, and labelling and diagnostic practice, current evidence suggests that BHR has no greater (and may even have lesser) validity than symptom questionnaires. Furthermore, the increased response rates and larger sample sizes obtainable with symptom questionnaires indicate that they will, in general, have greater validity and precision than BHR testing. In comparisons across populations and/or language groups there is currently little evidence on the relative validity of BHR testing and symptoms, although it is evident that, once again, response rates are generally better, and possible study sizes larger, with symptom questionnaires. Furthermore, the video questionnaire approach provides an alternative method of collecting standardised data on the prevalence of asthma symptoms across populations.15

To explore the reasons for differences in the prevalence of asthma within and between populations, questionnaires can be supplemented with BHR and other testing in subsamples of symptomatic and non-symptomatic subjects. However, the main reason for doing this is to ascertain the mechanism by which population prevalence differences have occurred, and not because BHR provides a more valid measure of asthma prevalence. Furthermore, if we define asthma by combining symptoms with BHR, we not only lose validity, but we also lose the possibility of studying the separate contributions of these factors to differences in asthma prevalence. Asthma symptoms and BHR should therefore be analysed separately rather than combined in a single definition. The method of choice for population prevalence comparisons is standardised written or video symptom questionnaires, and BHR testing should be regarded as a supplement to, rather than a replacement for, such questionnaires.