Abstract
Two potential sources of systematic variation in output from Mefar dosimeters, the system used in the European Community Respiratory Health Survey (ECRHS) study have been evaluated: individual nebulizer characteristics and dosimeter driving pressure.
Output variation from 366 new nebulizers produced in two batches for the second ECRHS were evaluated, using a solute tracer method, at a fixed driving pressure. The relationship between dosimeter driving pressure was then characterized and between-centre variation in dosimeter driving pressure was evaluated in an Internet-based survey.
A systematic difference between nebulizers manufactured in the two batches was identified. Batch one had a mean±sd output of 7.0±0.8 mg·s−1 and batch two, 6.3±0.7 mg·s−1 (p<0.005). There was a wide range of driving pressures generated by Mefar dosimeters as set, ranging between 70–245 kPa, with most outside the quoted manufacturer's specification of 180±5%. Nebulizer output was confirmed as linearly related to dosimeter driving pressure (coefficient of determination (R2)=0.99, output=0.0377×driving pressure–0.4151). The range in driving pressures observed was estimated as consistent with a variation of about one doubling in the provocative dose causing a 20% fall in forced expiratory volume in one second.
Systematic variation has been identified that constitutes potentially significant confounders for between-centre comparisons of airway responsiveness in the European Community Respiratory Health Survey, with the dosimeter driving pressure representing the most serious issue. This work confirms the need for appropriate quality control of both nebulizer output and dosimeter driving pressure, in laboratories undertaking field measurements of airway responsiveness. In particular, appropriate data on driving pressures need to be collected and factored into between-centre comparisons. Comprehensive collection of such data to optimize quality control is practicable and has been instigated by the organizing committee for the European Community Respiratory Health Survey II.
Mefar nebulizers and dosimeters have been used in the European Community Respiratory Health Survey (ECRHS) from its inception 1. The ECRHS is now consolidating its second study. The authors have established previously that aerosol output can differ significantly between batches of the same model of nebulizer 2, and that differences in the driving pressure developed by different Mefar dosimeters also affects aerosol output 3. This would imply that potential systematic differences in Mefar dosimeter output could confound both absolute measurements of airway responsiveness (AR), as well as differences in between-centre comparisons of AR, to the degree that has been reported by the ECRHS 4, 5. Thus, small but real differences between centres may remain undetected if potential differences in dosimeter output are not allowed for, while conversely, apparent trends or even significant regional differences may simply reflect systematic bias caused by different dosimeter system outputs.
The potential importance of such issues has been recognized by the ECRHS coordinating committee, who commissioned its one Australian centre in Melbourne to calibrate all nebulizers to be used in the second ECRHS. Therefore, the authors set out to measure the aerosol output from 366 new Mefar nebulizers, which had been produced in two batches. Following the authors' previous study, the ECRHS coordinating committee also recognized the possibility that appropriate data on driving pressures needed to be collected and factored into between-centre comparisons 3. Before comprehensive collection of such data was instigated for the ECRHS II, further evaluation of the potential issue was undertaken in a two-part experiment: in the first part, the effect of changing dosimeter driving pressure on the aerosol output of five representative nebulizers was directly assessed. In order to assess how much variation exists in the field, an Internet-based survey was then undertaken to collect information about actual operational between-centre variations in Mefar dosimeter driving pressures.
The hypotheses of this study were that driving pressure affects output from Mefar dosimeters, and that driving pressure variation in the field is a significant potential confounder of AR measurement in both an absolute and relative sense.
Materials and methods
Three hundred and sixty-six new Mefar nebulizers were studied, in conjunction with a single Mefar (IV) dosimeter (MefarTM, Brescia, Italy). The nebulizers calibrated were manufactured in two batches, designated batches one and two for the purposes of this study.
Static driving pressure
Each individual nebulizer has a distinct but variable resistance to flow and when attached in series with the dosimeter this is reflected in variable driving pressures measured downstream. Static driving pressures developed by the Mefar dosimeters were therefore measured without a nebulizer attached. Driving pressure was measured with a simple analogue pressure gauge (Dobbie Instruments, Sydney, Australia) connected to the flow outlet of the dosimeter. The pressure gauge had a range of 0–250 kPa, a 100-mm face diameter and bottom-fitting nipple inlet. In accordance with the ECRHS protocol, during pressure measurement the dosimeter was programmed with a 6-s delay between activations to allow sufficient time for the gas pressure to build up within the dosimeter holding chamber between activations.
Nebulizer output measurements: variation of nebulizers used in the European Community Respiratory Health Survey phase II
A previously validated method was used for determining true solute as opposed to gravimetric nebulizer output, because the latter is markedly affected by evaporative loss 6, 7. Briefly, each nebulizer was filled with 4 mL of LiCl 0.24 M solution and activated five times with a 6-s pause between activations. Aerosol was entrained onto a glass fibre filter paper and the LiCl subsequently desorbed in 5 mL of purified water and measured by flame photometry 6, 7.
All 366 nebulizers from the two manufactured batches were calibrated using a single dosimeter with a static driving pressure accurately adjusted to 180 kPa. Prior to the calibration of each batch of five nebulizers, the dosimeter was checked to ensure a pressure of 180 kPa ±5% was produced. If the pressure had declined with use, it was corrected by increasing the output pressure using the pressure control valve inside the dosimeter. Driving pressures were re-measured in triplicate before and after each set of five nebulizer calibrations.
Variation of nebulizer output with Mefar dosimeter driving pressure
Five nebulizers, selected at random, were studied with the dosimeter set sequentially at different static driving pressures, which were achieved easily by adjusting the pressure control valve inside the dosimeter. Measurements were performed in triplicate at 150, 180, 210 and 250 kPa, and again driving pressure was checked in triplicate before and after measurements.
Field survey of Mefar dosimeter driving pressures
An Internet-based survey of dosimeter driving pressures used in different centres around the world was undertaken, supplemented where possible by personal contact with researchers using Mefar dosimeters. The survey was intended to assess the degree of possible variation in centres using Mefar dosimeters. It included European and nonEuropean ECRHS centres, as well as field researchers assessing AR in non-ECRHS projects (“Healthwise”, Australia). All centres assessing AR in ECRHS II were notified of the survey, by an e-mail canvas carried out through the ECRHS coordinating committee.
The departmental website included illustrated instructions on how to alter dosimeter pressure and flow characteristics 8.
Statistical methods
Aerosol output was related to driving pressure by linear regression and the square of the correlation coefficient (R2) was calculated. The coefficient of repeatability (i.e. the value below which the difference between two measurements will lie, with probability 0.95) of the nebulizer calibrations was examined in previous publications 3, 7 using the Bland and Altman method 9.
Results
Variation in output from nebulizers used in European Community Respiratory Health Survey phase II
There was a significant difference between the two batches of nebulizers. The mean±sd aerosol output for batch one was 7.0±0.8 mg·s−1 (fig. 1⇓) and for batch two, 6.3±0.7 mg·s−1 (p<0.005 batch one versus batch two, fig. 2⇓). The overall mean output of all the nebulizers was 6.4±0.8 mg·s−1, with a range of 3.1–8.4 mg·s−1. The coefficient of repeatability for the lithium analysis was 0.8 mg 3, 7.
Variation of nebulizer output with changing Mefar dosimeter driving pressure
Figure 3⇓ represents a scatter plot of the relationship between aerosol output and driving pressure for the five individual nebulizers studied, and shows that aerosol output for each was related linearly to dosimeter driving pressure. The mean aerosol output of the five nebulizers increased linearly as a function of driving pressures. (fig. 4⇓, R2=0.99, output=0.0377×driving pressure–0.4151).
Field survey of Mefar dosimeter driving pressures
Data on 15 dosimeters from Australia, Europe and North America (table 1⇓) were obtained. There was a wide range of driving pressures (70–245 kPa) generated by Mefar dosimeters from different centres (table 1⇓). The median driving pressure for centres using unchanged manufacturer's equipment was 186 kPa, interquartile range 159–213, range 70–245 kPa.
The factory-specified driving pressure for the Mefar dosimeter is 180 kPa. For the centres using unchanged equipment, the mean difference from this was 3 kPa, but with marked variation (sd 54 kPa). There was also variability between individual Mefar dosimeters used in the same location (e.g. a centre that used three individual Mefar dosimeters had individual driving pressures of 215, 220 and 245 kPa).
As well as reported variation in the equipment around “manufacturer's specifications”, there was evidence of previously unreported individual customizing of equipment following local repair. For example, one North American centre reported that their Mefar dosimeter internal compressor had broken down and been replaced with a compressed air cylinder set at 180 kPa, and a Swedish centre used wall-based “hospital air”, as a replacement for the original compressor.
Discussion
Two potential sources of systematic variation in output from Mefar dosimeters, the system that was chosen and which has been uniformly used in the ECRHS, were evaluated. This study quantified the extent of output variation from 366 new Mefar nebulizers intended for the second ECRHS. A systematic difference was identified between nebulizers manufactured in the two batches, and a great deal of difference between centres in effective nebulizer driving pressures was also observed.
In an extension of the commission to act as the ECRHS nebulizer calibration centre, the potential for variation in driving pressures to constitute an important determinant of dosimeter output was evaluated. In this part of the work dosimeter output was shown as a linear function of dosimeter driving pressure, and in a complementary Internet-based field survey, the hypothesis suggesting that there was considerable variation in operational driving pressures in the field was confirmed. The authors, therefore, further characterized systematic variation that constituted potentially significant confounders for between-centre comparisons of AR, measured using the Mefar system. This emphasizes the practical importance of the decision of the ECRHS organizing committee to instigate comprehensive collection of such data for ECRHS II, following the preliminary work in this area 3.
This work is a logical sequel to a previous report, in which the authors observed that driving pressures from their own laboratory's three Mefar dosimeters were significantly different from the manufacturer's specifications 3. This initial work was unable to comment on the prevalence of variability elsewhere in the ECRHS, but helped to construct the hypothesis that there would be considerable variation in driving pressures in the rest of the field study centres. Indirect support for this came from the published experience of the ECRHS. In a retrospective study of outputs from Mefar nebulizers used in phase I, a mean aerosol output of 4.8 mg·s−1 was observed with the dosimeter used, utilizing a similar fluoride tracer technique that has been described previously 1. This output was unusually low from experience, since typically a mean output between 5.5–8.0 mg·s−1 is observed 2, 3, 6 with this system. Therefore, it was speculated that the observed output may have been caused by the individual Mefar dosimeter being used, having been set inadvertently and unknowingly, with an inappropriately-low dosimeter driving pressure 6. From current data, it could be estimated by extrapolation that this dosimeter was, in fact, set at a driving pressure of ∼140 kPa. The current data confirm that there is considerable variation in Mefar dosimeter driving pressures, with some operating at extremely low pressures, consistent with the apparently low aerosol outputs published by the ECRHS team.
It is noteworthy that the majority of the instruments tested in this study were not operating according to manufacturer's specifications, with respect to driving pressures. These results, particularly with regard to dosimeter driving pressures, indicate that there are likely to be systematic differences between aerosol output from different ECRHS centres, and, indeed, within the same centre where different dosimeters are used. Based on the range in dosimeter driving pressures reported for uncustomized equipment, the potential effects can be modelled on the predicted dose of methacholine, calculated to cause a 20% fall in the forced expiratory volume in one second (PD20). Hence, the authors predict (if a given dose-response is linear) that a “true” PD20 measurement of 100 µg, with a dosimeter driving pressure of 180 kPa, could be reported as being anywhere between 80 µg (a centre using a high driving pressure of 216 kPa) and 140 µg (a centre using a low driving pressure of 130 kPa). This represents a potential artefact that may confound comparisons of AR if necessary quality control assessments are not undertaken. The use of a log dose-response slope as the index for AR may normalize for this problem to some extent, but this depends on uniformity of dose-response linearity, and upon uniform and accurate local centre output calibrations.
Fortunately these results, while a cause for concern, are not of themselves an insurmountable problem for between-centre comparisons of AR within ECRHS, if knowledge of the extent of the problems is disseminated, and the prudent quality controls instigated by the ECRHS committee are adhered to by centres in the field. As a minimum, the authors reiterate that all centres should routinely perform and record appropriate checks on dosimeter driving pressures, and that this information is collated by the ECRHS for subsequent inclusion in between-centre analyses. The variability in nebulizer output from different batches can simply be addressed by central calibration of nebulizers followed by randomized allocation, as has now occurred in this phase of the ECRHS.
A potential criticism of the present study is that although data regarding all of the nebulizers used in the second ECRHS were recorded, driving pressure data from all centres were not. However, the authors tried to maximize this by working through the ECRHS coordinating committee and using e-mail communication. In an effort to obtain data about the potential extent for variation, the authors also included data from a non-ECRHS study that used the Mefar system to measure AR. It is possible, although highly unlikely, that all other centres run their dosimeters at the correct driving pressure, and always have. The routine quality control measurements that are recommended should eventually allow a more complete picture regarding this important variable, but the authors also feel that the present data are robust enough to represent important information for centres using the Mefar dosimeter system, and particularly for the ECRHS.
The situation will become more problematic again, if more centres make unilateral modifications of the dosimeter system, or other aspects of protocol, such as replacing the internal compressor with an air cylinder or hospital air. This was described by two of the ECRHS centres surveyed, one of which as a nonEuropean centre has not taken part in the formal ECRHS quality programme (personal communication, ECRHS committee). Such a replacement is likely to alter the characteristic pressure profile associated with the mechanical compressor. In an observation associated with the authors' previous work with the Mefar dosimeter 3, it was shown that with a programmed activation time of 1 s, 50% of which was spent at peak pressure (180 kPa) with a nonlinear fall-off of driving pressure thereafter, systems such as compressed air, with no reservoir to compress, exhibited a squared activation profile, with >80% of the activation time spent at peak driving pressure. This would have a directly proportional effect on increasing nebulizer output. This comment is not meant as a criticism of centres trying their best in the face of frustrating operating conditions, but it was a worrying feature of the present survey that one-third of centres reported breakdowns with the Mefar system, which led to some ad hoc arrangements. Such problems, together with the batch variations in Mefar nebulizers and the fact that the majority of dosimeters do not comply with the manufacturer's own specifications with regard to driving pressures in their operational set position, represent a concern for a system that needs to be robust enough for field surveys. Reassuringly, this individual centre's experience is that after initial equipment breakdowns had been fixed, subsequent operations of the dosimeters were reliable. In addition, the available quality control data indicate that dosimeter driving pressures, once correctly set, are stable for up to 12 months.
The implication of this, should it be verified by the more comprehensive quality control data collection underway in ECRHS II, is that following a simple adjustment, all ECRHS equipment could be readily and verifiably standardized with respect to driving pressure, with little subsequent drift. The authors hope to report on this in the future.
In summary, the authors have identified systematic variables that constitute potentially significant confounders for between-centre comparisons of airway responsiveness in the European Community Respiratory Health Survey. This work confirms the need for appropriate quality control in laboratories undertaking field measurements of airway responsiveness and indicates a need for appropriate data on driving pressures to be collected and collated. Between-centre comparisons need to factor in such variations. The authors have demonstrated that collection of such data is practicable and worthwhile, and a logical extension of this work will be future analysis of the comprehensive collection of such data instigated for European Community Respiratory Health Survey II, and furthermore, how airway responsiveness measurements have been affected in the field.
Acknowledgments
The authors would like to thank D. Jarvis and J. Knox of the ECRHS coordinating team, P. Liakakos and K. Bennetts for laboratory assistance, and all who responded to the survey, in particular, E. Norrman, F. Burgos, J. Maldonado Perez, and J. Ikin (for “Healthwise” Australia).
- Received March 9, 2001.
- Accepted October 12, 2001.
- © ERS Journals Ltd