Estimates of influenza vaccine effectiveness have mostly been derived from nonrandomised studies and therefore are potentially confounded. The aim of the current study was to estimate influenza vaccine effectiveness in preventing mortality among the elderly, taking both measured and unmeasured confounding into account.
Information on patients aged ≥65 yrs from the computerised Utrecht General Practitioner database on eight influenza epidemic periods and summer periods was pooled to estimate influenza vaccine effectiveness in preventing mortality. Summer periods (during which no effect of vaccination was expected) were used as a reference to control for unmeasured confounding in epidemic periods.
After adjustment for measured confounders using multivariable regression analysis, propensity score matching and propensity score regression analysis, influenza vaccination reduced mortality risk (odds ratios (ORs) 0.58 (95% confidence interval (CI) 0.46–0.72), 0.56 (95% CI 0.44–0.71) and 0.56 (95% CI 0.45–0.69), respectively). After additional adjustment for unmeasured confounding (as observed during summer periods), the association between influenza vaccination and mortality risk decreased (OR 0.69 (95% CI 0.52–0.92)).
We conclude that after state-of-the-art adjustment for typical confounders such as age, sex and comorbidity status, unmeasured confounding still biased estimates of influenza vaccine effectiveness. After taking unmeasured confounding into account, influenza vaccination is still associated with substantial reduction in mortality risk.
Annually, influenza epidemics are associated with high mortality rates, notably among elderly persons 1, 2. Since the introduction of influenza vaccines, only one randomised double-blind trial has been conducted among (younger) elderly persons, and influenza infection was halved in the vaccine group compared with the placebo group 3. Large-scale trials evaluating more serious outcomes such as mortality are not available, in part because of the large sample size needed, and also due to ethical constraints. Instead, several nonrandomised observational studies have set out to estimate the effects of influenza vaccination on serious outcomes among elderly persons 4, 5. In 2007, results were published of a 10-yr United States Health Maintenance Organisation data-pooling project, including more observations than available in all meta-analyses, and findings of substantial mortality reduction of the same magnitude as in previous studies were observed 6. However, recently there has been a debate regarding the validity of findings from such nonrandomised studies 7, 8. The main concern is that selection of patients for influenza vaccination in daily practice has resulted in incomparable groups of vaccinated and unvaccinated subjects, which may have led to considerable confounding bias 9, 10.
Several methods have been proposed in order to adjust for measured confounders, but unmeasured confounders are likely to result in residual bias. For example, functional health status, which is not routinely collected in medical databases, is an important potential confounder 11, 12. Nichol et al. 6 quantified the potential effect of such an unmeasured confounder using sensitivity analysis and showed that only in unlikely confounder scenarios was influenza vaccination not associated with mortality reduction.
Alternatively, the use of reference periods has also been proposed to quantify unmeasured confounding, since vaccine effectiveness can be considered known during these periods 7. For example, in pre-influenza 7, 13, 14, or summer periods 6, 15, 16, during which influenza virus activity is low or virtually absent, vaccine effectiveness is expected to be low or absent as well. Pre-influenza (or peri-influenza) epidemic periods, however, cannot be considered first-choice reference periods, since influenza virus activity is low (not absent) 2, and expected effects are, therefore, unclear. Hence, the use of summer periods without influenza activity has been suggested as a valid reference period to quantify unmeasured confounding 15, 17. The present study assessed mortality risk after influenza vaccination among community-dwelling elderly persons in a retrospective cohort study in the Netherlands during eight influenza seasons, taking both measured and unmeasured confounding into account.
Until 2007, the Dutch immunisation guideline on influenza vaccination recommended vaccination for specific patient groups with high-risk medical conditions and for all persons aged ≥65 yrs. In the Netherlands, the uptake of influenza vaccination among elderly persons has been high, with levels well over 70% after 1995 5, 18. The computerised medical database of the Netherlands University Medical Center Utrecht, General Practitioner (GP) Research Network, includes cumulative information on ∼60,000 patients enlisted with 33 GPs. The database complies with Dutch guidelines on the use of medical data for research purposes and has been shown to be valid in influenza vaccination studies 5, 19. Diagnoses are coded according to the International Classification of Primary Care (ICPC) coding system.
For the present study, we obtained clinical information on all elderly persons aged ≥65 yrs over eight influenza epidemic periods (1995/1996–2002/2003). In accordance with previous studies, influenza epidemic periods were defined as periods of at least two consecutive weeks in which each week accounted for at least 5% of the season’s total number of influenza isolates 2, 20. The number of isolates was in accordance with a laboratory-based surveillance conducted by the Weekly Sentinel System of the Dutch Working Group on Clinical Virology in the Netherlands. Importantly, peak influenza periods were largely separated from peak respiratory syncytial virus (RSV) periods 2. Furthermore, similar information was obtained during eight consecutive summer periods in which influenza isolates were infrequent or absent (1996–2003). Summer periods were defined as periods from week 20 until week 40 of each year. This period was selected as a reference period, for which we expected vaccination to provide no benefit, since influenza is not circulating during this summer period 6, 7, 17. In agreement with other observational studies, we collected extensive information on exposure to seasonal influenza vaccination and on potential confounders, such as age, sex, comorbidity and prior healthcare consumption for each observation period. Vaccination status was ascertained by registration of the ICPC code R44.1. Earlier studies have shown a high agreement between the presence of this code in the medical database and vaccination status (κ = 93%) 5. Comorbidity status was based on registration of ICPC codes during the 12 months preceding each year’s influenza epidemic period: cardiovascular comorbidity (acute myocardial infarction (code K75), congestive heart failure (K77), other cardiovascular diseases (K74, K76, K78–80, K82–84) or stroke (K90)); pulmonary comorbidity (lung cancer (R84, R85), asthma or chronic obstructive pulmonary disease (R91, R95, R96)); diabetes (T90); and malignancies (B72, B73, B74, D74–77, S77, T71, U75–77, X75–77, Y77). Furthermore, healthcare consumption (number of GP visits) and medication use in the year preceding each influenza epidemic period were recorded 5.
Based on an earlier study, we expected a vaccination rate of 70% 18 and a mortality rate of 1% during influenza epidemic periods 2. To detect a relative mortality risk reduction of at least 30%, with a statistical power of 80% and a two-tailed α-level of 0.05, the minimum required sample size was 51,000 periods of observation.
Methods to adjust for measured confounders
Three hierarchical sets of confounders were defined. The first set included only demographics (age and sex). The second set added information on prior healthcare use (number of GP visits) to the set of demographics. The third set added information on comorbidity status (cardiovascular and pulmonary comorbidity, diabetes mellitus and malignancies) and prior medication use. We used three methods (propensity score matching, propensity score regression analysis and multivariable regression analysis) to adjust for the measured confounders, and each method was used on each of the three sets of confounders 8. All methods were used on data from influenza epidemic periods as well as summer periods.
Propensity score matching
Propensity scores estimate the probability of being exposed, independently of outcome status 21, 22. Using multivariable logistic regression modelling, propensity scores of being vaccinated were calculated, including potential confounders as predictors in the model. We developed different models for each set of confounders. The main aim of propensity score analysis was to balance confounder distributions between groups of vaccinated and unvaccinated subjects for different strata of the propensity score (ranging from 0 to 1). Propensity scores were stratified in quintiles and subjects were pair-matched on vaccination status within these quintiles. In the matched dataset, the effects were estimated using conditional logistic regression analysis. This procedure of matching and analysis was repeated 1,000 times and the resulting distribution of effect estimates provided an overall effect estimate (mean) and 95% confidence intervals (CIs).
Propensity score regression analysis
Similarly to the propensity score matching procedure, propensity scores were calculated for the different sets of confounders. These scores were included as a single, continuous covariate in a logistic regression model estimating the association between influenza vaccination and mortality.
Multivariable regression analysis
Multivariable logistic regression analysis was used to calculate effect estimates. Inclusion of potential confounders in the model was based on univariate associations with both vaccination status and mortality. Three hierarchical models were constructed, based on the aforementioned sets of confounders.
Subjects could contribute more periods of observation to the study. These periods were assumed to be independent, when applying propensity score methods and logistic regression analysis. Subsequent influenza epidemic periods within one subject might, however, not be independent. This was verified by means of generalised estimating equations (GEE) techniques, which can be considered longitudinal logistic regression analysis 23, in which the influence of potential within-person dependency was assessed.
With GEE, potential within-person dependency is taken into account by assuming a correlation structure for the observations within persons. We used the least restrictive correlation structure, i.e. the unstructured correlation structure 23. Clearly, if the results of multivariable logistic regression analysis and GEE analysis are similar, within-subject dependency does not affect estimates and the assumption holds that different observations within the same subject can be considered independent.
Method to adjust for unmeasured confounders
For each method and each set of confounders an effect estimate was calculated in influenza epidemic periods and in summer periods. The latter was used to adjust the effect estimate obtained during the influenza periods for unobserved confounding. During summer periods no benefit of vaccination was expected with an expected odds ratio (OR), as a measure of association, of 1.0 6, 7. Therefore, deviations of the associations during summer periods from the expected OR (1.0) were used to quantify unmeasured confounding bias. Effect estimates calculated for influenza epidemic periods were adjusted for the amount of unobserved confounding measured during summer periods as follows:
ORadj = ORepidemic/ORsummer = exp(βepidemic-βsummer)
in which β indicates the regression coefficient for influenza vaccination 24. To estimate a 95% CI of this ratio of ORs, we sampled 100,000 times from the distributions of effect estimates for epidemic and summer periods. By each time taking the ratio of the two sampled numbers, we arrived at a distribution based on 100,000 ratios. The 2.5% and 97.5% quintiles of this distribution indicated the lower and upper bound of the 95% CI of the ratio of ORs, respectively. All analyses were carried out in R for Windows (version 2.5.1; R Foundation for Statistical Computing, Vienna, Austria).
Pooling of different influenza epidemic periods resulted in 50,906 periods of observation. In 37,501 (73.7%) of these periods the influenza vaccine was taken. Vaccinated subjects were older and had a higher prevalence of different classes of comorbidity, and they more often visited their GP during the 12 months preceding influenza vaccination (table 1⇓). These numbers did not materially differ in individual years that were studied. In total, 415 subjects died during the influenza epidemics (1.04 per 1,000 weeks of observation). Pooling of consecutive summer periods resulted in 50,069 periods of observation and, in 36,757 (73.4%) periods, influenza vaccine was administered in the vaccination year preceding the summer period. During the summer periods, 854 subjects died (0.85 per 1,000 weeks). Without adjustment for confounders, influenza vaccination did not show a clear effect on mortality risk during influenza epidemic periods (OR 0.86, 95% CI 0.69–1.06), whereas during summer periods influenza vaccination was associated with increased mortality risk (OR 1.20, 95% CI 1.02–1.40). Adjustments for age, sex and prior healthcare use as confounders resulted in a decreased OR of the association between influenza vaccination and mortality risk compared with the crude association in all three methods (fig. 1⇓). Importantly, additional inclusion of the potential confounders, presence of high-risk comorbidity and medication use, did not further affect the adjusted association, even though these covariates were univariately associated with both vaccination status and mortality (table 1⇓).
After full adjustment using a multivariable logistic regression analysis, an OR of 0.58 (95% CI 0.46–0.72) was observed. Propensity score regression analysis (OR 0.56, 95% CI 0.45–0.69) and propensity score matching (OR 0.56, 95% CI 0.44–0.71) showed similar associations (table 2⇓). Confounders were well balanced between groups of vaccinated and unvaccinated subjects among different propensity score quintiles (table 3⇓). In accordance with table 1⇑, the group of patients with the highest propensity score (those with the highest probability of being vaccinated) had the highest prevalence of comorbidity. Within quintiles of the propensity score, vaccinated and nonvaccinated subjects were comparable with respect to demographics and comorbidity status.
After adjustment for measured confounders, influenza vaccination reduced mortality during summer periods, even though no effect was expected (e.g. OR 0.84, 95% CI 0.71–1.00, for multivariable regression analysis). Each estimated association during summer was taken as a measure of unobserved confounding for the respective adjustment method applied and the specific set of confounders. After adjustment for this unmeasured confounding, the OR of the association between influenza vaccination and mortality risk stabilised at ∼0.7 for all sets of confounders and all methods applied (table 4⇓ and fig. 1⇑).
For measured confounding, adjusted estimates of vaccine effectiveness during influenza seasons were somewhat lower for persons aged ≥75 yrs (OR 0.66, 95% CI 0.50–0.86, for multivariable regression analysis) than among those aged 65–74 yrs (OR 0.45, 95% CI 0.30–0.67), although 95% CIs were largely overlapping (p-value for interaction 0.74). After additional adjustment for unmeasured confounders with summer as reference, vaccine effectiveness remained higher in those aged 65–74 yrs (OR 0.57, 95% CI 0.33–0.98) than among persons aged ≥75 yrs (OR 0.76, 95% CI 0.54–1.06).
Taking potential within-person dependency into account by means of GEE did not materially affect the effect estimates: after full adjustment for measured confounders and taking dependence into account, the multivariable regression analysis resulted in OR 0.58 (95% CI 0.47–0.72) during epidemic periods, and for summer data the OR was 0.84 (95% CI 0.72–0.99).
This large cohort study among elderly persons covering several years of observation showed that, after adjustments for measured and unmeasured confounding, influenza vaccination was associated with a reduction in mortality risk of ∼30%. Since full adjustment, for measured confounders only, resulted in a higher estimate of vaccine effectiveness in reducing mortality risk of ∼40%, 10% of this observed effect is likely to be caused by healthy user bias.
In a recently published study by Jackson et al. 14 on the effects of influenza vaccination on the risk for community-acquired pneumonia, no association was observed (OR 0.92, 95% CI 0.77–1.10). In the present study, pre-influenza data were used to select potential confounders for a multivariable model, such that the model provided an OR of 1.0. This model was then used to assess the effects of influenza vaccination during the influenza epidemic. Since influenza virus activity is low during pre-influenza periods 2, expected effects are unclear and the expected association between influenza vaccination and the risk for pneumonia may not be an OR of 1.0. Therefore, the selection of covariates for the multivariable model based on pre-influenza data could be biased. Furthermore, subjects that are likely to die shortly after vaccination, yet before the influenza epidemic, typically will not apply for the vaccine. Adjustment for typical confounders such as age, sex, comorbidity status and healthcare use may not control this confounding 13. However, the effects of such possible deterioration of health status may have faded by the time the influenza epidemic starts. During summer periods, however, the (short-term) reasons not to take the vaccine will possibly have less impact on mortality rates than during pre-influenza periods and, hence, these reference periods have been suggested previously to provide a more valid estimation of unmeasured confounding 6.
An important finding of our study is that, during summer periods, influenza vaccination appeared associated with a reduction in mortality of ∼16%, after adjustment for measured confounders. This finding accords with previous studies and might indicate potential for unmeasured confounding 13, 15, 16. For example, in a population-based cohort study over three influenza seasons by Ortqvist et al. 15, influenza vaccine effectiveness against all-cause mortality was estimated to be 44, 40 and 37% for the different seasons. Adjustment by means of summer periods decreased these numbers to 14, 19 and 1%. The low 1% effectiveness might be due to limited influenza virus activity during the 2000/2001 winter season. Our study size was adequate to answer our primary research question, but inadequate to conduct analyses in individual influenza epidemic periods or in selected periods during influenza seasons (e.g. early- or late-season periods), or to make comparisons between seasons with high and low influenza virus activity. Ortqvist et al. 15 defined the influenza seasons as the period December 1 to April 30, whereas in our study influenza epidemic periods were based on the relative number of influenza isolates per week 2, which indicates the period with pronounced influenza activity. Hence, in the study by Ortqvist et al. 15, the effect estimate might be diluted due to inclusion of nonepidemic weeks. Since the Netherlands is a relatively small country, nationwide surveillance data are appropriate to indicate epidemic periods in the study region. In addition, since in the Netherlands almost all citizens are registered with a specific general practice and GPs are the entry points for secondary care, for the vast majority of subjects virtually all medical data (including hospital discharge letters) are recorded in primary care. Hence, misclassification of vaccination status, confounders or mortality is unlikely. Furthermore, our study population comprised community-dwelling elderly persons, whereas the Swedish study included nursing home patients as well. Finally, in the Netherlands, a country with no large-scale pneumococcal vaccination, confounding by pneumococcal vaccination would not have materially affected our estimates of influenza vaccine effectiveness.
In a study by Mangtani et al. 17, in which data over ten influenza epidemic and summer periods was pooled, no effect of influenza vaccination was observed during summer periods, after adjustment for observed confounders (OR 1.01, 95% CI 0.96–1.06). During influenza seasons, influenza vaccination reduced the risk for death due to a respiratory disease by 12% (OR 0.88, 95% CI 0.84–0.92). This estimate is lower than our estimated 30% reduction of all-cause mortality by influenza vaccination, possibly due to inclusion of nursing home residents in the study by Mangtani et al. 17. Furthermore, in this British study, effects of influenza vaccination might have been underestimated due to RSV activity during the influenza season. In our study, peak influenza periods were largely separated from peak RSV periods 2.
The observed OR of the association between influenza vaccination and mortality risk during summer periods was lower than anticipated (i.e. lower than 1.0), probably due to healthy user bias. However, which unmeasured confounder yields this bias is unclear. Functional health status has been proposed as an important confounder. In contrast, a study in a population of Dutch elderly persons did not indicate functional health status as a confounder 25. Another explanation might be that, even though influenza activity is hardly detected during summer, a small amount of virus is still present and active, thus resulting in a reduced mortality risk among vaccinated subjects. Since the number of reported isolates was low during these periods, this seems highly unlikely. A third explanation might be that functional health status deteriorates in the course of influenza illness during influenza epidemic periods and remains impaired even several months after the influenza epidemic has ended 26. If this deterioration is prevented by influenza vaccination, lower mortality rates can be observed after epidemic periods, i.e. during summer periods. Finally, selection bias might have been the cause for the observed associations during summer. Only the subjects (either vaccinated or unvaccinated) that survived influenza epidemic periods contributed to the observations in summer periods. However, baseline characteristics for the vaccinated and nonvaccinated subjects from epidemic and summer periods were similar. Furthermore, the association between influenza vaccination and mortality, adjusted for both measured and unmeasured confounding, remained constant for different sets of observed confounders (fig. 1⇑), because adjustment for measured confounders had the same effect in both influenza epidemic and summer periods. Therefore, it is unrealistic to assume that a selected subgroup was included in summer periods, and residual confounding or selection bias is therefore unlikely to have materially affected the estimated associations of influenza vaccine effectiveness.
We used three methods to adjust for observed confounders, namely multivariable regression analysis, propensity score matching, and including propensity scores as a covariate in regression analysis. These methods produced similar results and were also approximately equally precise. These findings correspond to previous studies indicating that these methods give approximately the same results 10, 27–29. Propensity score methods can be useful to reduce the number of covariates to be included in a multivariable model in case of limited sample size. Unfortunately, propensity score methods and multivariable regression analysis can only adjust for measured confounders. For interventions such as influenza vaccination, reference periods can be used to adjust for unmeasured confounding. In other cases, possible effects of unmeasured confounding can be quantified by means of sensitivity analysis 6, 30, 31.
In conclusion, nonrandomised studies on influenza vaccine effectiveness are prone to confounding bias. Measured confounding can be adjusted for by several methods. Using summer reference periods is a powerful method to take unmeasured confounding into account. After adjusting for both measured and unmeasured confounding, influenza vaccination was associated with a 30% reduction in all-cause mortality during influenza epidemics among elderly persons and efforts should continue to vaccinate these high-risk persons.
What is already known on this topic
Since most evidence for influenza vaccine effectiveness in terms of reduction of mortality among the elderly has been derived from nonrandomised studies, selection of patients for influenza vaccination may have induced confounding bias and, hence, vaccine effects might have been overestimated. Summer periods have been used as a reference period to quantify unmeasured confounding.
What this study adds
In the present study, in which data on eight influenza epidemic periods were pooled, unmeasured confounding was taken into account by estimating influenza vaccine effectiveness during a summer reference period. After adjustment for both measured and unmeasured confounding, influenza vaccination was still associated with substantial mortality risk reduction.
This study is part of a personal grant of E. Hak, to study confounding in observational intervention studies, from the Netherlands Scientific Organization (The Hague; NWO-VENI no. 916.56.109).
Statement of interest
None declared.None declared.
- Received December 16, 2008.
- Accepted January 19, 2009.
- © ERS Journals Ltd