Flu Vaccine Effectiveness: Questions and Answers for Health Professionals

How do we measure how well influenza vaccines work?

Two types of studies are used to determine how well influenza vaccines work. The first type of study is called a randomized control trial (RCT). In a RCT, volunteers are assigned randomly to either a group that receives vaccine or a group that receives a placebo (e.g., a shot of saline), and vaccine efficacy is measured by comparing the frequency of influenza illness in the vaccinated and the unvaccinated groups. RCTs are required before a new vaccine is licensed for routine use by a national regulatory authority, such as the Food and Drug Administration (FDA) in the United States. The second type of study is called an observational study. In observational studies the study participants make their own decisions about whether or not to be vaccinated. In this type of study, vaccine effectiveness is measured by comparing the frequency of influenza illness in the vaccinated and unvaccinated groups, usually with adjustment for factors (like presence of chronic medical conditions) that may vary between the groups. (See below for further details.)

What is 'vaccine effectiveness'?

Vaccine effectiveness is a measure of how well influenza vaccines work to protect against influenza infection and illness when they are used in routine circumstances in the community, and not specifically in a RCT. Effectiveness represents the percentage reduction in the frequency of influenza infections among people vaccinated compared with the frequency among those who were not vaccinated, assuming that the vaccine is the cause of this reduction. These studies are conducted in community settings, and researchers have no control over those who choose to be vaccinated or not.

How do vaccine effectiveness studies differ from vaccine efficacy studies?

Vaccine efficacy refers to studies of vaccine effects that occur under randomized, controlled conditions, where individuals are randomly assigned to either a group that is given influenza vaccine or to a second group that is not given influenza vaccine, but instead, given a placebo. A RCT is a study designed by researchers to minimize factors that could lead to invalid study results. For example, vaccine allocation is usually double-blinded, which means neither the study volunteers nor the researchers know if a given person has received vaccine or placebo. This methodology reduces bias that can occur if the researchers or the individuals receiving the intervention know which study volunteers have received placebo versus vaccine. Bias is an unintended systematic error in the way researchers select study participants, measure outcomes, or analyze data that can lead to inaccurate results.)

Top

When can vaccine effectiveness studies be conducted?

The most common approach now used to evaluate how well licensed influenza vaccines work is an observational or vaccine effectiveness study. Once an influenza vaccine has been licensed by FDA, recommendations are typically made by CDC's Advisory Committee for Immunization Practices (ACIP) for its routine use. For example, ACIP now recommends annual influenza vaccination for all U.S. residents aged 6 months and older. These universal vaccine recommendations make it unethical to perform efficacy (i.e., experimental randomized) studies with persons who are explicitly recommended to receive vaccine, especially because assigning people to a placebo group could place them at risk for serious complications from influenza.

What factors can affect the results of influenza vaccine effectiveness studies?

Effectiveness studies are subject to various forms of bias (see above for definition), more so than are efficacy studies. Therefore, it is important when evaluating the results of an influenza vaccine effectiveness study that researchers identify the potential biases and introduce methods to minimize them. There are at least three forms of bias that are especially important in interpreting the results of influenza vaccine effectiveness studies: confounding bias, selection bias, and information bias.

• Confounding bias occurs when the effect of vaccination on the risk of the outcome of interest (e.g., RT-PCR-confirmed influenza infection) is distorted by other factors associated both with vaccination and influenza infection. For example, confounding bias would occur if the majority of influenza cases in a case-control study have a chronic medical condition that places them at a greater risk of influenza hospitalization and makes them more likely to receive an influenza vaccine than non-cases. Not taking these associations into account for this study population would lead to an estimate of effectiveness that is too low due to confounding bias caused by cases having more chronic medical conditions than controls.
• Selection bias refers to errors introduced into a study because of differences between people who are enrolled in a study compared with people who are not enrolled. For example, people who are willing to participate in vaccine effectiveness studies might seek health care sooner, exercise more, or live healthier lifestyles than people who don't participate in such studies. As a result of this bias, study participants may not be representative of the general population, and the study results may be biased towards finding higher vaccine effectiveness, if vaccination worked better in such persons. Taking into account "health seeking behaviors" is especially important in vaccine effectiveness studies conducted among the elderly.
• Finally, information bias occurs when there are differences in the quality or accuracy of measuring vaccination status or influenza illness in the groups of people being compared in a study. For example, if researchers obtain information on vaccination for influenza cases from medical records but use verbal interviews to get this information from non-cases, this difference in data collection procedures could bias the results of the study.

The methods used to conduct observational studies of vaccine effectiveness must be reviewed carefully to see if these and other possible forms of bias have been described and addressed by adjustment for the factors that differ between groups.

Top

What outcomes are measured in influenza vaccine effectiveness studies?

The interpretation of vaccine effectiveness studies depends on the outcomes measured in a particular study. These outcomes may include prevention of laboratory-confirmed influenza illness or hospitalization, prevention of medically attended, acute respiratory illness (MAARI), prevention of influenza-like illness (or ILI, defined as an illness with fever and cough or sore throat), and prevention of pneumonia requiring hospitalization. In general, the more specific the outcome used (e.g., laboratory-confirmed influenza compared with ILI) the more accurate the measurement of the effect of vaccination. A recent study suggests that using serology (a laboratory test which measures the amount of antibody against a particular virus in a person's body) to determine whether or not study participants have been infected with influenza may potentially overestimate vaccine efficacy (Petrie et al, 2011). The reason for this is that following an influenza vaccination, a person's immune system may produce a significant amount of antibody against influenza. If that same person were to become infected with influenza despite being vaccinated (i.e., a "vaccine failure"), it is possible that enough additional antibody may not be produced to provide a positive result in a serology test. Diagnosis of influenza infection with serology tests requires two blood samples: one taken prior to infection, and the second taken after infection. Confirmation of influenza infection by serology tests requires a four-fold increase in antibody in the post-infection serum as compared to the level in the pre-infection serum. Because an influenza infection does not always produce a four-fold increase in antibody in people who have received the flu vaccine, vaccine failures can be missed, thus increasing estimates of vaccine efficacy. Despite this potential for bias, studies using serology-confirmed outcomes can still provide valid estimates of vaccine efficacy when considered with other disease endpoints.

Which outcomes provide the best estimates of vaccine effectiveness?

Studies that use more specific outcomes, such as laboratory-confirmed influenza outcomes (e.g., culture positive or reverse-transcriptase polymerase chain reaction (RT-PCR) positive results), provide the best and most specific estimates of the impact of influenza vaccines in preventing influenza. In general, when non-laboratory-confirmed outcomes are used (e.g., all pneumonia hospitalizations or influenza-like illness, which include many non-influenza illnesses), vaccine effectiveness estimates are lower. For example, a study by Bridges et al (2000) among healthy adults found that the inactivated influenza vaccine was 86% effective against laboratory-confirmed influenza, but only 10% effective against all respiratory illnesses in the same population and season.

How can vaccine effectiveness against non-laboratory-confirmed outcomes be interpreted?

The interpretation of vaccine effectiveness against less specific, non-laboratory-confirmed outcomes is influenced by the proportion of the outcome used that is actually caused by influenza virus infections compared with other pathogens. One non-laboratory-confirmed outcome that is often used is influenza like illness (ILI). The proportion of ILI caused by influenza viruses varies by year, and even varies within a specific year over the course of the winter. For example, in the results of a theoretical study graphed below, vaccine was 75% effective against laboratory-confirmed influenza, but it was only 30% effective against ILI when influenza caused 40% of ILI in unvaccinated people (Figure). Influenza vaccine would be estimated to be only 15% effective, however, if influenza viruses were responsible for only 20% of ILI at a particular point during the winter. This relationship is important because the percentage of ILI caused by influenza varies widely over time and geography.

Figure: The effect of non-influenza illnesses on an estimate of influenza vaccine effectiveness.

Top

Why do estimates of influenza vaccine effectiveness vary widely?

Estimates of influenza vaccine effectiveness are affected by several factors, including the specific study biases discussed above, the match between the vaccine influenza strains and the circulating strains, host factors and the sample size of a specific study. As noted above, the specificity of the outcome measured in a study has an important influence on the observed effectiveness. As more data are collected globally from annual studies that estimate effectiveness for RT-PCR confirmed influenza, it is expected that our estimates will become more refined. However, vaccine effectiveness will always vary from season to season, based upon the degree of similarity between the viruses in the vaccine and those in circulation, as well as other factors. In years when the vaccine strains are not well-matched to circulating strains, vaccine effectiveness is generally lower. In addition, host factors also affect vaccine effectiveness. In general, influenza vaccines are less effective among people with chronic medical conditions and among people age 65 and older, as compared to healthy young adults and older children.

How well do inactivated influenza vaccines work in randomized control trials?

As noted above, effectiveness varies with vaccine match and the age and immune function of the recipient. In general, the greatest benefits of influenza vaccines have been reported in randomized controlled trials (RCTs) conducted among healthy adults. For example, recent RCTs of inactivated influenza vaccine among adults under 65 years of age have estimated 50-70% vaccine efficacy during seasons in which the vaccines' influenza A components were well matched to circulating influenza A viruses (Beran et al., 2009, 2006-2007 season; Jackson et al., 2010, 2005-2006 season; Monto et al., 2009, 2007-2008 season). As vaccine efficacy from a randomized clinical trial is the gold standard for how well a vaccine actually works, vaccine effectiveness estimates obtained from observational studies can equal, but not exceed, estimates of efficacy. Many factors that can result in substantial bias in effectiveness studies tend to bias the vaccine effect downwards.

How well do influenza vaccines work during seasons in which the vaccine strains are not well matched to circulating influenza viruses?

When vaccine strains are not well matched with circulating influenza viruses, the benefits of vaccination may be reduced. For example, inactivated influenza vaccine effectiveness against laboratory-confirmed influenza was 60% among healthy persons and 48% among those with high-risk medical conditions in a case-control study among people 50–64 years old during the 2003-2004 influenza season, when the vaccine strains were not optimally matched to viruses in circulation (Herrera et al., 2007). However, in a year when the influenza vaccine and predominant circulating influenza viruses were poorly matched, researchers were not able to measure an effect of influenza vaccination against the respective vaccine component (Bridges et al., 2000). It is not possible to predict how well the vaccine and circulating strains will be matched in advance of the influenza season, and how this match may affect vaccine effectiveness.

Top

How well do influenza vaccines work in people with chronic high-risk medical conditions?

The presence of chronic medical conditions may also affect the effectiveness of influenza vaccines. For example, in an observational study of people 50–64 years of age, the vaccine was 60% effective in preventing laboratory-confirmed influenza among otherwise healthy adults 50–64 years of age, but only 48% effective among those who had high-risk medical conditions (Herrera et al., 2006). In general, vaccine efficacy and effectiveness estimates among people with high-risk conditions may be somewhat lower than among people of similar age without high-risk conditions. However, because the risk of influenza-related complications among this group is much higher, vaccination still provides important benefits.

Adults 65 years or older

Only one large randomized, controlled trial of influenza vaccine has been conducted among an elderly population. During the 1991-1992 influenza season, a group of Dutch people 60 years of age and older not living in long-term care facilities (e.g., nursing homes) was studied (Govaert et al., 1994). In this study, vaccine efficacy was 58% in preventing clinically-defined influenza with serologic confirmation of infection. There are no published studies of the efficacy or effectiveness of influenza vaccines in preventing laboratory-confirmed, serious outcomes of influenza such as hospitalization, primarily because the size of the study would be large, and therefore, such a study is very expensive to conduct. Published observational studies conducted among people 65 and older not living in long-term care facilities have used non-specific outcomes, such as pneumonia hospitalizations or all-cause mortality. These studies may be subject to substantial confounding and selection bias, and they use outcomes in which the proportion of illness associated with influenza virus infections vary by season (as other respiratory viruses can cocirculate). As a result, it is difficult to interpret the results of these studies.

Adults 65 years or older in long-term care facilities

All residents of long-term care facilities s (e.g., nursing homes) should receive annual influenza vaccination, as outbreaks of influenza can be explosive and result in substantial morbidity and mortality among residents of such facilities. There is evidence that vaccination prevents respiratory illnesses during periods of influenza circulation for elderly nursing home residents. For example, one study conducted during the 1991-1992 influenza season found that vaccination was associated with a 34% reduction in total respiratory illnesses and a 55% reduction in pneumonia during the two-week peak of influenza activity (Monto, 2001). In addition, one study conducted in UK nursing homes found that vaccinating health care workers decreased deaths during periods of influenza activity during one season with substantial influenza circulation, but not during the next year, when influenza activity was low throughout the winter (Hayward, 2006).

Children

In a four-year randomized, placebo-controlled study of inactivated and live influenza vaccines among children aged 1–15 years, vaccine efficacy was estimated at 77% against influenza A (H3N2) and 91% against influenza A (H1N1) virus infection (Neuzil et al., 2001). A two-year study of children aged 6–24 months found that the vaccine was 66% effective in preventing laboratory-confirmed influenza in one year of the study (Hoberman et al., 2003). Only children who were fully vaccinated (i.e., had either two doses if not previously vaccinated, or one dose if previously vaccinated) versus unvaccinated children were included in the analysis. In the other year of this study, few cases of influenza occurred, making it difficult to assess the vaccine's efficacy (Hoberman et al., 2003). Children younger than 9 years of age who have not been vaccinated previously are recommended to receive two doses of vaccine the first year they get vaccinated. In subsequent years, they need only one dose. This recommendation was made because many children younger than 9 years of age have not been infected with influenza viruses previously, and a booster dose is needed for them to produce a protective immune response.

Top

How effective is the live attenuated influenza vaccine (LAIV)?

This vaccine currently is licensed only for healthy, non-pregnant people between 2 and 49 years of age.

Healthy Children

Because LAIV (nasal spray) vaccine was licensed more recently than inactivated vaccines, there are more data available on its effects from large randomized trials. For example, a RCT conducted among 1,602 healthy children initially aged 15–71 months assessed the efficacy of trivalent LAIV against culture-confirmed influenza during two seasons (Belshe et al., 1998; 2000). In season one, when vaccine and circulating virus strains were well-matched, efficacy in preventing laboratory-confirmed illness from influenza was 93% for participants who received two doses of LAIV. In season two, when the A (H3N2) component was not well-matched between vaccine and circulating virus strains, efficacy was 86% overall.

Healthy Adults

A randomized, double-blind, placebo-controlled trial among 4,561 healthy working adults aged 18–64 years assessed multiple endpoints (i.e., targeted outcome measures), including reductions in self-reported respiratory tract illness without laboratory confirmation, absenteeism, health care visits, use of antibiotics, and use of over-the-counter medications for illness symptoms during peak and total influenza outbreak periods (Nichol et al., 1999). The study was conducted during the 1997-1998 influenza season, when the influenza vaccine and circulating A (H3N2) viruses were poorly matched. Vaccination was associated with reductions in severe febrile illnesses of 19%, and febrile upper respiratory tract illnesses of 24%.

Vaccination was also associated with fewer days of illness, fewer days of work lost, fewer days with health care provider visits, and reduced use of prescription antibiotics and over-the-counter medications. Among a subset of 3,637 healthy adults aged 18–49 years, LAIV recipients (n = 2,411) had 26% fewer febrile upper-respiratory illness episodes; 27% fewer lost work days as a result of febrile upper respiratory illness; and 18%–37% fewer days of health care provider visits caused by febrile illness, compared with placebo recipients (n = 1,226). Days of antibiotic use were reduced by 41%–45% in this age subset.

A randomized, double-blind, placebo-controlled influenza virus challenge study among 92 healthy adults (LAIV, n = 29; placebo, n = 31; inactivated influenza vaccine, n = 32) aged 18–41 years assessed the efficacy of both LAIV and inactivated vaccine (Treanor et al., 1999). The overall efficacy of LAIV and inactivated influenza vaccine in preventing laboratory-documented influenza from all three influenza strains combined was 85% and 71%, respectively. These results were obtained after study participants, all of whom were susceptible to recently circulating influenza viruses before vaccination, were experimentally exposed to viruses. The difference in efficacy between the two vaccines was not statistically significant.

How do live attenuated vaccine and inactivated vaccines compare in vaccine efficacy and effectiveness studies?

Few studies that directly compare live attenuated influenza vaccine (LAIV) and trivalent inactivated influenza vaccine (TIV) have been conducted, and results appear to differ for adults and children. More data are available for children than for adults. Among children, each of three RCTs comparing inactivated and live vaccines demonstrated that live vaccine offered better protection than inactivated vaccine. However, none of the studies included a placebo group, so the absolute efficacies of the two vaccines could not be assessed. One study included 2,187 children aged 6–71 months who had recurrent respiratory tract infections (Ashkenazi et al., 2006) and found overall influenza rates of 2.3% among live vaccine recipients and 4.8% for TIV, for a 52.7% decrease in children receiving live vaccine compared to those receiving inactivated vaccine. In a randomized study of 2,229 children aged 6–17 years with asthma, 4.1% of live vaccine recipients and 6.2% of TIV recipients developed influenza, for a relative reduction of 34.7% (Fleming et al., 2006). Finally, in 2004-2005 a multinational RCT was conducted among 8,352 children aged 6–59 months (Belshe et al., 2007). For the primary endpoint in this trial, culture-confirmed influenza-like illness, there were 45% fewer cases of influenza for well-matched influenza strains and 58% fewer for mismatched strains among live versus inactivated vaccine recipients.

In contrast to the studies in young children described above, a RCT conducted among primarily college age healthy adults was conducted during three influenza seasons using three assignment groups, including a placebo group. Overall, the results suggested that inactivated vaccine may be more efficacious than live vaccine for this age group. For example, in the final season of the study, absolute efficacy against the influenza A virus was 72% for the inactivated vaccine and 29% (not significant) for the live attenuated vaccine. Therefore, the relative improvement in efficacy offered by the inactivated vaccine was 60% (Monto et al., 2009).

The above studies taken together indicate that live and inactivated influenza viruses perform differently relative to each other in children and young adults.

Top

What information is necessary to make assessments of vaccine effectiveness?

Ideally, influenza vaccine effectiveness should be assessed on an annual basis, using a consistent methodology and similar populations. Use of a laboratory-confirmed outcome to assess vaccine effectiveness is important to provide the most specific results of the benefits of vaccination and to limit the impact of the co-circulation of non-influenza respiratory pathogens on estimates of vaccine effectiveness. Because the current recommendation for the United States is that all persons aged 6 months and older receive a vaccine each season, it is ideal if estimates of effectiveness can be made for children, adults and older adults. Because a proportion of older adults have chronic medical conditions and most in this age group seek vaccination, it is difficult to conduct and interpret influenza vaccine effectiveness in this population. CDC currently conducts annual vaccine effectiveness studies among people of all age groups recommended for annual vaccination (i.e., all aged 6 months and older). In addition, CDC conducts special studies targeted at answering more specific questions, such as estimating the effectiveness of inactivated vaccine in preventing laboratory-confirmed influenza hospitalizations among older U.S. residents.

Top

References

Ashkenazi S, Vertruyen A, Arístegui J, Esposito S, McKeith DD, Klemola T, Biolek J, Kühr J, Bujnowski T, Desgrandchamps D, Cheng SM, Skinner J, Gruber WC, Forrest BD; CAIV-T Study Group. Superior relative efficacy of live attenuated influenza vaccine compared with inactivated influenza vaccine in young children with recurrent respiratory tract infections. Pediatr Infect Dis J. 2006;25(10):870-9. PDF Link

Belshe RB, Mendelman PM, Treanor J, King J, Gruber WC, Piedra P, Bernstein DI, Hayden FG, Kotloff K, Zangwill K, Iacuzio D, Wolff M. The efficacy of live attenuated, cold-adapted, trivalent, intranasal influenza virus vaccine in children. N Engl J Med. 1998;338(20):1405-12. PDF Link

Belshe RB, Gruber WC. Prevention of otitis media in children with live attenuated influenza vaccine given intranasally. Pediatr Infect Dis J. 2000;19(5 Suppl):S66-71.

Belshe RB, Edwards KM, Vesikari T, Black SV, Walker RE, Hultquist M, Kemble G, Connor EM; CAIV-T Comparative Efficacy Study Group. Live attenuated versus inactivated influenza vaccine in infants and young children. N Engl J Med. 2007;356(7):685-96.

Beran J, Vesikari T, Wertzova V, Karvonen A, Honegr K, Lindblad N, Van Belle P, Peeters M, Innis BL, Devaster JM. Efficacy of inactivated split-virus influenza vaccine against culture-confirmed influenza in healthy adults: a prospective, randomized, placebo-controlled trial. J Infect Dis 2009;200(12):1861-9. PDF Link

Bridges CB, Thompson WW, Meltzer MI, Reeve GR, Talamonti WJ, Cox NJ, Lilac HA, Hall H, Klimov A, Fukuda K. Effectiveness and cost-benefit of influenza vaccination of healthy working adults: A randomized controlled trial. JAMA. 2000;284(13):1655-63. PDF Link

Fleming DM, Crovari P, Wahn U, Klemola T, Schlesinger Y, Langussis A, Øymar K, Garcia ML, Krygier A, Costa H, Heininger U, Pregaldien JL, Cheng SM, Skinner J, Razmpour A, Saville M, Gruber WC, Forrest B; CAIV-T Asthma Study Group. Comparison of the efficacy and safety of live attenuated cold-adapted influenza vaccine, trivalent, with trivalent inactivated influenza virus vaccine in children and adolescents with asthma. Pediatr Infect Dis J. 2006;25(10):860-9.

Govaert TM, Thijs CT, Masurel N, Sprenger MJ, Dinant GJ, Knottnerus JA. The efficacy of influenza vaccination in elderly individuals. A randomized double-blind placebo-controlled trial. JAMA. 1994;272(21):1661-5. PDF Link

Herrera GA, Iwane MK, Cortese M, Brown C, Gershman K, Shupe A, Averhoff F, Chaves SS, Gargiullo P, Bridges CB. Influenza vaccine effectiveness among 50-64-year-old persons during a season of poor antigenic match between vaccine and circulating influenza virus strains: Colorado, United States, 2003-2004. Vaccine. 2007;25(1):154-60. PDF Link

Hayward AC, Harling R, Wetten S, Johnson A, Munro S, Smedley J, Murad S, Watson JM; Effectiveness of an influenza vaccine programme for care home staff to prevent death, morbidity, and health service use among residents: cluster randomized controlled trial. BMJ 2006;333:1241. PDF Link

Hoberman A, Greenberg DP, Paradise JL, Rockette HE, Lave JR, Kearney DH, Colborn DK, Kurs-Lasky M, Haralam MA, Byers CJ, Zoffel LM, Fabian IA, Bernard BS, Kerr JD. Effectiveness of inactivated influenza vaccine in preventing acute otitis media in young children: a randomized controlled trial. JAMA. 2003;290(12):1608-16. PDF Link

Jackson LA, Gaglani MJ, Keyserling HL, Balser J, Bouveret N, Fries L, Treanor JJ. Safety, efficacy, and immunogenicity of an inactivated influenza vaccine in healthy adults: a randomized, placebo-controlled trial over two influenza seasons. BMC Infect Dis. 2010;10:71. PDF Link

Monto AS, Hornbuckle K, Ohmit SE. Influenza vaccine effectiveness among elderly nursing home residents: a cohort study. Am J Epidemiol. 2001;154(2):155-60. PDF Link

Monto AS, Ohmit SE, Petrie JG, Johnson E, Truscon R, Teich E, Rotthoff J, Boulton M, Victor JC. Comparative efficacy of inactivated and live attenuated influenza vaccines. N Engl J Med. 2009;361(13):1260-7 .

Neuzil KM, Dupont WD, Wright PF, Edwards KM. Efficacy of inactivated and cold-adapted vaccines against influenza A infection, 1985 to 1990: the pediatric experience. Pediatr Infect Dis J. 2001;20(8):733-40 .

Nichol KL, Mendelman PM, Mallon KP, Jackson LA, Gorse GJ, Belshe RB, Glezen WP, Wittes J. Effectiveness of live, attenuated intranasal influenza virus vaccine in healthy, working adults: a randomized controlled trial. JAMA. 1999;282(2):137-44. PDF Link

Petrie JG, Ohmit SE, Johnson E, Cross RT, Monto AS. Efficacy Studies of Influenza Vaccines: Effect of End Points Used and Characteristics of Vaccine Failures. JID. 2011;203(9):1309-1315. PDF Link

Treanor JJ, Kotloff K, Betts RF, Belshe R, Newman F, Iacuzio D, Wittes J, Bryant M. Evaluation of trivalent, live, cold-adapted (CAIV-T) and inactivated (TIV) influenza vaccines in prevention of virus infection and illness following challenge of adults with wild-type influenza A (H1N1), A (H3N2), and B viruses. Vaccine. 1999;18(9-10):899-906. PDF Link

Top