On January 11, 2013, the CDC published an article in its Morbidity and Mortality Weekly Report (MMWR) with early-season data on the effectiveness of this year’s flu vaccine. The key finding: Vaccination was only 62% effective overall in keeping people from getting a bad enough case of influenza to end up seeing a doctor about it.
To people who follow flu, the fact that the flu vaccine was only 62% effective wasn’t surprising. But the CDC’s candor about it was. CDC Director Dr. Tom Frieden and CDC flu epidemiologist Dr. Joe Bresee actually held a press briefing to talk about the MMWR article and other current flu developments. I was so impressed by the briefing that I wrote a column praising it. The column’s title says it all: “The CDC Tells More of the Truth than Usual about Flu Vaccine Effectiveness.”
Enthusiastic but not giddy, I wrote: “It remains to be seen whether Frieden’s candor about vaccine effectiveness is a one-off or a new policy.”
Well, it didn’t take long to find out.
On February 21, 2013, MMWR published an update with three weeks more of data. The overall vaccine effectiveness estimate had slid a bit from 62% to 56% – though the CDC rightly pointed out that the two numbers are statistically equivalent, each within the other’s statistical confidence interval. More interestingly, the additional data made it feasible to subdivide the “overall” vaccine effectiveness estimate, calculating separate estimates for different age groups and different flu strains. (The earlier data had been subdivided by flu strain but not by age group.)
The most stunning finding was the vaccine effectiveness estimate for people 65 and over (by far the age group most likely to get seriously ill, require hospitalization, or die from the flu), for the H3N2 strain (by far the most virulent circulating strain in humans and the main one circulating this year). That number was 9%. In other words, this year’s flu vaccine is estimated to work less than 10% of the time at protecting the most vulnerable age group from needing to see a doctor for the most dangerous kind of flu.
It’s worth noting the 95% confidence interval for the 9% estimate: -84% to +55%. That means we are 95% sure that the actual effectiveness of this year’s flu vaccine in keeping seniors from going to the doctor with H3N2 is somewhere between -84% (vaccinees are 84% worse off than non-vaccinees) and +55% (vaccinees are 55% better off than non-vaccinees). This is a huge confidence interval, mostly because there weren’t enough unvaccinated seniors in the study sample – only 85 of them, to be precise. (This in turn is because most Americans 65 and over do as they’re told and get their flu shots.) With such a large confidence interval, we can’t be anything like sure that 9% is the right number. But it’s the best guess so far.
The MMWR article presented vaccine effectiveness estimates for four age groups, for two kinds of flu – influenza H3N2 and influenza B. (There are several kinds of B circulating, but they weren’t distinguished.) That’s eight estimates. Seven of the eight ranged between 46% and 75%. So the 9% estimate for the 65-and-over age group for H3N2 is definitely odd-man-out. It is highly probable that this year’s flu vaccine gives seniors little if any protection from H3N2.
Stop and ponder this bad news for a minute. Insofar as we’re interested in protecting the most vulnerable age group from needing medical attention for the most dangerous flu strain, our best guess is that this year’s vaccine helped a little (9%). Statistically, the 9% estimate isn’t significantly different from zero – but the confidence interval for seniors was so big that even a much higher number probably wouldn’t have been significantly different from zero. Statistics aside, what matters is that it’s damn close to zero.
This is a far cry from proof that flu shots are useless for seniors. Although all experts agree that the elderly benefit less than younger people from flu vaccination, most experts think that in most years a flu vaccination is probably better than nothing, even for seniors. Even this year, when a flu vaccination wasn’t demonstrably better than nothing at keeping seniors from seeking medical attention for symptoms that turned out to be H3N2 influenza, it probably did seniors some good against influenza B (the effectiveness estimate was 67%, with a confidence interval of -10% to +90%). And it might – might – have done some good even against H3N2 with regard to other outcome measures that the MMWR article didn’t address, like reducing hospitalizations or deaths.
Still, there’s no way around the fact that this year’s flu vaccine effectiveness estimate for seniors is disappointing.
Decent effectiveness estimates from previous flu seasons are extremely hard to come by – so it isn’t clear whether or not seniors were better protected in previous H3N2-dominated seasons. A 2012 meta-analysis of flu vaccine efficacy studies by Osterholm and colleagues found no studies of flu vaccine in the elderly that met their inclusion criteria. None. Not one. Some of the media coverage of this new study implies or states that this year’s vaccine was way less effective in seniors than usual. Maybe so, but the evidence is weak. If the CDC’s experts think 9% is unexpectedly and untypically low, then this is a bad year for the flu vaccine for seniors, and they ought to say so. If they think 9% isn’t all that surprising, then the flu vaccine is routinely much less effective for seniors than we have been led to believe – and they ought to say that. Either way, the study is bad news for seniors and bad news for flu vaccination proponents.
The risk communication question is the extent to which the CDC was candid about the bad news.
- On the plus side: The CDC funded this research – and however reluctant it might have been to do so, the CDC published the resulting paper in its premier, carefully watched journal, the MMWR.
- On the minus side: The CDC spun the story like crazy in its effort to avoid its devastating implications for seniors, the people who most loyally get their flu shot every year.
Spin is a much less serious sin than cover-up would have been, but it falls short of the standard of flu vaccine effectiveness candor the CDC set five weeks earlier.
The evidence of spin is most readily found in a Q&A posted on the CDC website to accompany the MMWR article … and presumably to replace it for readers interested enough to find their way to the CDC website but not quite geeky enough to immerse themselves in the article itself. The Q&A is entitled “What You Should Know for the 2012–2013 Influenza Season: Questions & Answers.” Its first and longest section, “Vaccine Effectiveness for 2012–2013,” is devoted to the MMWR article.
I’m not going to annotate the vaccine effectiveness section of the Q&A in as much detail as I annotated the CDC’s press briefing five weeks earlier. A few bad examples from the Q&A should suffice.
But the single worst example isn’t in the Q&A. It’s about the Q&A. More specifically, it’s a CDC tweet announcing and linking to the Q&A. Here it is:
From: @CDCFlu Sent: Feb 22, 2013 8:19a
Flu #vaccine offered good protection for children and adults younger than age 65 this season. http://t.co/ToMv0viBXA
Neither the MMWR article nor the Q&A uses the word “good” to describe this year’s flu vaccine effectiveness. Both use “moderate” – itself a debatable word choice, but not as indisputably dishonest as “good.” An agency that until recently claimed 70–90% effectiveness for the flu vaccine in healthy adults under 65 is now prepared to call 56% – a few percentage points higher than 56% when you leave out the seniors – “good”? But the biggest dishonesty in the tweet isn’t its claim that this year’s flu vaccine “offers good protection” to those under 65. The biggest dishonesty is its omission of what is obviously the study’s most important finding about this year’s flu vaccine: protection for seniors against the H3N2 strain that is indistinguishable from no protection at all.
The Q&A is more honest than the tweet. It doesn’t start with the most important finding – it buries the lede, in journalists’ jargon – but it does discuss that finding in considerable detail.
Here is the Q&A’s answer to the question “How well is the vaccine working this season?” (For the sake of simplicity, I have cut the confidence interval information; I have kept the comparisons to data from earlier in the season and to other studies, but cut the links and some of the details.)
Interim data from 2,697 children and adults enrolled through the U.S. Influenza Vaccine Effectiveness (Flu VE) Network during December 3, 2012–January 19, 2013, were used to estimate the overall effectiveness of seasonal flu vaccine for preventing laboratory-confirmed flu virus infection. Overall, the VE estimate was 56% … after adjustment for age, study site, race/ethnicity, self-rated health, and days from illness onset to enrollment. This finding is very similar to the earlier interim estimate…. The adjusted VE against laboratory-confirmed flu A (H3N2) and flu B was 47% … and 67% … respectively….
These results indicate the vaccination with the 2012–2013 flu season vaccine reduced the risk of flu-associated medical visits from influenza A (H3N2) by one half and from influenza B by two-thirds for most of the population. Overall VE estimates suggest that the 2012–2013 influenza vaccine has moderate effectiveness for most age groups against circulating flu viruses, similar to previously published reports. The one exception to this was the VE among people 65 and older against influenza A (H3N2) viruses. The adjusted VE against outpatient medical visits due to laboratory-confirmed influenza A (H3N2) in adults aged 65 and older was 9%.
These overall estimates are within the range of what is expected during seasons when most circulating flu viruses characterized by CDC are like the viruses included in the vaccine, which is what we are seeing this season. These findings also are similar to those published in a recent meta-analysis … which summarized the benefits of flu vaccines using data from randomized controlled clinical trials. In addition, the estimates also are consistent with mid-season flu vaccine effectiveness (VE) estimates for preventing medically-attended flu in Canada and the United Kingdom published in the journal Eurosurveillance on January 31, 2013.
Influenza vaccination, even with moderate effectiveness of about 60%, has been shown to also reduce the following: flu-related illness, antibiotic use, time lost from work, hospitalizations, and deaths.
Except for the last two sentences of the second paragraph, this summary is all about what isn’t new: As usual, the flu vaccine was somewhere around 60% effective overall. This is still a surprisingly low number to most people (including most journalists and many doctors). But as the summary stresses, it’s comparable to the CDC’s numbers earlier in the season and in previous seasons, comparable to recent findings in Canada and the U.K., and comparable to the results of a 2012 meta-analysis of the best studies.
The last paragraph quoted here is a little over-broad in its claim that the flu vaccine “has been shown” to reduce “flu-related illness, antibiotic use, time lost from work, hospitalizations, and deaths.” I didn’t cut any citations or links from that paragraph; there weren’t any. The evidence on behalf of some of these outcomes is spotty, especially if you drill down to inquire separately about each age group and each flu strain.
But my main complaint is about the last two sentences of the second paragraph: “The one exception to this was the VE among people 65 and older against influenza A (H3N2) viruses. The adjusted VE against outpatient medical visits due to laboratory-confirmed influenza A (H3N2) in adults aged 65 and older was 9%.” What’s missing from this account?
- The CDC doesn’t mention – here or anywhere in the Q&A – that the “one exception” to its overall results is the age group that accounts for 90% of influenza deaths and the flu strain that most often kills them.
- The CDC doesn’t mention – here or anywhere in the Q&A – that in this study 9% was not significantly different from zero statistically, and that there is therefore no evidence that this year’s vaccine gave seniors any protection at all from H3N2.
- The CDC doesn’t mention – here or anywhere in the Q&A – that this is hugely upsetting news to the most loyal constituency for influenza vaccination, seniors.
Something more complicated is also missing. Nowhere in the Q&A does the CDC grapple with the obvious follow-up question: Is 2012–2013 unusual in some way, or might the flu shot be routinely as ineffective for seniors as it was this year, at least in flu seasons dominated by the H3N2 strain?
The closest it comes is in its response to the question, “Do flu vaccines work in people 65 and older?” The statistical answer to this question this year is no. (Because of the very large confidence intervals for seniors, even the effectiveness estimate for influenza B wasn’t statistically significant.) Here’s the CDC’s answer:
Some older people, including those with certain chronic illnesses, are at greater risk of serious flu complications, but also may respond less well to vaccination. Human immune defenses become weaker with age, which places some people 65 and older at greater risk of flu-related complications. People 65 and older account for about 65% of flu-related hospitalizations each year and about 90% of flu-related deaths. Aging and chronic health problems can also diminish the body’s ability to mount a protective immune response after flu vaccination, which can result in lower levels of vaccine effectiveness (VE) in some older people. Reported VE estimates in people 65 and older have varied. Some years, significant protection has been measured. In other years, little to no protection has been measured. In general, however, when overall VE results are stratified by age, the flu vaccine seems to work less well in people 65 and older.
Obviously, the statement that “some older people … may respond less well to [flu] vaccination” is an understatement. As a group, seniors do respond less well, period. They may not respond at all! Similarly understated is the point that the diminished immune response of the elderly “can result in lower levels of vaccine effectiveness (VE) in some older people.” It does result in lower VE in most older people. And the paragraph ends with the vivid understatement that overall “the flu vaccine seems to work less well in people 65 and older.” The unvarnished truth: It has long been known to work a lot less well in people 65 and older, and this year it seems not to have worked at all on H3N2, the strain of greatest concern (especially to seniors).
What bothers me more is the absence of detail in the claim that “Some years, significant protection has been measured” in the elderly, while “[i]n other years, little or no protection has been measured.” This sounds like it’s probably true, but it is terminally vague. How often does the flu vaccine help seniors, and how often does it fail? Does it work less well in H3N2 seasons than in seasons dominated by a different strain? Is there a downward trend in flu vaccine effectiveness for seniors? Is there evidence that the vaccine works less well in seniors who have been vaccinated in previous years than in those who were just vaccinated for the first time? I don’t know whether the CDC has answers to all these questions, even tentative answers. But speaking as a senior who has complied for years with the CDC’s annual urging to go get my flu shot, “yeah, sometimes it works and sometimes it doesn’t” strikes me as awfully casual.
Predictably, the Q&A’s answer to the question, “Should people 65 and older still get vaccinated?” is yes: “Despite the fact that flu vaccines seem to work less well in people who are 65 and older, there are many reasons why people in that age group should be vaccinated each year.” Four bullet points follow, emphasizing that the elderly are at greater risk from the flu than younger people; that “some protection is better than no protection at all”; that impacts on the severity of flu cases and the number of hospitalizations and deaths may (just may) be significant even when there’s no measurable impact on doctor visits; and that a healthy 68-year-old like me is likely to get a lot more benefit from the flu vaccine than, say, a 90-year-old in bad health.
I have no quarrel with these points. I plan to get my flu shot as usual next year.
The time may come when many experts start saying maybe seniors shouldn’t get an annual flu shot after all. (The time may come when many experts start saying maybe younger people shouldn’t either.) If and when that time comes, I don’t expect the CDC to be in the vanguard of the skeptics. It has a lot of ego and a lot of reputation invested in annual flu shots, and for the foreseeable future it will probably keep recommending annual flu shots and downplaying any evidence to the contrary … even when that evidence results from studies funded by the CDC and published in the CDC’s own Morbidity and Mortality Weekly Report.
So it’s important to keep on the lookout for a flu vaccination paradigm shift. One may be coming, and you probably won’t hear about it first from the CDC.
What saddens me as a risk communication professional is my growing conviction that the CDC will continue to downplay – not cover up, just downplay – news that looks bad for flu vaccination. The January press briefing was a remarkable exception, but now the CDC is up to its old tricks again.
And for no discernible reason. News coverage of the MMWR article and the accompanying Q&A was pretty decent. Some reporters got some details wrong. Some took it for granted that this year was an exception. Most didn’t do much to serve up the tough questions the CDC was ducking, like whether we really know if effectiveness in seniors was higher in the last few H3N2-dominated seasons, and whether there is reason to expect continued little-to-no effectiveness in seniors in years to come. Most missed the fact that the 9% effectiveness figure for seniors vis-à-vis H3N2 was statistically equivalent to zero. But nobody missed the lede: that this year’s flu vaccine offered people 65 and over little if any protection against the flu.
Reporters got the story right. The CDC might as well have been candid about it.
Copyright © 2013 by Peter M. Sandman
