In early April 2014, the Cochrane Collective published two journal articles and a news release that went out of their way to understate the value of Tamiflu and Relenza, the two antiviral drugs used against influenza. When Nature reporter Declan Butler asked for our comment, we quickly sent back the short email posted below.
In the days that followed, we drafted this more expansive introductory note to accompany the email we had sent to Declan.
Like previous Cochrane Reviews about flu antiviral drugs and flu vaccines, going back at least to 2006, the one released in April 2014 aggressively overstates its limited findings – findings limited by the very research methods that should make Cochrane Reviews a useful source of information about the subjects they study.
Our risk communication interest in these influenza-related Cochrane Reviews is based on three main factors:
- The mismatch between the apparent professionalism of the reviews’ data analysis and the juvenile way many of the co-authors promote and “misoversimplify” (to put it kindly) their findings.
- The credulous way many journalists report these findings, grossly exaggerating the researchers’ already hyped and misoversimplified explanations.
- Our long-term interest in risk communication about pandemic and seasonal influenza.
The Cochrane Collaboration is a loose-knit consortium of academic medical researchers that publishes periodic “Cochrane Reviews” of the current state of medical knowledge on various topics. These Reviews are frequently referred to as “highly respected.” (Google search “Cochrane Review” + “highly respected” to see what we mean.)
Cochrane Reviews set exacting methodological standards for what studies are reliable enough to include, almost always insisting on randomized controlled trials (RCTs) rather than epidemiological or observational studies. The reviews are thus extremely conservative. If a Cochrane Review says a particular effect has been proved, it’s a good bet that the effect is real. But if a Cochrane Review says an effect hasn’t been proved, that doesn’t mean it isn’t real. There may well be studies – even lots of studies – that suggest the effect is probably real and clinically significant. But if those studies aren’t RCTs, the Cochrane Review will have ignored them.
Cochrane Reviews are invaluable to readers who understand their conservativeness. But they can mislead readers who mistakenly assume that the reviews are distinguishing true claims from false ones, rather than proven claims from not-quite-so-proven ones. Journalists frequently make this mistake when reporting on a newly-published Cochrane Review. We are unaware of any occasion when a Cochrane Review author has complained about this endlessly replicated journalistic mistake: “No, you got it wrong again. We didn’t find that X isn’t true. We merely found that the evidence supporting X isn’t very strong so far.”
The problem is greatly compounded when a Cochrane Review is preceded or accompanied by a news release that actively encourages the mistake by implying that absence of evidence is evidence of absence – implying, in other words, that any claim is a false claim if it is not established by studies meeting Cochrane Collaboration methodological standards. Of course news releases about science necessarily simplify and frequently oversimplify the research they summarize. But some news releases announcing a new Cochrane Review have gone beyond oversimplification to misoversimplification, and even at times to misdirection.
Influenza-related Cochrane Reviews are usually directed by Rome-based epidemiologist Tom Jefferson. Dr. Jefferson naturally shares the institutionalized methodological skepticism of the Cochrane Collaboration: Only randomized controlled trials count as evidence. He surely understands that absence of evidence is not evidence of absence. But he also happens to believe (going beyond his absence of evidence) that vaccines and antiviral medications have relatively little value in preventing or treating flu. His news releases and media interviews often comingle his conservativeness about research methodology with his personal professional opinion about influenza medications; he seems to be claiming that the Cochrane Reviews he has worked on prove the worthlessness of flu vaccines and antivirals.
That’s certainly what a lot of reporters think he is claiming. Here are some headlines from articles covering the new review:
- Ministers blew £650MILLION on useless anti-flu drugs: Cash spent on stockpiling treatments that “worked no better than paracetamol”
- Tamiflu: Major report claims drug is “ineffective”
- Flu Drugs Stockpiled May Do More Harm Than Good, Researchers Say
Especially on the flu vaccine controversy, Dr. Jefferson is the mirror image of the public health establishment. In their zeal to convince people to get vaccinated, many public health agencies routinely exaggerate the efficacy of flu vaccine and the strength of the evidence on its behalf. Dr. Jefferson’s claims about the minimal benefits of flu vaccines and the weakness of the evidence on their behalf are similarly exaggerated. For good reason, anti-vaccination activists often deploy Jefferson quotations to support their contention that flu vaccination is a fraud.
The controversy regarding the flu antiviral drugs Tamiflu and Relenza is more complicated. Dr. Jefferson clearly believes that these drugs, too, have little if any value. And he often appears to believe that the research on flu antiviral efficacy proves that he is right (rather than merely failing to prove to his satisfaction that he is wrong).
The position of the public health establishment on flu antivirals is more nuanced (or perhaps more confused). The official recommendation of the U.S. Centers for Disease Control and Prevention (CDC), for example, is that all patients hospitalized with suspected or confirmed influenza should be started on antivirals right away, and that all outpatients at high risk for influenza complications should be started on antivirals as quickly as possible if they present with suspected or confirmed influenza during periods when flu is circulating. That bottom-line recommendation certainly implies that the CDC thinks antivirals are unambiguously useful for serious influenza cases – despite the paucity of proof from randomized controlled trials. CDC experts (and other experts) responded to the April 2014 Cochrane Review with a ringing defense of antiviral use to treat influenza.
On the other hand, the CDC doesn’t advise healthy people with ordinary flu symptoms to ask their doctor for antivirals; in fact, it doesn’t advise them to go see their doctor at all. And the CDC’s “Influenza Antiviral Medications: Summary for Clinicians” says only that antivirals “may” reduce flu complications and deaths, and shorten hospitalizations.
On April 10, 2014, BMJ (British Medical Journal) published two articles (one on oseltamivir/Tamiflu and one on zanamivir/Relenza), both based on a Cochrane Review on “Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children.” (Neuraminidase inhibitors are the antiviral drugs Tamiflu and Relenza.) The articles were accompanied by an editorial, a news release, and a Cochrane Collaboration summary with links to additional sources. With varying degrees of explicitness, all these documents suggested that antivirals are minimally useful against the flu – thus implying that the substantial government expenditures on antiviral stockpiles for use in a possible flu pandemic were a waste of money.
In a news conference and in interviews with reporters, Dr. Jefferson’s coauthors Carl Heneghan and Peter Doshi explicitly stated that Tamiflu and Relenza are nearly worthless. They also created several straw men. For example, they falsely asserted that one of the main rationales used to justify national and international antiviral stockpiles was to “prevent” a pandemic. (In fact the dominant rationales were to treat sick people, thereby reducing mortality and morbidity, and to protect healthcare and other essential workers.) Then they attacked that straw man. Consider this quotation from Dr. Heneghan in an article headlined “The drugs don’t work: Britain wasted £600m of taxpayers’ money on useless flu pills stockpiled by Government in case of pandemic”:
“There is no credible way that these drugs could prevent a pandemic. You’d have to treat 80 per cent of the population for eight weeks and have a vaccine immediately available even if you assume that it actually does work,” Professor Heneghan said.
In Nature’s first article about the review, Dr. Heneghan said that money for stockpiling “has been thrown down the drain.”
In contrast, the World Health Organization’s rationale for stockpiling antivirals in advance of a flu pandemic rests only to a small degree on a possible Hail Mary campaign to stop a pandemic. WHO does indeed acknowledge the possible use of antiviral stockpiles for “rapid containment” 
if the world is lucky enough to catch a novel influenza virus before it spreads. Rather than oversell the odds that this intervention would work (or even the likelihood of being able to try it), WHO claims only that it would be unconscionable not to be ready to try it, given the potential global catastrophe of a severe pandemic.
WHO pandemic planning focuses far more on using antiviral drugs to reduce mortality, morbidity, and social disruption, and to protect healthcare workers and other priority groups. And national pandemic plans base their antiviral stockpiling decisions almost entirely on the expected value of antivirals during a full-blown pandemic – not on their just-barely-conceivable use to try to stomp out the first outbreak of a pandemic-to-be.
Media coverage of the new Cochrane Review was substantial. Earlier reviews in 2006 and in 2012 on the same topic had relied on published studies only. The new review was based on internal studies wrested from the files of the two drugs’ manufacturers, Roche and GlaxoSmithKline, which gave it a bit of the cachet of an exposé – even though its conclusions were mostly unchanged from the earlier reviews.
As he worked on his coverage of the new study, reporter Declan Butler of Nature sent us an email asking for comment. Below is our response.
All the flaws we found in the Cochrane Review – and especially in the news release promoting the Cochrane Review, which as always was the document that journalists were likeliest to read – point in the same direction: against the value of antivirals to treat flu. Ordinary errors should distribute randomly, with some making antivirals look worse than they deserve, while others make them look better than they deserve. But Jefferson et al. remind us of a cashier who never gives you back too much change, only too little. Such consistency suggests systematic rather than random error – that is, “errors” that may reflect bias and even animus, not just sloppiness.
We will give two examples.
- Ignoring the downsides of accepting only randomized controlled trials.
Cochrane Reviews set very exacting methodological standards for what studies they are willing to consider, ruling out all but randomized controlled trials. But while it is feasible to conduct an RCT of the impact of antivirals on ordinary cases of flu, it is not feasible to do so with regard to the most severe cases. So when you decide to do a Cochrane Review of antiviral effectiveness, you know before you start that you will not be able to draw any conclusions about how well antivirals work on very sick patients.
Jefferson et al. might have overcome this drawback by supplementing their analysis with a précis of the findings of other sorts of studies (albeit methodologically weaker ones) with regard to important questions on which they had no RCT data. At a minimum, they were obliged to emphasize that they had no basis for drawing any conclusions about antiviral effectiveness on very sick patients. Their article makes this sufficiently clear. But their news release says simply that “although the review has confirmed small benefits on symptom relief, there is little to justify any belief that it [antiviral use] reduces hospital admission or the risk of developing confirmed pneumonia.”
This is technically accurate, just as it would be accurate to say they had found little to justify any belief that there is life on Mars. But Jefferson et al. had to know that casual readers (and even some careful ones) would assume that they had looked where the evidence should have been, and that they would have found it if it were there.
Jefferson has been disingenuous in this way before. A news release promoting his 2006 Lancet article (with other coauthors) “Antivirals for influenza in healthy adults” had the headline “Four antivirals not suitable for routine seasonal influenza control.” That release included an aside that received a huge amount of media coverage: “The researchers also found no evidence zanamivir (Relenza) and oseltamivir (Tamiflu) were effective against avian influenza.” When we dug into the actual article, it became clear that there was “no evidence” that the antivirals had helped against H5N1 avian influenza in humans because only a few deathly ill humans in Asia had been lucky enough to be treated with antivirals – and most of those did not get it early in their illness. The media headlines should have said “Tamiflu not well-studied in bird flu patients yet” or “Too soon to say whether Tamiflu helps prevent bird flu deaths,” not (to cite just one example): “Flu drugs ‘will not work’ if pandemic strikes.”
- Massaging and cherry-picking their own results to make them look worse for antivirals.
One of Jefferson et al.’s findings was that antivirals shorten the duration of symptoms of the average adult flu case by 0.7 days (16.8 hours). We’re not as convinced as the authors seem to be that this is a trivial improvement; it’s 10% of a week-long illness. That may have struck them as insufficiently discouraging, so in the release they rounded 0.7 days to “about half a day.”
Even 0.7 days understates the Cochrane Review’s findings about the effectiveness of antivirals against actual influenza. The review (not the news release) states: “We found that both drugs shorten the duration of symptoms of influenza-like illness (unconfirmed influenza or ‘the flu’) by less than a day” in adults. Neuraminidases have no known impact on ILIs other than influenza. So if they cut the duration of ILI symptoms by 0.7 days, they must have reduced the duration of actual influenza symptoms by more than that – perhaps a good deal more.
The press release also omits the finding, reported in the actual article, that symptoms were shortened by 29 hours in healthy non-asthmatic children.
So the Cochrane Review looked at studies of patients with influenza-like illnesses, only some of whom had influenza, and found 16.8 hours in average illness reduction for adults and 29 hours in average illness reduction for children. And the news release “rounded” these findings to an insupportable claim of “about half a day” – a mere 12 hours – in illness reduction for people with influenza.
Another data point in the study omitted from the news release concerned the effectiveness of oseltamivir (Tamiflu), when used prophylactically, in reducing the incidence of symptomatic influenza: an impressive 55% reduction. That is a pretty significant benefit to fail to mention, especially with regard to pandemic preparedness. One of the main rationales for prophylaxis in national pandemic plans was to protect healthcare workers and other essential personnel during a pandemic, so fewer of them would be unable to work when they were desperately needed to cope with the crisis.
Copyright © 2014 by Peter M. Sandman and Jody Lanard
