Yesterday, the Guardian’s Science Weekly podcast published an update of its episode titled: “What role should the public play in science?” [1] For those who might have missed it, the episode revolved around a Cochrane review on exercise therapy for chronic fatigue syndrome (CFS). [2] A Reuter’s news article had announced that a withdrawal of the review was imminent following a formal complaint to Cochrane’s editor in chief. [3]
The Science Weekly podcast gave the impression that the decision to withdraw the Cochrane review was due to pressure from activists. Unfortunately, none of the critics of the Cochrane review were interviewed on the program. Instead, two authors of the controversial PACE trial were allowed to present their view without counterargument. They said the news about Cochrane, if true, was “very alarming” and an example of evidence being revised “not by a change in science but by campaigns.” The update of the podcast seems to admit that this narrative was incorrect. In the added introduction, host Ian Sample, explains that “the details of the complaint have been made public and they show that it was based in science.”
In their defense, the Science Weekly team seems to argue that they could not have known this at the time. The update explains:
“…in the podcast we talk about a paper on CFS interventions that appeared in the prestigious Cochrane database of systematic reviews. It was later temporarily removed after a complaint from the public. This was our starting point when looking at the role of public pressure when it comes to science and when we made the podcast we didn’t know the exact nature of the complaints or who had made them. We asked, but we weren’t told.”
This argument is problematic for several reasons (for one, the Cochrane review was never removed, not even temporarily).
Let’s go back to when the podcast was aired, 2 November 2018, and see what was known at the time. The issues raised in the formal complaint were already mentioned in the comment section of the Cochrane review by the same person: the late Robert Courtney. Similar arguments were raised by Tom Kindlon. Their comments were published in the same document [3] as the Cochrane review, so any journalist who read the paper in full would have been aware of them. On 21 October 2018, renowned skeptic and health psychologist James Coyne published a blog post about Robert Courtney and the importance of his critique on the exercise review for CFS. [4] It included the cover letter and documents Courtney had used in his formal complaint to Cochrane. [5] In addition, detailed analysis and criticism of the Cochrane review were published by Vink & Vink-Niese in Health Psychology Open on 8 October 2018. [6] In other words, there was sufficient information to understand that valid criticism of the Cochrane review was being made.
It might be true that the Science Weekly podcast could not have known the exact nature or origin of the complaint Cochrane was responding to. But the issues raised in the complaint were already in the public domain. So it should have been evident to any investigative journalist that complaints about the review were based on sound scientific arguments. One could argue that uncertainty about the nature and origin of the complaint to Cochrane made it difficult to reach strong conclusions about the revision process. But in that case, one shouldn’t make a podcast accusing Cochrane and the ME/CFS patient community of damaging the integrity of science, either.
There is very little evidence that does support the narrative highlighted in the podcast. The main source for Cochrane giving in to a campaign of activists instead of scientific arguments is a Reuter’s news article that appeared on 17 October 2018. [3] The article suggested that Cochrane would temporarily withdraw the review due to pressure and complaints from online activists. Yet, when David Tovey, Cochrane’s editor in chief at the time, was contacted he stressed that “this is not about patient pressure.” [3] So why did the Guardian decide to do a podcast on this news story as an example of patient pressure overruling science?
The formal statement Cochrane gave to the Science Weekly podcast was as follows:
“The Cochrane review of exercise therapy for chronic fatigue syndrome was last updated in 2014 and is subject to an ongoing process of review and revision following the submission of a formal complaint to the Editor in Chief. Cochrane carefully considers all feedback and complaints and revises or updates reviews when appropriate to do so. The review author team have advised us that a resubmission of this review is imminent. A decision on the status of this review will be made once this resubmission has been through editorial process, which we anticipate will be towards the end of November 2018.”
Notice there is no mention of withdrawal or pressure from campaigners. Instead, the statement speaks of a formal complaint, an ongoing review process, and an imminent resubmission. Despite the update, it remains unclear what information prompted the Science Weekly podcast to promote a narrative that conflicts with what Cochrane told them.
It is equally unclear why it took 9 months to issue this update. Cochrane already announced on 30 November 2018 that the review was being revised because of the complaint submitted by Robert Courtney (they mention him by name). [7] Journalist and Senior Fellow at Berkeley, David Tuller, had written about the details of Courtney’s complaint and confirmation of its validity by Cochrane on 14 March 2019. [8] Why did it take the science weekly podcast many more months to issue an update?
When a mistake is made, the right thing to do is to correct it. I doubt that many listeners of the podcast will check to see if an episode that was aired 9 months ago, has any updates or corrections. A more reasonable option would be to invite those who have criticized the Cochrane review on a new episode of the podcast so that they can explain to regular listeners that their complaints were indeed based on science.
References
[1] What role should the public play in science? – Science Weekly podcast. The Guardian, 7 August 2019. https://www.theguardian.com/science/audio/2018/nov/02/what-role-should-the-public-play-in-science-science-weekly-podcast
[2] Larun L, Brurberg KG, Odgaard-Jensen J, Price JR. Exercise therapy for chronic fatigue syndrome. Cochrane Database Syst Rev. 2017 Apr 25;4:CD003200. https://doi.org/10.1002/14651858.CD003200.pub7
[3] Kelland K. Exclusive: Science journal to withdraw chronic fatigue review amid patient activist complaints. Reuters. October 17, 2018. https://www.reuters.com/article/us-health-chronicfatigue-dispute/exclusive-science-journal-to-withdraw-chronic-fatigue-review-amid-patient-activist-complaints-idUSKCN1MR2PI
[4] Coyne C. How to get a flawed systematic review and meta-analysis withdrawn from publication: a detailed example. Mind the Brain. 21 October 2018. https://web.archive.org/web/20190503184207/https://webcache.googleusercontent.com/search?q=cache:Qwoqlcv7C_MJ:https://mindthebrain.blog/2018/10/21/how-to-get-a-flawed-systematic-review-and-meta-analysis-withdrawn-from-publication-a-detailed-example/+&cd=1&hl=en&ct=clnk&gl=uk&client=firefox-b-d
[5] ME/CFS notes. Comments Submitted to the Cochrane Review of Exercise Therapy for Chronic Fatigue Syndrome. https://sites.google.com/site/mecfsnotes/
[9] Wooden bookcases, wall cabinets, and library cabinets. In solid wood and made-to-measure.
[6] Vink M, Vink-Niese A. Graded exercise therapy for myalgic encephalomyelitis/chronic fatigue syndrome is not effective and unsafe. Re-analysis of a Cochrane review. Health Psychol Open. 2018 Oct 8;5(2):2055102918805187. https://doi.org/10.1177/2055102918805187
[7] Cochrane Database of Systematic Reviews. Exercise therapy for chronic fatigue syndrome, see what’s new. https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub7/information#whatsNew
[8] Tuller D. Trial By Error: Cochrane’s Report on Courtney’s Complaint. Virology Blog. 12 March 2019. http://www.virology.ws/2019/03/12/trial-by-error-cochranes-report-on-courtneys-complaint/