On 25 November a new paper was published by David Tuller (University of California, Berkeley), Caroline Struthers (Oxford University) and one of the authors of this blog. The paper is about exercise therapy for chronic fatigue syndrome and bias due to lack of blinding. This blog post provides a short summary.
The paper was published in the journal Fatigue: Biomedicine, Health & Behavior and is called: “Bias caused by reliance on patient-reported outcome measures in non-blinded randomized trials: an in-depth look at exercise therapy for chronic fatigue syndrome.“
Our main analysis focuses on a discrepancy between several randomized trial that report graded exercise therapy (GET) for chronic fatigue syndrome (CFS) is effective and multiple surveys by patient organizations that indicate just the opposite.
All trials on GET for CFS were not blinded and relied on patient-reported outcome measures. Some (for example Edwards 2017 and Wilshire et al. 2018) have suggested that this trial introduces bias that could distort the main outcomes. Patients who know they are receiving an active intervention rather than a passive control might, for example, be more optimistic about its effect on their health or report symptoms according to what they think will please the investigators etc. We’ve tried to investigate this hypothesis further.
We investigated the size of patient-reported improvements, their consistency over time and congruence with objective measurements. We reviewed mediation analyses, plausible mechanisms of improvement and the risk of expectancy effects.
First, we found that the reported improvements were really small, comparable to the minimally clinically important difference for each of the scales and questionnaires, and similar to bias that has been attributed to blinding in other studies.
Second, we note that the differences between the GET and control group are often no longer statistically significant if assessments are made not directly after treatment, but a couple of months later. (caveat: this can also be due to decreased statistical power).
Third, Vink & Vink-Niese showed in their analysis of the Cochrane review that objective measures indicate little improvements, in contrast to subjective outcomes. This is important because objective outcomes are believed to be more robust to bias due to lack of blinding.
Fourth, mediation analyses suggests that the reported improvements in GET-trials are not the result of an increase in fitness as the deconditioning theory behind this treatment assumed. It is unclear how the reported reductions in fatigue or improvements in physical function came about, given that patients did not significantly increase their activity levels, physical
fitness or employment status. We therefore conclude that exercise therapy in CFS currently lacks a plausible mechanism for improvement.
Lastly, the risk of expectancy effects is large as the treatment manuals of GET show how therapists were instructed to encourage optimism while patients were told to interpret their CFS symptoms as a consequence of deconditioning, stress or anxiety instead of an unknown disease.
Based on the arguments outlined above, we conclude that treatment effects seen in exercise trials for CFS are likely the result of bias associated with a lack of blinding. We acknowledge a level of uncertainty about our hypothesis but argue that the burden of proof lies, as for all interventions, on those claiming efficacy. Those who claim that a treatment is effective should provide evidence that reported improvements are not merely the result of methodological weaknesses. We hope this case study furthers critical assessment of patient-reported improvements in areas of medicine where blinding of therapists and trial participants faces practical limitations.
Further discussion of the paper can be found on the Science for ME forum, an online forum where all scientific (and other) publications on ME/CFS are analyzed and discussed.