Why the Cochrane review on exercise therapy for chronic fatigue syndrome is still misleading

On Wednesday, October 2, Cochrane published a long-awaited amendment to its review of exercise therapy for chronic fatigue syndrome (CFS) [1] following a formal complaint to Cochrane’s Editor in Chief. Unfortunately, the published amendment does not address the main flaws of the review and continues to overestimate the evidence for exercise therapy in CFS.

In this blog post, I will argue that treatment effects found in the review are (1) small and lower than some estimates of the minimal clinically important difference (2) no longer statistically significant at follow-up and (3) contradicted by objective measurements. I will argue that there is currently no plausible mechanism for the effectiveness of exercise therapy in CFS and that the treatment effects found are better explained as bias due to a lack of blinding. Finally, I will explain how patient surveys suggest that some CFS patients deteriorate following exercise therapy.

The post-treatment differences are really small

The Cochrane review includes 8 randomized trials. I will focus my analysis on the main comparison of exercise therapy versus a passive control condition (treatment as usual or relaxation therapy).

For fatigue, the primary outcome of the review, the post-treatment effect size corresponds to a 3.4 point difference on the 33-point Chalder Fatigue Scale. This is close to the reported minimal clinically important difference (MCID) for this scale. [2-4] Ridsdale and colleagues, for example, used an MCID of 4 points, explaining that “our consensus view was that a difference of less than four, using a Likert scale, is not important.” [5]

The same is true for the post-treatment difference for physical function measured by the Short Form 36 Health Survey Questionnaire (score 0-100). Here the results are obscured by an extreme outlier that reported a remarkable 30-point difference. If this trial is excluded, the mean post-treatment difference between the exercise and passive control group is only 7.37 points, below most estimates of the MCID for this scale. [6-7]

The same principle applies to other outcomes for which lesser quality data is available. Despite the comparison with a passive control condition, exercise therapy only produces minor differences in reported health; differences so small that their clinical relevance could be reasonably questioned.

‘Follow-up’:  no longer statistically significant

At follow-up differences between the exercise and control group become even cloudier. Whether one looks at fatigue, physical function, depression, anxiety or self-perceived changes in overall health, the difference between the exercise and control group is no longer statistically significant. The only exception is sleep, an outcome for which only three trials reported data.

It seems that twelve weeks of building up exercises under the guidance of a trained healthcare professional doesn’t make much difference in the long run. Patients, who received treatment as usual or listened to a relaxation tape, had a similar outcome. The term ‘follow-up’ might be somewhat confusing because the three largest trials providing most of the data in this review defined their primary outcome at this time point.

Objective outcomes: not reported

A major flaw of the Cochrane review is that it did not report on objective outcomes (the sole exception is service use). Given that none of the trials were blinded, one would expect that reviewers focus on objective outcomes as these are less influenced by the hopes and expectations of trial participants. The largest study to date on bias in randomized trials, the BRANDO project, gave the following recommendation:

“Our results suggest that, as far as possible, clinical and policy decisions should not be based on trials in which blinding is not feasible and outcome measures are subjectively assessed. Therefore, trials in which blinding is not feasible should focus as far as possible on objectively measured outcomes, and should aim to blind outcome assessors.” [8]

Unfortunately, the authors of the Cochrane review did just the opposite: they focused on the subjective outcomes and ignored the objective outcomes. The 8 randomized trials had data on employment [9], disability benefits [10], activity levels [11] and fitness tests [12-13] that were not presented in the Cochrane review. These showed no significant difference between the exercise group and the passive control group. [14]

No plausible mechanism

The lack of improvement on objective measures of fitness is puzzling given that the rationale for exercise therapy was to recondition CFS patients. [15] Nonetheless, the results are clear and consistent: the four trials [12-13, 16-17] that conducted a mediation analysis all found that self-reported improvements in fatigue or physical function are not mediated by objective measures of fitness. In CFS, exercise therapy does not work by increasing physical fitness, which is contrary to the offered treatment rationale. [15] This means that exercise therapy currently lacks a plausible mechanism for improvement. 

A lack of blinding

There is one mechanism that could easily explain small improvements on patient-reported outcomes that disappear at follow up and are not supported by objective measurements. It’s called bias due to a lack of blinding. If patients know that they are getting the intervention they might be more positive about their current health status (a placebo-effect) or they might report symptoms according to what they think will please the investigators (response bias). That’s why drug trials usually blind both patients and therapists so that their expectations do not influence the results. In the exercise trials, however, blinding was not possible and this could have distorted the outcomes. A 2014 review by Hrobjartsson et al. on trials that compared blinded and non-blinded groups, found that the average difference in effect size for patient-reported outcomes was -0.56. [18] That’s very similar to the effect sizes reported in the Cochrane review.

There are reasons to think that bias was particularly high in the exercise trials. For one, exercise therapy for CFS was already recommended by healthcare institutions when the largest trials were being conducted. [19-20] During the largest trial, the infamous PACE trial, patients were sent a newsletter that highlighted how the new NICE guideline had recommended exercise therapy. [21]

A second reason is that treatment manuals consisted of assertive encouragement to raise patients’ expectations. One patient booklet told trial participants:  “You will experience a snowballing effect as increasing fitness leads to increasing confidence in your ability. You will have conquered CFS by your own effort and you will be back in control of your body again.” [22] An online description of the exercise therapy used in the FINE trial was even more assertive: “Focus on your achievements now. Symptoms and limitations are temporary” and “There is no disease. Go for 100% recovery.”[23] The manual for therapists in the PACE trial said about patients “[…] it is important that you encourage optimism about the progress that they may make with this approach. You can explain the previous positive research findings of GET and show in the way you discuss goals and use language that you believe they can get better.” [24] Perhaps one shouldn’t be surprised that, after receiving such instructions and pep talk, patients say they do a little better.  

‘MitCure’: an analogy

A thought-experiment might help clarify things. Imagine an unblinded randomized trial for patients with CFS called ‘MitCure’. MitCure is just a sugar pill, but the researchers conducting the trial tell participants that it’s a drug that helps to restore mitochondrial function. Patients are given booklets that explain in detail how mitochondrial dysfunction could account for all their CFS symptoms and how MitCure could make them back in control of their own life. In the first sessions with their therapist, patients are told that MitCure has already shown promising results and that it will help them to become fitter and healthier. Patients are told that it’s important that they take the right amount of MitCure and that they should gradually build up their dose. Activity diaries are filled in to monitor progress. Over a period of 12-15 weeks, patients get weekly sessions with a healthcare professional who emphasizes their achievements and reminds them of how important it is to take Mitcure in order to keep making improvements. Patients in the control group are only given medical care, the same care that patients in the intervention are receiving in addition to MitCure. Many are on a waiting list to receive MitCure as well. Now, what would happen if you ask both groups to fill in symptom questionnaires directly after the treatment ended? Wouldn’t you suspect the intervention group to report a small improvement in fatigue compared to the controls, say … a 3.4 point difference on the Chalder Fatigue Scale?

The oxford criteria: an outdated case definition

A further problem with the Cochrane review is the case definition used to select patients. Approximately 85 percent of patients in the randomized trials were selected using the Oxford criteria for CFS. [25] These criteria date back to 1991, focus solely on the symptom of fatigue and are considered outdated. According to a 2016 document issued by the Agency for Health Research and Quality (AHRQ), the Oxford case definition is the least specific of the definitions and less generalizable to the broader population of patients with CFS. [26] The AHRQ document advised that “future research should retire the use of the Oxford (Sharpe, 1991) case definition.” [26] A CFS working group for the NIH wrote that “continuing to use the Oxford definition may impair progress and cause harm.” [27]

Post-exertional malaise

In the amendment, the authors of the Cochrane review tried to account for this problem by adding a brief statement that “patients diagnosed using other criteria may experience different effects.” In my view, this is a meaningless statement that could apply anywhere. What the authors should have said is that the diagnostic criteria used in their review do not require patients to experience ‘post-exertional malaise’ or a marked worsening of symptoms following exertion, while all other and more recent criteria do. This is particularly relevant in a review that tries to assess the efficacy and safety of exercise therapy.

Both the 2011 International Consensus Criteria (ICC) [28] and the 2015 case definition proposed by the National Academy of Medicine [29] consider a marked worsening of symptoms following exertion and not fatigue to be the hallmark symptom of the illness. Consequently, these criteria propose a name that no longer refers to fatigue such as systemic exertion intolerance disease (SEID) or myalgic encephalomyelitis (ME). Studies show that ME/CFS patients experience prolonged debility after physical exertion, usually a cardiopulmonary exercise test. [30] Researchers now use these exercise tests to temporary provoke post-exertional malaise in the hope of finding a clue into the underlying pathology of the illness. [31]

Patient surveys indicate harm

There is evidence that suggests some ME/CFS patients respond differently to exercise therapy than the patients included in the randomized trials. In multiple patient surveys, approximately 50% of respondents indicate to have deteriorated following exercise therapy. [32-33] As these surveys were not randomized controlled trials, it is difficult to determine whether deterioration was due to exercise therapy or some other factor such as the fluctuating course of the illness. There are however reasons to take the patient surveys seriously.

For one, there have been more than a dozen of these surveys covering thousands of patients from multiple countries and spanning a period of almost 30 years. [32-33] Many of these surveys did not solicit a negative assessment of exercise therapy. In a 2010 survey by the British ME Association, for example, members were simply asked how useful or harmful they found several CFS treatments. [34] These included hydrocortisone, sleep medication, antidepressants and allergy medication, treatments known to have significant side-effects. Yet, it was exercise therapy that received by far the most negative responses. This suggests that the negative responses towards exercise therapy cannot fully be explained by selection bias. And contrary to what is sometimes claimed, there is no evidence to suggest that the reported harms of exercise therapy are due to inappropriate delivery of the treatment. In a 2008 survey by Action for ME there was little difference in the reported rate of harms by exercise therapy whether the treatment was delivered by an NHS specialist (31%), the GP (45%) or others (29%). [35] In a 2015 survey of the ME Association, graded exercise therapy courses “held by therapists stated to have an ME/CFS specialism made symptoms worse for 57% of respondents.” [36]

Unfortunately, the Cochrane review does not mention the patient surveys indicating harm by exercise therapy. It only refers to previous reviews that support the conclusion that exercise therapy is a safe and effective treatment. The review does admit that the 8 randomized trials provide little data on possible harms of exercise therapy and that “we are uncertain about the risk of serious adverse reactions because the certainty of the evidence is very low.” [1]

An influential review

Despite its many shortcomings, the Cochrane review on exercise therapy for CFS has been quite influential. National guidelines in Denmark [37], Germany [38], Australia [39], and Belgium [40-41] referred to the Cochrane review in recommending exercise therapy for CFS patients. In the US, Mayo Clinic uses the Cochrane review to justify the following inappropriate advice to CFS patients:

“Gradually increasing the intensity of your exercise over time may help reduce your hypersensitivity to exercise, just like allergy shots gradually reduce a person’s hypersensitivity to a particular allergen.” [42]

In the UK, the Cochrane review has been used to deflect criticism of the PACE trial by the Health Research Authority [43], the Medical Research Council [44] and the current Minister for Care [45], all claiming that Cochrane has independently validated the PACE trial’s findings. In Australia, there is an ongoing randomized trial, testing how to educate healthcare professionals about the “level 1 evidence” for graded exercise therapy for patients with CFS. The intervention consists of an educational program “based on a manual developed by the research group that drew on Cochrane reviews of […] graded exercise therapy (GET) interventions for CFS.” [46] Overall, it seems that the Cochrane logo is hindering a critical assessment of the evidence.

Corrections as soon as possible

In October 2018, almost exactly a year ago, 42 experts in the field of ME/CFS signed an open letter asking Cochrane to address the shortcomings in its review on exercise therapy for CFS. [47] The recent amendment leaves the main flaws unresolved. Given the influence of the review, the poor evidence base for its conclusions and the potential harm it could cause I would advise Cochrane to issue a full update as soon as possible.

References

[1] Larun L, Brurberg KG, Odgaard-Jensen J, Price JR. Exercise therapy for chronic fatigue syndrome. Cochrane Database Syst Rev. Version published: 02 October 2019. https://doi.org/10.1002/14651858.CD003200.pub8

[2] Pouchot J, Kherani RB, Brant R, Lacaille D, Lehman AJ, Ensworth S, et al. Determination of the minimal clinically important difference for seven fatigue measures in rheumatoid arthritis. J Clin Epidemiol. 2008 Jul;61(7):705-13.

[3] Goligher EC, Pouchot J, Brant R, Kherani RB, Aviña-Zubieta JA, Lacaille D, et al. Minimal clinically important difference for 7 measures of fatigue in patients with systemic lupus erythematosus. J Rheumatol. 2008 Apr;35(4):635-42. Epub 2008 Mar 1.

[4] Nøstdahl T, Bernklev T, Fredheim OM, Paddison JS, Raeder J. Defining the cut-off point of clinically significant postoperative fatigue in three common fatigue scales. Qual Life Res. 2019 Apr;28(4):991-1003.

[5] Ridsdale L, Godfrey E, Seed P. Chronic Fatigue in general practice: authors reply. Br J Gen Pract 2001, 51:317–318.

[6] Brigden A, Parslow RM, Gaunt D, Collin SM, Jones A, Crawley E. Defining the minimally clinically important difference of the SF-36 physical function subscale for paediatric CFS/ME: triangulation using three different methods. Health Qual Life Outcomes. 2018 Oct 19;16(1):202.

[7] Wyrwich KW, Tierney WM, Babu AN, Kroenke K, Wolinsky FD. A comparison of clinically important differences in health-related quality of life for patients with chronic lung disease, asthma, or heart disease. Health Serv Res. 2005 Apr;40(2):577-91.

[8] Savović J, Jones H, Altman D, Harris R, Jűni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol Assess. 2012 Sep;16(35):1-82.

[9] Jason LA, Torres-Harding S, Friedberg F, Corradi K, Njoku MG, Donalek J. Non-pharmacologic Interventions for CFS: A Randomized Trial. J Clin Psychol Med Settings. 2007, 14:275–296.

[10] McCrone P, Sharpe M, Chalder T, Knapp M, Johnson AL, Goldsmith KA, et al. Adaptive pacing, cognitive behaviour therapy, graded exercise, and specialist medical care for chronic fatigue syndrome: a cost-effectiveness analysis. PLoS One. 2012;7(8):e40808.

[11] Wallman KE, Morton AR, Goodman C, Grove R, Guilfoyle AM. Randomised controlled trial of graded exercise in chronic fatigue syndrome. Med J Aust. 2004 May 3;180(9):444-8.

[12] Wearden AJ, Emsley R. Mediators of the effects on fatigue of pragmatic rehabilitation for chronic fatigue syndrome. J Consult Clin Psychol. 2013 Oct;81(5):831-8.

[13] Chalder T, Goldsmith KA, White PD, Sharpe M, Pickles AR. Rehabilitative therapies for chronic fatigue syndrome: a secondary mediation analysis of the PACE trial. Lancet Psychiatry. 2015 Feb;2(2):141-52.

[14] Vink M, Vink-Niese A. Graded exercise therapy for myalgic encephalomyelitis/chronic fatigue syndrome is not effective and unsafe. Re-analysis of a Cochrane review. Health Psychol Open. 2018 Oct 8;5(2):2055102918805187.

[15] Clark LV, White PD. The role of deconditioning and therapeutic exercise in chronic fatigue syndrome (CFS). J Ment Health. June 2005;14(3):237 – 252.

[16] Fulcher KY, White PD. Randomised controlled trial of graded exercise in patients with the chronic fatigue syndrome. BMJ. 1997 Jun 7;314(7095):1647-52.

[17] Moss-Morris R, Sharon C, Tobin R, Baldi JC. A randomized controlled graded exercise trial for chronic fatigue syndrome: outcomes and mechanisms of change. J Health Psychol. 2005 Mar;10(2):245-59.

[18] Hróbjartsson A, Emanuelsson F, Skou Thomsen AS, Hilden J, Brorson S. Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies. Int J Epidemiol. 2014 Aug;43(4):1272-83.

[19] Department of Health (2002) A report of the CFS/ME working group: report to the Chief Medical Officer of an independent working group. London: Department of Health. Available at: https://www.meassociation.org.uk/wp-content/uploads/CMO-Report-2002.pdf

[20] Chronic fatigue syndrome Clinical practice guidelines – 2002. Produced by a Working Group convened under the auspices of the Royal Australasian College of Physicians. Available at: https://www.mja.com.au/system/files/issues/cfs2_2.pdf

[21] Tuller D. TRIAL BY ERROR: The Troubling Case of the PACE Chronic Fatigue Syndrome Study. Virology Blog. October 2015. http://www.virology.ws/2015/10/21/trial-by-error-i/  

The relevant newsletter is available here: https://www.me-pedia.org/images/a/a0/Participants%27_newsletter_Issue_3_December_2008_PACE_trial.pdf

[22] Powell, P (2005) FINE Trial. Patient booklet version. 9 (29/04/05) Chronic Fatigue Treatment Programme. Liverpool: Infectious Diseases Unit, Royal Liverpool University Hospital.

[23] Tuller D. Trial By Error, Continued: Why has the PACE Study’s “Sister Trial” been “Disappeared” and Forgotten? Virology Blog. November 2015. http://www.virology.ws/2015/11/09/trial-by-error-continued-why-has-the-pace-studys-sister-trial-been-disappeared-and-forgotten/

[24] Bavinton et al. 2004. Manual for Therapists Graded Exercise Therapy (GET) for CFS/ME. Final trial version: version 7 (MREC Version 2). https://me-pedia.org/images/8/89/PACE-get-therapist-manual.pdf

[25] Sharpe MC, Archard LC, Banatvala JE, Borysiewicz LK, Clare AW, David A, et al. A report–chronic fatigue syndrome: guidelines for research. J R Soc Med. 1991 Feb;84(2):118-21.

[26] Smith MEB, Nelson HD, Haney E, Pappas M, Daeges M, Wasson N, et al. Diagnosis and treatment of myalgic encephalomyelitis/chronic fatigue syndrome. Evidence Reports/Technology Assessments, No. 219. Addendum July 2016. Rockville (MD): Agency for Healthcare Research and Quality (US). https://www.ncbi.nlm.nih.gov/books/NBK293931/pdf/Bookshelf_NBK293931.pdf

[27] Green CR, Cowan P, Elk R, O’Neil KM, Rasmussen AL. National Institutes of Health Pathways to Prevention Workshop: Advancing the Research on Myalgic Encephalomyelitis/Chronic Fatigue Syndrome. Ann Intern Med. 2015 Jun 16;162(12):860-5.

[28] Carruthers BM, van de Sande MI, De Meirleir KL, Klimas NG, Broderick G, Mitchell T, et al. Myalgic encephalomyelitis: International Consensus Criteria. J Intern Med. 2011 Oct;270(4):327-38.

[29] Beyond myalgic encephalomyelitis/chronic fatigue syndrome: redefining an illness. Washington, D.C: The National Academies Press; 2015. https://www.ncbi.nlm.nih.gov/books/NBK274235/pdf/Bookshelf_NBK274235.pdf

[30] VanNess JM, Stevens SR, Bateman L, Stiles TL, Snell CR. Postexertional malaise in women with chronic fatigue syndrome. J Womens Health (Larchmt). 2010 Feb;19(2):239-44.

[31] Analysis of Post-exertional Malaise Using a Two-day CPET in People With ME/CFS. ClinicalTrials.gov Identifier: NCT04026425. https://clinicaltrials.gov/ct2/show/NCT04026425

[32] Kindlon T. Reporting of Harms Associated with Graded Exercise Therapy and Cognitive Behavioural Therapy in Myalgic Encephalomyelitis/Chronic Fatigue Syndrome. Bulletin of the IACFS/ME. 2011;19(2): 59-111. http://iacfsme.org/PDFS/Reporting-of-Harms-Associated-with-GET-and-CBT-in.aspx

[33] Geraghty K, Hann M1, Kurtev S. Myalgic encephalomyelitis/chronic fatigue syndrome patients’ reports of symptom changes following cognitive behavioural therapy, graded exercise therapy and pacing treatments: Analysis of a primary survey compared with secondary surveys. J Health Psychol. 2019 Sep;24(10):1318-1333.

[34] Managing my M.E. – What people with ME/CFS and their carers want from the UK’s health and social services. Gawcott, England: ME Association; May 2010. Available at: http://www.meassociation.org.uk/wp-content/uploads/2010/09/2010-survey-report-lo-res10.pdf  

[35] Action for ME and Association of Young People with ME. ME 2008: What progress? 2008 May. Available at: https://ssb4mesupport.weebly.com/uploads/8/0/5/0/8050248/action_for_me_survey_2008.pdf

[36] ME Association. ME/CFS illness management survey results: No decisions about me without me. Patient Survey. May 2015. Available at: https://www.meassociation.org.uk/wp-content/uploads/2015-ME-Association-Illness-Management-Report-No-decisions-about-me-without-me-30.05.15.pdf

[37] Hansen R. Denmark has some recommendations for ME… and they’re pretty awful. ME Action. July 2018. https://www.meaction.net/2018/07/09/denmark-has-some-new-recommendations-for-me-and-its-pretty-awful/

 [38] Antwort der Bundesregierung auf die Kleine Anfrage der Abgeordneten Maria Klein-Schmeink, Dr. Kirsten Kappert-Gonther, Kordula Schulz-Asche, weiterer Abgeordneter und der Fraktion BÜNDNIS 90/DIE GRÜNEN – Drucksache 19/12204 – Myalgische Enzephalomyelitis/Chronic Fatigue Syndrome – Aktuelle Situation in Versorgung und Forschung. http://dip21.bundestag.de/dip21/btd/19/126/1912632.pdf

[39] Graded exercise therapy: chronic fatigue syndrome. The Royal Australian College of General Practitioners (RACGP). March 2015. https://www.racgp.org.au/clinical-resources/clinical-guidelines/handbook-of-non-drug-interventions-(handi)/other/graded-exercise-therapy-chronic-fatigue-syndrome

[40] CVS-patiënten verwerpen ‘klinische gids’ RIZIV (Update). ME-gids. July 2018. https://www.me-gids.net/module-ME_CVS_docs-viewpub-tid-1-pid-1982.html?utm_source=dlvr.it&utm_medium=twitter

[41] Fysieke training voor de behandeling van het chronisch vermoeidheidssyndroom? Minerva. 2016. http://www.minerva-ebm.be/NL/Article/2071

[42] Tuller D. Trial By Error: Mayo Still Champions GET. Virology Blog. August 2018. http://www.virology.ws/2018/08/06/trial-by-error-mayo-still-champions-get/

[43] Re: The PACE trial and the Committee’s inquiry on Research Integrity. Health Research Authority. January 2019. https://www.parliament.uk/documents/commons-committees/science-technology/Correspondence/190129-Sir-Jonathan-Montgomery-Health-Research-Authority-to-Chair-re-PACE-trial.pdf

[44] Criticism of the PACE trial. Medical Resourch Council. August 2018. https://mrc.ukri.org/news/browse/criticism-of-the-pace-trial/

[45] House of commons Hansard. Westminster Hall. PACE Trial: People with ME. Volume 636. February 2018. https://hansard.parliament.uk/commons/2018-02-20/debates/990746C7-9010-4566-940D-249F5026FF73/PACETrialPeopleWithME

[46] Li SH, Sandler CX, Casson SM, Cassar J, Bogg T, Lloyd AR, et al. Randomised controlled trial of online continuing education for health professionals to improve the management of chronic fatigue syndrome: a study protocol. BMJ Open. 2017 May 10;7(5):e014133.

[47] A Statement in Support of Cochrane. Virology Blog. October 2018. http://www.virology.ws/2018/10/23/a-statement-in-support-of-cochrane/