Image credit

Six months ago, I wrote a post called “When journal, scientific society, and community values clash.” I recounted the tale of the Cochrane Collaboration’s controversial exit from updating its review on exercise and ME/CFS. They did this by publishing an editorial note on the review, walking back their previous statement on it being out-of-date; re-issuing the review as a new version despite there being no change to the review itself; and announcing this in the week before Christmas – without any warning to the review’s Independent Advisory Group (which I lead).

In the months since then, they have changed the editorial note twice – wiping the versions from December 2024 and January 2025 from the review’s records. I’ve logged the versions of that editorial note here. The current editorial note (March 2025) is an improvement on the first one – it no longer seems to positively endorse the review – but it still doesn’t make it clear to readers that the review, though presented as a new version dated 2024, is seriously out-of-date. People are definitely being misled. For example, in February, a Wikipedia page described this Cochrane review as “updated in 2024.”

The review was last actually updated with evidence from 2014. When some amendments were made in 2019, Cochrane’s Editor in Chief announced that the review required an update from the ground up. It was, she said, “still based on a research question and a set of methods from 2002, and reflects evidence from studies that applied definitions of ME/CFS [sic] from the 1990s.”

It has only gotten worse since that statement was made in 2019 – and not only because of new data. The review foregrounds the claim that “exercise therapy is recommended by treatment guidelines,” when the guideline it cites no longer does so. This no longer reflects guidelines in countries like the US, UK, and Germany.

In giving this review the status of a 2024 version, Cochrane is remaining active on one side of a major controversy, rather than retiring the review and leaving the field with a neutral posture. This is extremely problematic on a topic where systematic reviews are cherry-picked and weaponized, with claims that reviews unfavorable to one side or the other represent caving in to pressure. The status Cochrane has given this review encourages the perception that it yielded to vested interests rather than staying true to its mission of providing reliable evidence and health information.

The problems this situation causes the Cochrane Collaboration aren’t going away. A quick look at the citations for the review makes it clear why this mischaracterized review is still a current problem.

The mean number of citations of a Cochrane review in the Web of Science in the fifth year after publication is around 12 (Zhu 2022). The exercise and CFS review had over 60 in Google Scholar in 2024. That’s a high rate – even allowing for the higher number of citations in Google Scholar (Martín-Martín 2018). (Details on citations below this post.) Those citations reveal 3 key reasons this can’t be put to rest any other way than Cochrane retiring, updating, or withdrawing the review:

• The review continues to mislead, and potentially harm people with ME/CFS – by telling readers that the therapy is recommended (eg Fricke-Comellas 2024), or citing the review as evidence of effectiveness (eg Flóvenz 2024);

• Continuing criticisms of the review in the literature will whittle away at Cochrane’s reputation – even if they are as mild as designating it “no longer supported” (Broadbent 2025); and

• The controversy is extending, as the literature on Long Covid grows (close to 1 in 5 recent citations).

There are many people who care about the harm this outdated review can do, and won’t let it go – myself included. And it’s only a matter of time till other influential reviews that are misleading by virtue of being out-of-date stir up the same kind of conflict. Cochrane’s response to the complaints about their exit from updating the ME/CFS review show they aren’t well-placed to deal with that – and reveal some other issues of concern. Here are my main 3.

1. Key features that give Cochrane reviews special status have been dismantled.

Cochrane reviews are published in the Cochrane Database of Systematic Reviews. It’s a journal, but not a regular one. It is literally what its name says – a database of reviews – with just one review on any specific topic. That makes it unique. But it’s not that alone that gives it a special status: It’s because they are meant to be reliable by updating in response to new evidence and valid criticism. Decades ago when we started the organization, other things made Cochrane reviews special, like the methods, but they are no longer unique to Cochrane.

Updating is central to what makes a Cochrane review a Cochrane review. Here’s how the Cochrane website currently answers the question, “Why are Cochrane reviews reliable?”:

“Cochrane reviews are internationally recognised as the highest standard in evidence-based health care. We update Cochrane reviews regularly to incorporate new research, so that you can base treatment decisions on the most up-to-date and reliable health evidence. Cochrane reviews are widely trusted due to our research methods and policies.”

Obviously, the ME/CFS review does not come anywhere close to that standard.

I started studying updating of Cochrane reviews with colleagues more than 20 years ago. Our latest study in 2019 concluded, as do others, that in practice, the reviews have been getting more out-of-date over time. But until then, the mechanisms that supported the promotion of being reliable and updated were still in place. If reviews were found to be out-of-date but were not being updated – like the exercise and ME/CFS review – they could be formally withdrawn from the Database. In July 2019, though, this option was ended. There were a few stragglers later that year, but since 2020, no Cochrane review has been withdrawn.

Editorial notes became the way that readers could be alerted to a change in status of a review, including being out-of-date. Now, though, Cochrane is saying this is now against policy, too: We’ll get to that later.

What about responding to criticism? Along with having out-of-date methods and data, that’s a major reason the ME/CFS review needs an update – so many problems have been identified with it. (The IAG’s content analysis identified over 100 points of criticisms that remain unaddressed.)

In 2022, Cochrane centralized editorial responsibility for all Cochrane reviews. The response to criticisms as well as all amendments (Cochrane-speak for corrections), editorial notes, and withdrawals was centralized. (See page 9 here.) Previously, reviews had been the domain of editorial review groups, but now, reviews are edited centrally, and can be submitted directly too. However, in mid-2023, the centralized editorial process for responding to feedback was ended. It is now the authors’ responsibility whether or not they respond to criticisms. Yikes!

In abdicating this editorial responsibility, along with a new policy of not informing readers when they know a review is out-of-date, Cochrane has dismantled a core part of what defined Cochrane reviews. Meanwhile, the reviews are getting progressively more out-of-date. With the loss of funding for so many editorial review groups recently, and the centralization of editing in a small group, the rate of reviews being out-of-date is presumably escalating, while key mechanisms for dealing with outdated or inaccurate reviews have been curtailed.

Meanwhile, the organization’s promotional rhetoric hasn’t been updated to reflect the changed nature of Cochrane reviews. The Cochrane website has had a major update in the last few weeks. They still have the central goal and claim of of providing the “best evidence to help patients and health organizations to make informed decisions.” A new section was added: “Why our evidence is trusted,” with the tagline, “We provide high-quality, unbiased health information.” And there is this claim:

“We are committed to regular reviews of every piece of our evidence. We assess whether findings are still relevant, and we identify aspects to investigate further or update. We believe this is critical to ensure the ongoing reliability and trust in our work as medicine progresses.”

They go on to say, “Governments in all continents of the world regard our evidence so highly that they give their entire populations access to our systematic reviews by funding national provision through the Cochrane Library.” Of course, governments don’t typically do this for medical journals: They do this because of the unique nature the Database used to have. If key pillars of reliability of the Database for users are no longer there, and Cochrane doesn’t take action on high profile unreliable reviews, how long can this special status last?

This is existential for an organization whose current long-term economic strategy involves publicly-funded subscription, especially national ones. I wrote about this earlier this year. Since then, one of only 10 countries outside the UK funding like this has announced it will end soon (New Zealand).

2. Expediency and lack of transparency is compromising the organization and editorial decisions.

The IAG has submitted 3 pieces to Cochrane: An open letter to the Chair and Board in January, a submission to the Editorial Board on policies on outdated and potentially misleading reviews in March, and a comment on the review in March. We received answers from Cochrane to the first couple (posted online here), and the review’s lead author has replied to our comment here.

The author reply is a good example of why leaving response to criticism up to authors isn’t good editorial policy. But it’s the answers from Cochrane related to the update process, declaring reviews out of date, and issuing a new version of an unchanged review that I want to address here.

Their responses center around a pair of claims: The first is a narrative about why the update was abandoned, and the second relates to editorial policies which they say determine their actions. Neither stands up to scrutiny.

Cochrane said that “following the departure of key personnel,” they were unable to maintain support for the updating project, and on that basis, decided to end it. The key personnel was the project officer, who left in July 2022. Late in 2023, I had meetings with Cochrane to discuss how to manage the update in the absence of that support, as well as responding to criticisms of the review. That resulted in the commitment of resources to support me taking on additional roles – we signed a contract for that early in 2024.

One of the first things I did was develop a plan, in consultation with senior staff, of who would do what, and how the update would proceed. That was ready for final sign-off in the middle of 2024. Also in the middle of 2024, the editors responded favorably to the IAG’s proposed editorial note for the review, stating the review was out of date and should not be used. There was no mention of this being against policy.

In fact, both policy and practice show flexibility in the use of editorial notes. The policy provides 5 uses of editorial notes, 4 of which are very specific. The fifth purpose is to add a note “about the status or context of a Cochrane review or protocol.” That’s very broad, and does not preclude saying a review is out of date. I found 59 editorial notes on Cochrane reviews: 27 of them since this policy was introduced. Of those 27, only 17 are for any of the 4 specific reasons. Of the other 10, several relate to the currency or quality of the review. [As yet unpublished data.]

On the other hand, issuing a new citation for a review just because an editorial note has been edited is definitely not covered by policy: Indeed, this is the only time that has been done. That’s because a new citation is constructed to show as an update. See for example the entry for the previous version in PubMed, which automatically flags it is updated in the new version. On Cochrane’s website, it is automatically flagged with, “This is not the most recent version.”

We weren’t the only people complaining, of course. And there was a flash of openness about this in their reply to a complaint by Science For ME:

“This decision does not breach our editorial policy but uses operational discretion in applying the publication guidance regarding editorial notes.” [Cochrane Complaints, March 3, 2025]

Policy is not standing in the way of making it clear to readers that the ME/CFS review is out of date.

3. Claims of high organizational commitment to consumer involvement don’t hold up.

Cochrane points to its commitment to listening to patients and the public as one of the reasons to trust their evidence, and they have a statement of principles about consumer involvement: They say they regard it is as vital. That was evident in establishing the IAG and embarking on the update. However, both my January post and the IAG’s open letter to Cochrane stressed the extremely poor behavior towards the IAG subsequently. From the IAG:

“Our experience, as members of a stakeholder IAG, stands in stark contrast to these statements, indicating a troubling disconnect between Cochrane’s stated commitment to consumer involvement and their actions towards consumers they chose to involve.”

They did not discuss their plans with the IAG, and wiped the record of the IAG and its work from their website without warning too. And they did all this the week before Christmas. Doing this to the people with ME/CFS on the IAG, and the community of people affected by the condition who are concerned about this review, was unconscionable. This is some of what I wrote in January:

“As serial bad news came from Cochrane across that week, the distress and emergency emails and calls and meetings had predictable results for some: They crashed. Even Cochrane’s duty of care to the consumers it appointed to its IAG wasn’t honored. I don’t know how to put my sorrow and anger about this into words.”

I discussed this directly with key senior people at Cochrane after that post as well. One of the most striking things about Cochrane’s responses is what they didn’t do: There’s no apology or concern about the harm they did – and no indication that they are considering the implications for the future.

Of course, a genuine commitment to their stated principles would have been to see the update through, or at least retire the review. But the way they ended it reveals a lot as well. The closest they came to addressing this in their response to the IAG was to say they needed to get the news of their decision out to people. The one thing you wouldn’t do if you really wanted as many people as possible to hear news is to make an announcement at Christmas! And after years of silence to the community, it couldn’t wait a few weeks?

Releasing news around Christmas, is a classic strategy to bury bad news. And that’s not just about media attention – it counts for many people’s inboxes and bandwidth as well. It is a stressful time of year. I think the awfulness of this episode should spur a policy of not releasing major non-urgent news around holidays like Christmas, and the development of a mechanism for serious incident reporting for harm to patients they are involving in their work. Instead, Cochrane’s leadership has demonstrated little respect for their stated principles.

Six months ago, I ended my post with this comment about the organization’s current trajectory: “It needs to change course. The new conflict it has created around the ME/CFS review presents it with a valuable opportunity to do so.” The organization’s leadership have, so far, chosen not to take that opportunity. Given the Cochrane Collaboration’s reliance on their exceptional status and members, that is untenable. Not retiring influential out-of-date reviews is a ticking time bomb.

Absolutely Maybe posts tagged ME/CFS are here, and all posts tagged Cochrane here. I have a “talk page” for the ME/CFS project here. And I recently wrote a rapid response to a BMJ opinion piece on care for people with severe ME/CFS (here).

You can keep up with my work at my newsletter, Living With Evidence. I’m active on Mastodon: @hildabast@mastodon.online, and less so on Bluesky.

~~~~

Disclosures: I am an Emeritus Member of the Cochrane Collaboration. I was a member of the founding group of the organization and founder of its Consumer Network (1993). I was a member of its governing board (then called the Steering Committee) and Coordinator of the Consumer Network until 2001, and the editor of the quarterly Cochrane News from 1994 to 2000. I was the coordinating editor of a Cochrane review group for 7 years, involved in some Cochrane methods groups, and an employee of the Australasian Cochrane Centre. My PhD research was on some factors affecting the reliability of systematic reviews, including a focus on post-publication events (such as retractions and updates) and Cochrane reviews.

I am currently the lead of an Independent Advisory Group for a pair of Cochrane reviews on the HPV vaccine (a network meta-analysis [protocol] and a review on effects on community rates of related disease and vaccination harms [protocol]), with some financial support from 2021 to 2023. I was also appointed the lead of the Independent Advisory Group (IAG) for the review on exercise and ME/CFS in February 2020, with some financial support from 2020 to 2022, resuming in 2024. I am the spokesperson for the IAG. I have a talk page that I began for responding to questions about the Cochrane ME/CFS project. I wrote a rapid response to a BMJ opinion piece on care for people with severe ME/CFS (July 2025).

I have frequently written about the Cochrane Collaboration and Cochrane reviews, both in support (such as a series of 6 posts here on the controversy around a previous HPV vaccine review) and critically, including strong criticism of both of the reviews mentioned in this post (the controversial pandemic-related review, for example here and here, and ME/CFS, for example here and here). I have also participated in criticism of policy in recent years (for example, as a co-signatory of a letter to the BMJ). With this decades-long involvement, I have a considerable track record with many individuals, including staff and authors of Cochrane reviews. My ME/CFS-specific declaration of interests as at November 2023 is here. I keep my financial disclosures up-to-date here.

Back to the top of the article

The cartoon is my own (CC BY-NC-ND license). (More cartoons at Statistically Funny.)

Details on my quick and dirty analysis of citations

I used the citations of the 2019 version of the Larun review on Google Scholar on July 18, 2025. There were 675 citations, with 110 showing from the beginning of 2024, with 68 in 2024. Of the 119, 9 were duplicates, not journals or theses, or citations without links, and 1 of the ones I checked in full text did not actually cite the Cochrane review.

The 100 remaining citations from 2024 and 2025 related to:

• ME/CFS and/or exercise or rest (33);

• Long Covid (19);

• Other conditions (47);

• Methodology (1).