How we handle retracted research

What a retraction is

A retraction is a journal withdrawing a paper it has already published. The journal prints a retraction notice that says why: an honest error that changes the results, data that can't be verified, copied or duplicated work, or misconduct such as made-up data. The paper usually stays online, marked as retracted, so the record stays complete.

PubMed, the index of medical research that our pages link to, marks a retracted paper as retracted and links to its notice. If you open a study we cite and PubMed says it has been retracted, we want to know (see the end of this post).

What an integrity concern is

Sometimes a journal has serious doubts about a paper but isn't ready to retract it. It can then publish an expression of concern: a notice that questions have been raised about the paper's data or how the work was done, often while an investigation runs. An expression of concern doesn't say the paper is wrong. It tells readers not to rely on it until the questions are answered.

We use "integrity concern" for both of these, and for one more case: a researcher with many retracted or flagged papers.

Why some authors' papers never support a grade

A small number of researchers have had many papers retracted or flagged, often after an investigation found problems that ran through their work. Retractions take time, sometimes years. When one author already has many, the papers still standing were usually produced in the same way, and some have simply not been examined yet.

So our rules treat authors with multiple retractions or expressions of concern, listed on our internal integrity list, as a group whose papers never support a grade or the direction of an effect on Winnowleaf, whether or not a particular paper has been retracted. Many researchers share a name, so we match a paper to an author by affiliation and co-authors as well as by name. A paper by someone else with the same name is not affected.

This is a rule about evidence, not a judgement of anyone's character, and we don't name people on the blog. In the corrections log, a fix made under this rule says the study was removed because it comes from the integrity list.

Why a meta-analysis that leans on a retracted trial can't support a claim

A meta-analysis pools the results of several trials into one estimate. It is often the strongest evidence there is, which is exactly why a bad trial inside one matters. If a retracted trial carries much of the pooled result, the pooled number carries the problem too. And a meta-analysis published before the retraction is rarely corrected afterwards.

So we look inside. For each result we use, we check which trials the meta-analysis pooled. If one is retracted, or comes from an author on our integrity list, we ask whether that result depends on it. The review's own analysis with that trial left out, or a later review that doesn't include it, can show that it doesn't. If the result holds without it, we can still use the review, and the reason says so. If it can't be shown to hold, the review doesn't support that result on our pages, and the finding is set again from the evidence that is left.

We judge this one result at a time. A large review can include one problem trial and still be sound for an outcome that much larger trials decide. Dropping it entirely would throw away good evidence, and using it without checking would let a retracted trial back in.

You can see this in our guides. The weight-loss guide names a widely shared vinegar trial that was later retracted, and says that a review which pooled it leaves any benefit unproven. It also notes that the best-known green coffee study was retracted.

What you see on our pages

When a re-check finds that a finding rested on a retracted or flagged paper:

• the paper is taken off the finding, and the result, size and grade are set again from the studies that are left (How we grade has the full checklist);
• if no study in people is left, the finding loses its grade;
• while we fix a finding, it shows "Under review" and no letter, and it is never featured as a top finding or a pick.

On our supplement and condition pages, papers by authors on the integrity list are also left out of each finding's list of studies and its study count, and a finding whose only studies were theirs shows "Under review".

Not every finding has been re-checked yet. One without a "Reviewed" label may still carry a letter that was first set with such a paper among its studies; its re-check sets the letter again without it. Until then, read its letter as a starting point.

How to check our corrections log

Every fix is listed in the corrections log, newest first: what the finding said, what it says now, and why. To find the fixes that involve retracted or flagged papers:

1. Open the corrections log and choose a week.
2. Choose the range of supplement names you want.
3. Use your browser's find (Ctrl+F, or Command+F on a Mac) to search for "retracted", "expression of concern" or "integrity".

Each entry also says who found the problem: our own evidence review, or an independent audit.

If you spot one we missed

Papers keep being retracted, and one can be retracted after we cite it. If you see a study on our pages that has been retracted or carries an expression of concern, email hello@winnowleaf.com with the page address and the study. Our corrections policy says what happens next.