How systematic reviews work, and why cranberries failed one and passed the next

Key takeaways

• A systematic review collects all the studies on a clear question using methods set in advance, and a meta-analysis combines their numbers into one estimate⁴⁵.
• Cochrane’s 2023 cranberry review pooled 50 trials with 8,857 people and found cranberry products probably cut urinary infections in women who get them often and in children, and may help after treatments such as pelvic radiation, but showed no clear benefit in pregnancy¹.
• Every good review grades its own evidence as high, moderate, low or very low certainty, and that rating matters as much as the headline result⁸.
• Reviews can magnify bad data: one withdrawn ivermectin study supplied 15.5% of the effect in a published meta-analysis⁹.
• In a BMJ comparison, seven industry-supported drug reviews recommended the drug without reservations, while none of the matched Cochrane reviews did, so always check who paid¹³.

In 2012, Cochrane, an international group known for rigorous health reviews, looked at every trial it could find on cranberries and urinary tract infections (UTIs). Its verdict, as one industry trade magazine summed it up, was that cranberry products did not work to prevent them³.

Eleven years later, the same review was updated. It now held 50 trials and 8,857 people. This time the answer changed: cranberry products probably lower the risk of infection in women who get UTIs often and in children, and may help people whose medical treatment raises their risk¹.

Nobody cheated, and nobody flip-flopped on a whim. Twenty-six new trials arrived, and the review was built to absorb them¹.

That’s the promise of a systematic review: all the evidence on a question, weighed by rules set in advance. But a review can also be outdated, sloppy, padded with bad data or paid for by someone with a product to sell.

This guide shows how reviews are built, how to read the chart at their heart, and how to check one in about five minutes.

One study is a single witness, and a systematic review calls them all

Any single study has quirks: too few people, an unusual group, or plain luck. Results bounce around, so one study alone can mislead.

A systematic review tries to gather every study on a clearly stated question, using explicit methods that someone else could repeat⁴. That’s different from an ordinary review article, where an expert may cite the studies they know or like best.

Cochrane, an international research network named for Archie Cochrane, lays out the steps⁵¹⁴:

• Publish the plan first. A protocol states the question, which studies count, which outcomes matter and how the analysis will run. It’s a public record, so authors can’t quietly move the goalposts later.
• Search widely, including for unpublished studies.
• Screen and judge each study against the preset rules, noting its strengths and flaws.
• Combine and summarize. When studies are similar enough to pool, statistics merge them into a single estimate. That pooling step is the meta-analysis. Not every review includes one.
• Update as new studies appear, because new evidence can change the answer⁵.

A reporting rulebook called PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) tells authors what to disclose. Its 2020 version has 27 items, including the full search strategy, how certain the evidence is, who has competing interests and whether the data are shared⁴.

The Cochrane logo is a chart that saved babies

Cochrane’s logo looks like an abstract design. It is a forest plot, the standard chart in a meta-analysis⁶.

The treatment behind it is a steroid shot for women likely to give birth too soon, which helps the baby’s lungs mature. Trials existed, but many obstetricians held back⁶. Pooling those trials, first by Crowley and colleagues and later in updates, made the case hard to ignore. Cochrane credits the shots with probably saving thousands of premature infants⁶.

The plot in the logo shows what the trials available in 1982 would have looked like when pooled⁷. A 2020 update added a large World Health Organization trial and rated the evidence high certainty: the shots reduce newborn deaths⁷. Fiona Stewart, the update’s lead author, said that a modern version of the logo would need far more lines, but the diamond would not move⁷.

Here is how to read any forest plot.

Part of the chart	What it shows	What to look for
Each horizontal line	One study’s result; its length shows the range of uncertainty	Short lines mean more precise studies
Square on each line	That study’s best estimate; bigger squares carry more weight	Is one study driving everything?
Vertical line	“No difference” between the groups	A study whose line crosses it is inconclusive on its own
Diamond at the bottom	The pooled estimate; its width is the uncertainty	Does the whole diamond sit on one side?
Labels under the axis	Which side favors the treatment	Read them; benefit isn’t always on the left

A certainty rating tells you how far to trust the answer

Cochrane and many other groups grade their findings with a system called GRADE. It sorts the evidence for each outcome into four levels⁸:

• High: reviewers are very confident the true effect is close to the estimate.
• Moderate: the true effect is probably close, but it could be substantially different.
• Low: the true effect may be substantially different.
• Very low: the true effect is likely substantially different.

Randomized trials, where chance decides who gets the treatment, start at high; other designs start lower. Reviewers then mark the rating down for five problems⁸:

• Risk of bias: flaws in how the studies were run.
• Imprecision: too few people or events, so the range of uncertainty is wide.
• Inconsistency: the studies disagree with each other.
• Indirectness: the studies tested different people, doses or outcomes than the question asks about.
• Publication bias: signs that unfavorable studies never got published.

A large effect or a clear dose-response pattern can move evidence from other designs up a level⁸. The certainty grade may be the most useful line in a review. “Cranberry reduces infections” and “cranberry reduces infections, low certainty” are different statements.

Reading a real review, step by step: cranberries

Here’s the 2023 cranberry review, read the way a careful researcher would¹.

1. The question. Do cranberry juice, tablets or capsules prevent UTIs compared with a placebo (a dummy product), no treatment or other options? The search ran through March 2023¹.

2. What’s inside. There are 50 randomized or nearly randomized trials with 8,857 people. Of those, 45 compared cranberry with a placebo or nothing¹.

3. The pooled answer. Across 26 studies with 6,211 people, cranberry users had 30% fewer UTIs that caused symptoms and were confirmed by a lab test¹. Reviewers call this a risk ratio of 0.70. It’s the risk in the cranberry group divided by the risk in the comparison group.

4. The absolute numbers. Percentages can sound bigger than they are. Among women with recurrent UTIs, about 243 of every 1,000 in the comparison groups got an infection during the trials, versus about 180 per 1,000 taking cranberry¹. That’s roughly six fewer infections for every 100 women.

5. Who it helps. The benefit showed up in women with recurrent UTIs, in children, where risk was roughly halved, and in people whose risk came from medical treatment, such as radiation to the pelvic area, gynecological surgery or a kidney transplant¹. It did not show up in pregnant women, older adults living in care facilities, or adults whose bladders don’t empty fully¹.

6. The certainty. Moderate for women and for children, and low for the group at risk from medical treatment¹. Small mismatches happen even in good reviews: this one’s abstract and its summary table list slightly different numbers for children (risk ratios of 0.46 and 0.53), though both mean roughly half the risk¹. The women’s result was marked down because trials disagreed. Overall, a measure of how much results varied beyond chance, called I-squared, was substantial at 69%¹.

7. The fine print. Dropout rates ranged from zero to 55%. The authors suspected many people tired of the juice, though most trials never recorded why¹. Of the 50 trials, 23 had some company involvement, five of them with Ocean Spray, and in 12 a company donated the product¹. Few people reported side effects; the most common was stomach pain. Compared with antibiotics, cranberry made little or no difference in two trials¹.

What does this mean for you? The federal research center on complementary health, NCCIH, says cranberry products may lower the risk of recurrent UTIs in women by about 25%, but it calls the findings inconsistent².

Cranberry is not a treatment for an infection you already have. If you have UTI symptoms, see a clinician². Evidence on whether cranberry interacts with the blood thinner warfarin conflicts, so ask before using it regularly if you take that or any other medicine².

Garbage in, garbage out: the ivermectin lesson

A meta-analysis works like a blender. If a spoiled ingredient goes in, it flavors the whole batch.

During the COVID-19 pandemic, a preprint, meaning a study posted online before peer review, reported that the drug ivermectin cut deaths by more than 90% in a trial of 400 people⁹. The paper drew well over 150,000 views and more than 30 citations, and several meta-analyses counted it⁹.

Then outside researchers examined the data. They found duplicated patient records, patients who seemed to have died before the study began, and copied text. Research Square, the preprint site, pulled the paper on July 14, 2021, citing ethical concerns⁹. In one meta-analysis, in the American Journal of Therapeutics, that one study made up 15.5% of the pooled effect⁹.

A second team, writing in Open Forum Infectious Diseases, retracted its meta-analysis after one of its largest studies was withdrawn over fraudulent data. The analysis now posted leaves out studies at high risk of bias and finds no statistically significant effect on survival¹⁰.

The lesson is not that reviews are useless. Good reviews judge each study’s risk of bias, and many rerun the math without the shakiest studies. That rerun is called a sensitivity analysis. If a benefit vanishes when weak studies leave, the benefit lived in the weak studies.

There are too many reviews, and some come with an agenda

Reviews became prestigious, and then they multiplied. Researcher John Ioannidis found that from 1991 to 2014, yearly publication of systematic reviews rose 2,728% and meta-analyses rose 2,635%. Biomedical papers overall rose 153%¹¹. He estimated that more systematic reviews of trials were probably being published each year than new randomized trials¹¹.

Some topics are crowded with near-copies. From 2007 to 2014, 185 meta-analyses on depression treatment appeared¹¹. Over seven years, 21 meta-analyses looked at statins for an irregular heartbeat after heart surgery¹².

Two other researchers counted 682 new systematic reviews indexed in a single month of 2014, about 22 a day. Basics often went missing: for each of several items, including a protocol, risk-of-bias methods and the funding source, at least a third of the reviews failed to report it¹².

Money shows up here, too. Ioannidis reported that antidepressant meta-analyses were often written by drug company employees or authors tied to industry, with results that lined up with the sponsors’ interests¹¹.

A 2006 BMJ study compared 24 pairs of reviews on the same drugs. Seven industry-supported reviews recommended the drug without reservations. None of the matched Cochrane reviews did¹³. The effect estimates were similar on average; the conclusions, and the candor about weaknesses, were not¹³.

The same scrutiny applies to supplements. Nearly half the trials in the cranberry review had company involvement¹. That doesn’t make them wrong, but it’s why independent reviewers matter.

Cochrane says it does not accept funding that could influence its reviews¹⁴. Our guide to sponsored research covers how funding shapes studies in more depth.

Reviews serve natural-health values, including skepticism of drug hype

Many natural-health readers say they want proof that isn’t bought, and they don’t want one flashy study setting the agenda. A good review runs on the same instincts: look at everything, show your work, declare your interests and change your mind when the evidence changes.

Reviews puncture drug hype, too. In April 2026, Cochrane announced a review of 17 trials with 20,342 people on drugs that clear amyloid, a protein that builds up in the brain in Alzheimer’s disease. According to Cochrane’s news release, their effects on memory and thinking were absent or trivial, below the smallest change patients would notice¹⁵. The drugs likely raised the risk of brain swelling and bleeding, mostly seen on scans in people without symptoms¹⁵.

The lead author argued that earlier trials’ statistically significant results never added up to a difference patients could feel¹⁵. That is one review’s judgment, and other experts may read the trials differently, so anyone weighing these drugs should talk with a specialist.

Reviews also reward patience. Cranberry looked like a failure in 2012 and a modest success in 2023¹³. A remedy can be unproven today and supported tomorrow, or the reverse. The honest answer is the one that fits the evidence available now.

Read a review in five minutes

Use this routine the next time a website, ad or friend says “a meta-analysis proves it.”

1. Find the search date. A review whose search ended years ago misses everything since.
2. Match the question to you. Same kind of person, same product and dose, same outcome you care about?
3. Jump to the summary-of-findings table. Write down the effect and the certainty grade for your outcome.
4. Convert to real numbers. Look for “per 1,000” columns and ask how many fewer people out of 100 had the problem.
5. Check the diamond and the spread. Does the pooled range cross “no difference”? Do the studies point the same way?
6. Check what’s inside. How many studies are at low risk of bias? Was any big study withdrawn? Is there a sensitivity analysis?
7. Check the plan and the money. Was a protocol registered first? Who funded the review and the trials? Do authors declare ties?
8. Look for harms. A review that reports only benefits is telling half the story.

If you find several reviews on the same question, favor the newest independent one with a published protocol and a clear certainty rating.

What the reviews in this article found

Claim	What the review found	Our rating
Steroid shots before an early birth reduce newborn deaths	High certainty, Cochrane 20207	Strong evidence
Cranberry products prevent repeat UTIs in women who get them often	Moderate certainty; trials disagree, and NCCIH calls findings inconsistent12	Promising but early
Cranberry products prevent UTIs in children	Moderate certainty from a few small trials1	Promising but early
Cranberry helps in pregnancy, in care-facility residents or with bladder-emptying problems	No clear benefit; mostly low certainty, and the review’s pages differ on the care-facility grade1	Not enough evidence
Cranberry treats an active UTI	Not recommended as a treatment2	Not enough evidence; see a clinician
Ivermectin cuts COVID-19 deaths	No significant effect once studies at high risk of bias were left out910	Evidence says no, or unsafe
Amyloid-clearing drugs give a noticeable benefit in early Alzheimer’s	Effects absent or trivial, with brain swelling and bleeding risk, per Cochrane’s 2026 release15	Evidence says no, or unsafe

Questions readers ask

The bottom line

A systematic review is the closest thing medicine has to a fair trial of all the evidence. When it’s done well, with a published plan, a wide search, honest certainty grades and independent funding, it beats any single study, any testimonial and any sales pitch.

But a review inherits the flaws of what goes into it. Read the certainty grade, turn percentages into real numbers, check the studies inside and follow the money, whether it leads to a drug company or a supplement brand.

Keep exploring: animal studies vs human studies, scientific uncertainty in health, study limitations, cherry-picked studies and how to read a research abstract.

Related reading

• Who paid for the study matters, even when the science is real
• Most health products aren’t scams, but many just haven’t been proven
• Traditional medicine got WHO codes, and testing it fairly is the harder part
• Only 1 in 20 treatments tested in animals wins approval for people
• What to ask when your doctor recommends a treatment

Sources

1. Williams G, Hahn D, Stephens JH, Craig JC, Hodson EM. “Cranberries for preventing urinary tract infections.” Cochrane Database of Systematic Reviews, 2023. pmc.ncbi.nlm.nih.gov
2. National Center for Complementary and Integrative Health. “Cranberry.” NIH, updated November 2024. nccih.nih.gov
3. NutraIngredients. “Comprehensive review backs cranberry products for urinary tract infections.” May 4, 2023. nutraingredients.com
4. Page MJ, McKenzie JE, Bossuyt PM, et al. “The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.” BMJ, 2021. pmc.ncbi.nlm.nih.gov
5. Cochrane Library. “About Cochrane Reviews.” cochranelibrary.com
6. Cochrane Switzerland. “The Cochrane logo.” swiss.cochrane.org
7. Cochrane. “Featured review: Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth.” December 10, 2020. cochrane.org
8. Cochrane Training. “Assessing the certainty of evidence” (GRADE training materials). January 2021. resources.cochrane.org
9. Nature News. “Flawed preprint highlights challenges of COVID drug studies.” Nature, August 12, 2021. media.nature.com
10. Hill A, et al. “Meta-analysis of Randomized Trials of Ivermectin to Treat SARS-CoV-2 Infection” (retracted; the authors published a corrected analysis separately). Open Forum Infectious Diseases, 2021. pubmed.ncbi.nlm.nih.gov
11. Ioannidis JPA. “The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses.” The Milbank Quarterly, 2016. milbank.org
12. Page MJ, Moher D. “Mass Production of Systematic Reviews and Meta-analyses: An Exercise in Mega-silliness?” The Milbank Quarterly, 2016. pmc.ncbi.nlm.nih.gov
13. Jørgensen AW, Hilden J, Gøtzsche PC. “Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review.” BMJ, 2006. pubmed.ncbi.nlm.nih.gov
14. Cochrane. “About us.” cochrane.org
15. Cochrane. “Anti-amyloid Alzheimer’s drugs show no clinically meaningful effect.” April 16, 2026. cochrane.org