Drugs that quieted extra heartbeats raised deaths, a lesson in reading a study’s limits

Key takeaways

• In a federal heart trial, 43 people taking drugs that suppressed risky extra heartbeats died of heart rhythm problems, compared with 16 on placebo.
• A form of niacin, vitamin B3, raised “good” HDL cholesterol by about 25% in a large trial but didn’t prevent heart attacks or strokes.
• When 270 researchers repeated 100 published psychology studies, only 36% of the repeats produced statistically significant results.
• A COVID-19 preprint claiming deaths fell by more than 90% was viewed more than 150,000 times before it was withdrawn amid evidence of duplicated and impossible patient records.
• Before trusting a study, check its size, length, outcome, dropouts, participants, subgroups, replication, review status and funding.

In the 1980s, doctors had a reasonable-sounding plan for heart attack survivors. Extra heartbeats after a heart attack were known to signal a higher risk of death². Drugs could suppress those extra beats, so suppressing them should save lives.

The National Heart, Lung, and Blood Institute tested the idea in the Cardiac Arrhythmia Suppression Trial, or CAST²¹⁰. Before volunteers were assigned to a group, they had to show that a drug cut their extra beats by at least 80%². On the heart monitor, the drugs worked.

Then patients were randomly assigned to take the drug or a placebo. After an average of 10 months, 43 people taking encainide or flecainide had died of heart rhythm problems, compared with 16 on placebo¹. Those two drugs were stopped¹². A third drug in a follow-up trial was stopped early for the same reason².

CAST is a classic lesson about one kind of study limitation. Every study has limits, and good authors list them. This guide shows you how to find them and judge which ones matter.

Start with the paragraph authors hope you’ll skip

Most papers include a section on limitations, usually near the end. Read it first, because it is where researchers admit what their study can’t show.

Reporting standards push authors to be open. CONSORT, the main guideline for reporting randomized trials, was updated in April 2025. The new version has 30 essential items and a diagram that tracks every participant from enrollment to analysis³. It added seven items and a new section on open science³.

The 2006 GAIT trial of glucosamine and chondroitin, paid for by the National Institutes of Health (NIH), is a good model of honest limits. Its authors noted that most volunteers had mild pain and that 60.1% on placebo improved, which made a benefit harder to detect⁴. They also said their drug-grade capsules might not match what stores sell⁴.

Far from weakening the study, that list told readers exactly how far to trust it. Our companion article on how clinical trials work walks through GAIT’s design in detail.

Small samples let luck write the headline

Picture a hypothetical: you flip a fair coin 10 times. Getting 7 or more heads happens about one time in six.

Flip it 1,000 times, and 700 heads would be close to impossible. Small numbers bounce around, while big numbers settle down.

Studies work the same way. In a small study, a few unusual people can make a treatment look great or useless. That’s why researchers calculate how many people they need before they start.

Subgroups are small studies hiding inside big ones. In GAIT, the combination of glucosamine and chondroitin seemed to help people with moderate to severe pain: 79.2% improved, compared with 54.3% on placebo⁴. But that group was small, and the authors called the finding exploratory, needing its own trial⁴.

When you see a striking result, ask how many people produced it. “Twice as many improved” means little if it’s 6 people versus 3.

Stand-in measures can point the wrong way

A surrogate endpoint is a stand-in: a lab value, scan or monitor reading that’s easier to measure than the outcome you care about. Blood pressure, cholesterol and heart-monitor readings are common examples. They’re useful when changing them reliably changes how people feel, function or survive.

CAST shows what happens when that link breaks. The drugs fixed the reading and harmed the patients¹².

Niacin offers a lesson closer to the supplement aisle. Niacin is vitamin B3, and at high drug doses it raises HDL, the “good” cholesterol.

The AIM-HIGH trial enrolled 3,414 people with heart disease, low HDL and high triglycerides, all already on a statin. Each was randomly assigned to extended-release niacin or a placebo⁷. HDL rose about 25% in the niacin group, versus about 10% on placebo⁷.

The number moved. The outcomes didn’t. Heart attacks, strokes and related events struck 16.4% of the niacin group and 16.2% of the placebo group, and the trial was stopped early because it showed no sign of benefit⁷.

A larger trial, HPS2-THRIVE, gave 25,673 people niacin combined with another drug or a placebo for about four years⁸. Major heart and blood vessel events were 13.2% versus 13.7%, a difference small enough to be chance⁸. The niacin group had more new diabetes, bleeding and infections, and the maker suspended the combination pill worldwide in 2013⁸.

Many studies, of drugs and supplements alike, report a change in a blood marker. When you see one, ask: has anyone shown that moving this marker helps people live longer or feel better? If you take high-dose niacin, talk with your clinician before changing anything.

Short studies and missing people

Length matters. GAIT ran 24 weeks⁴. A two-year follow-up with 662 volunteers found that no treatment, including the prescription drug celecoxib, produced a clinically important improvement in pain or function over placebo⁵. Short studies can also miss harms that take time to surface.

Dropouts matter, too: in GAIT, 20.5% of volunteers left before the end. Completion ranged from 76.3% in the glucosamine group to 83.6% in the celecoxib group⁴.

Why care? People who leave often differ from those who stay, because they may have felt no benefit or had side effects. If a study analyzes only those who finished, the treatment can look better than it is.

The fix is called intention-to-treat analysis. Everyone is counted in the group they were assigned to, whether or not they finished. It keeps the fairness that random assignment created. Look for the phrase in the methods, and look at the participant flow diagram that CONSORT asks trials to include³.

Who was studied decides whom the results fit

Every trial has entry rules. GAIT enrolled people 40 and older with at least six months of knee pain and arthritis visible on X-rays⁴. Pain at the start was mostly mild, the authors noted⁴. CAST enrolled heart attack survivors with frequent extra beats and weakened heart pumping².

Results apply most clearly to people like those studied. A knee trial doesn’t settle what happens in hips or hands.

For hips, a 2017 orthopedic guideline found glucosamine no better than placebo. For hands, one study of 162 people found chondroitin improved pain and function⁶. Same supplements, different joints, different answers.

The product matters, too. GAIT’s capsules were made to pharmaceutical standards rather than the standards for dietary supplements⁴. A store-bought bottle might differ in form or quality.

When you read a study, find the “participants” or “eligibility” section. Ask: are these people like me in age, sex, health and the problem I have?

Slice the data enough and something will turn up

Test 20 outcomes, and one may look “significant” by luck alone. When researchers keep re-running analyses until something crosses that line, it’s called p-hacking. Slicing results into subgroups works the same way.

The classic demonstration came from ISIS-2, a 1988 trial of aspirin and a clot-dissolving drug in 17,187 people with a suspected heart attack. Its authors split the results by astrological sign to show how easily subgroups mislead⁹¹⁵. Our article on correlation and causation tells that story.

Registering a study’s main outcome in advance makes this harder. Researchers Robert Kaplan and Veronica Irvin reviewed 55 large heart trials funded by the National Heart, Lung, and Blood Institute. Before 2000, 57% reported a benefit on their main outcome. Afterward, when every trial had registered its outcomes publicly, just 8% did¹⁰.

The authors weighed other explanations. A shift away from placebo comparisons didn’t fit the data, though better everyday heart care might explain part of the drop, and they said they couldn’t prove registration was the cause¹⁰. They also argued that a null result from a big, well-run trial still teaches science something¹⁰.

Even so, 12 of those 25 later trials reported benefits on other measures¹⁰. A headline built on a secondary measure is weaker than one built on the main question.

Replication is the real test

One study is a clue, while a finding that holds up when others repeat it is evidence.

In 2015, a team of 270 researchers called the Open Science Collaboration repeated 100 studies published in three psychology journals¹¹. Of the originals, 97% reported statistically significant results. Of the repeats, 36% did, and the average effect was about half as large¹¹.

Lab research on cancer shows a similar pattern. The Reproducibility Project: Cancer Biology picked 193 experiments from 53 high-profile cancer papers to redo, and in the end completed 50 experiments from 23 papers¹².

The hurdles were telling. Not one original paper explained its methods fully enough to copy without help from its authors. Roughly two in three repeats had to adjust the recipe, and about a third of the original teams offered no real help at all¹². When an original study had reported an effect, the repeat’s effect averaged 85% smaller¹².

A failed repeat doesn’t prove the first study wrong, since methods and conditions differ. But it should lower your confidence until the finding is confirmed.

Preprints and peer review are checks, not guarantees

A preprint is a paper posted online before peer review, when outside experts check it for a journal. The preprint site Research Square, for example, warns on its pages that a preprint has not been peer reviewed by a journal¹⁴.

Preprints speed up science, but they can mislead. A preprint led by a researcher at Benha University in Egypt claimed that ivermectin, an anti-parasite drug, cut COVID-19 deaths by more than 90% in a study of 400 people¹³. It was viewed more than 150,000 times and cited more than 30 times¹³.

Then outside researchers found copied phrases, duplicated patient records and patients who had died before the study began. The preprint site withdrew it on July 14, 2021¹³. Our article on systematic reviews explains how the study had already seeped into summary analyses.

Peer review helps, but it’s not a guarantee either. The replication projects above tested studies that had already passed peer review¹¹¹². Treat both preprints and published papers as steps toward an answer.

A 10-point check for any study

Use these questions on the next study you read, whether it’s about a drug, a diet or an herb.

#	Ask	Why it matters	Warning sign
1	How many people?	Small samples swing by chance	Dozens of people, big claims
2	How long?	Benefits fade and harms emerge over time	Weeks of data for a lifelong problem
3	What was measured?	Stand-in markers can mislead	A lab value, not symptoms or survival
4	Was there a comparison group?	Symptoms improve on their own	No placebo or control group
5	How many dropped out, and were they counted?	Leavers often differ	No intention-to-treat analysis
6	Who was studied?	Results fit people like those enrolled	Different ages, health or conditions than yours
7	Was the main outcome set in advance?	Prevents picking winners afterward	No registry number
8	Is the headline from a subgroup or side result?	Slicing creates chance findings	“Especially in women over 60”
9	Has anyone repeated it?	Many findings shrink on repeat	One study, never confirmed
10	Peer reviewed, and who paid?	Checks and conflicts	A preprint or undisclosed funding

What the evidence shows

Claim	Evidence rating	Basis
Improving a stand-in marker reliably improves health	Evidence says no, or unsafe	CAST drugs raised deaths; niacin raised HDL without fewer events178
A single exciting study is usually enough to trust a result	Evidence says no, or unsafe	Large replication projects found many results shrank or vanished1112
Registering outcomes in advance curbs overly positive results	Promising but early	One review of 55 trials found a sharp drop in positive results10
Glucosamine plus chondroitin helps moderate to severe knee pain	Promising but early	Exploratory subgroup result, not confirmed in a dedicated trial4

Questions readers ask

The bottom line

No study is perfect, and that’s fine. The question is whether a study’s limits are small enough to trust its main finding, and whether the finding has been repeated. Stand-in measures, small samples, missing participants, chance subgroups and unreviewed papers have all led smart people astray.

The 10-point check works for drug trials and supplement studies alike. Use it every time, and you’ll be harder to fool by a headline from any side.

Related reading

• How clinical trials decide, when most treatments tested in people fail
• Who paid for the study matters, even when the science is real
• Most health products aren’t scams, but many just haven’t been proven
• Chocolate tracks with Nobel prizes, and other lessons in telling a link from a cause
• How systematic reviews work, and why cranberries failed one and passed the next

Sources

1. Heart Rhythm Society. “The CAST Trial” (summary of the CAST final report, New England Journal of Medicine, 1991;324:781-8). HRS. hrsonline.org
2. National Heart, Lung, and Blood Institute. “Cardiac Arrhythmia Suppression Trial (CAST).” BioLINCC, NIH. biolincc.nhlbi.nih.gov
3. Hopewell S, Chan AW, Collins GS, et al. “CONSORT 2025 Statement: Updated Guideline for Reporting Randomised Trials.” The BMJ, 2025;389:e081123 (published April 14, 2025). bmj.com
4. Clegg DO, Reda DJ, Harris CL, et al. “Glucosamine, Chondroitin Sulfate, and the Two in Combination for Painful Knee Osteoarthritis.” New England Journal of Medicine, 2006;354:795-808. Full text accessed via farm.ucl.ac.be
5. Sawitzke AD, Shi H, Finco MF, et al. “Clinical Efficacy and Safety of Glucosamine, Chondroitin Sulphate, Their Combination, Celecoxib or Placebo Taken to Treat Osteoarthritis of the Knee: 2-Year Results From GAIT.” Annals of the Rheumatic Diseases, 2010;69(8):1459. pubmed.ncbi.nlm.nih.gov
6. National Center for Complementary and Integrative Health. “Glucosamine and Chondroitin for Osteoarthritis: What You Need To Know.” NIH, 2023. nccih.nih.gov
7. NEJM Journal Watch CardioExchange. “AIM-HIGH: No Benefit for Niacin on Top of Statins.” November 15, 2011. blogs.nejm.org
8. American College of Cardiology. “Niacin Fails to Reduce Heart Attacks and Strokes in HPS2-THRIVE” (meeting release). ScienceDaily, March 11, 2013. sciencedaily.com
9. ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. “Randomised Trial of Intravenous Streptokinase, Oral Aspirin, Both, or Neither Among 17,187 Cases of Suspected Acute Myocardial Infarction: ISIS-2.” The Lancet, August 13, 1988. statmodeling.stat.columbia.edu
10. Kaplan RM, Irvin VL. “Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time.” PLoS ONE, 2015;10(8):e0132382. journals.plos.org
11. Open Science Collaboration. “Estimating the Reproducibility of Psychological Science.” Science, 2015;349:aac4716, as summarized in University of Florida Research Integrity training materials, 2021. research.ufl.edu
12. Center for Open Science. “Reproducibility Project: Cancer Biology Reveals Challenges and Opportunities.” December 7, 2021. cos.io
13. Reardon S. “Flawed Ivermectin Preprint Highlights Challenges of COVID Drug Studies.” Nature, 2021;596:173-174. media.nature.com
14. Hill A, Mirchandani M, Pilkington V. “Ivermectin for COVID-19: Addressing Potential Bias and Medical Fraud.” Research Square preprint, 2021 (later published in Open Forum Infectious Diseases, 2022). researchsquare.com
15. Cates C. “The Perils and Pitfalls of Subgroup Analysis.” Testing Treatments (EBM website). en.testingtreatments.org