Why observational studies got smoking right and hormone therapy wrong

Key takeaways

• In 2002, a federal trial of 16,608 women found that estrogen plus progestin raised heart disease risk, after observational studies had suggested a 40% to 50% drop.
• When researchers reanalyzed the Nurses’ Health Study as if it were a trial, most of the apparent heart protection disappeared, which shows that how data are analyzed matters.
• In a heart trial reported in 1980, men who faithfully took their placebo died far less often than men who skipped doses, a warning that people who stick with pills differ from people who don’t.
• No one was ever assigned to smoke, yet watching people proved that smoking causes lung cancer, because smokers’ risk was nine to ten times higher and rose with the dose.
• A 2024 Cochrane review found that trials and observational studies usually reach similar answers, so the useful skill is spotting when they are likely to disagree.

On July 9, 2002, the National Heart, Lung, and Blood Institute made a dramatic call. It stopped a major women’s health trial early². More than 16,600 women had been taking either estrogen plus progestin or a placebo, a dummy pill².

The surprise was the direction of the result. For years, studies that followed women who chose hormones had suggested a 40% to 50% lower risk of heart disease¹. The trial found the opposite: 29% more coronary heart disease among women on hormones, plus more strokes, lung clots and invasive breast cancers¹.

Now rewind to 1950. Two British researchers, Richard Doll and Austin Bradford Hill, published their search for the cause of rising lung cancer deaths. Doll suspected car exhaust or tar from newly surfaced roads¹⁰. Their data pointed to cigarettes instead¹⁰.

Nobody was ever assigned to smoke for science. That research also came from watching what people did, and it was right.

So why did watching people get smoking right and hormones wrong? The answer explains many of the health headlines you’ll read this year, including the ones about supplements, diets and drinks.

Two ways to learn: watch people, or let chance decide

In an observational study, researchers record what people already do and what happens to them. They don’t assign anything. There are three common kinds.

• Cohort study: follows a group over time, recording habits first and illnesses later. The Nurses’ Health Study, which has tracked tens of thousands of nurses, is a famous example⁵.
• Case-control study: starts with people who already have a disease (the cases) and compares their past with similar people who don’t (the controls). Doll and Hill’s first study worked this way, with patients from 20 London hospitals¹⁰.
• Cross-sectional study: a snapshot that measures habits and health at one moment. It can show that two things travel together, but not which came first.

In a randomized controlled trial, researchers assign people by chance to a treatment or a comparison, often a placebo. Chance is the key ingredient. With enough people, a coin flip spreads every trait roughly evenly between the groups: age, income, diet, optimism, even traits nobody thought to measure.

That’s the one thing observational studies can’t buy. When people choose their own habits, the choosers differ from everyone else in countless ways. Researchers can adjust for the differences they measure. They can’t adjust for the ones they miss.

The hormone pill that looked protective until a trial tested it

Hormone therapy after menopause looked like a textbook case for prevention. In the Nurses’ Health Study, women currently taking estrogen plus progestin had about a third less coronary heart disease than women who had never used hormones³. Researchers expressed this as a hazard ratio, a measure of risk over time, of 0.68³.

The Women’s Health Initiative, a research program the National Institutes of Health launched in 1991, put the idea to a fair test². Its estrogen-plus-progestin trial enrolled 16,608 women aged 50 to 79 who still had a uterus¹. After an average of 5.2 years, here is how the hormone group compared with the placebo group¹:

Outcome	Hormones vs. placebo, per 10,000 women per year
Coronary heart disease events	7 more
Strokes	8 more
Blood clots in the lungs	8 more
Invasive breast cancers	8 more
Colorectal cancers	6 fewer
Hip fractures	5 fewer

What went wrong in the earlier studies? In 2008, a team led by epidemiologist Miguel Hernán reran the nurses’ own data as if it were a trial³. They compared women who newly started hormones with similar women who didn’t, and they counted outcomes from the moment of that choice, the way a trial does.

The protection mostly vanished. The new hazard ratio was 0.96, close to no effect³. In the first two years after starting, risk appeared higher, at 1.42, echoing the trial’s early figure of 1.68³. The authors traced most of the leftover gap to two things: how many years had passed since menopause when women started, and how long each study followed them³.

This is the subtle lesson. Comparing current users with never-users can leave out the risky early years for women who started hormones before the study began. Analyzed like a trial, the observational data told a story much closer to the trial’s.

The story kept moving. In November 2025, the FDA asked makers to drop boxed-warning language about heart disease, breast cancer and probable dementia, noting that WHI participants averaged 63, while bothersome hot flashes usually begin around 45 to 55⁴. The agency says the prescribing information has since been updated, and warnings about heart disease and breast cancer remain in the labeling of systemic products, the kind that act throughout the body⁴. If you use or are considering hormones, decide with a clinician who knows your history.

Vitamin E and beta-carotene followed the same script

The hormone reversal wasn’t a one-off. In a cohort of about 90,000 nurses, heart disease was 30% to 40% lower among women with the highest vitamin E intake, mostly from supplements, NIH’s Office of Dietary Supplements reports⁵.

Then the HOPE trial gave nearly 10,000 high-risk people natural vitamin E or a placebo for about four and a half years. The vitamin group had no fewer cardiovascular events⁵. Beta-carotene pills have a similar history, and our article on natural versus synthetic nutrients tells both stories in full.

How common is this pattern? In 2005, John Ioannidis examined 49 of the most-cited clinical studies published in major journals from 1990 to 2003⁶. Five of the six highly cited nonrandomized studies were later contradicted or found to have overstated their effects. Only 9 of 39 randomized trials met the same fate⁶.

Why people who take the pill are different people

One of the strangest results in medicine comes from the Coronary Drug Project, a trial of the cholesterol drug clofibrate in men who had survived heart attacks⁷⁸. Researchers checked how deaths related to how faithfully men took their capsules.

Among men on the drug, 15.0% of good adherers died within five years, compared with 24.6% of poor adherers⁷. That looks like proof the drug works.

But in the placebo group, the split was nearly identical: 15.1% versus 28.3%⁷. The placebo capsules contained no drug. The difference came from the men.

Researchers call this the healthy adherer effect, one form of healthy-user bias. A similar gap appeared in TORCH, a much later trial in chronic lung disease. Patients who took most of their study medicine survived better whether it was real or placebo, even after accounting for how sick they were⁸.

MeiLan Han, a lung specialist at the University of Michigan, suggested that faithful pill-taking may travel with other healthy habits, from eating well to taking other medicines as prescribed⁸.

Now picture an observational study of a daily supplement, hormone or health habit. The people who choose it and stick with it may resemble the men who thrived on placebo. That’s one reason links between pills and good health so often fade when a trial tests them.

Smoking: the case that watching people won

If observational research is so easily fooled, how did it convict cigarettes?

Doll and Hill followed their hospital study with a cohort. They wrote to about 60,000 British doctors, and about 40,000 replied¹⁰. By about the two-and-a-half-year mark, the lung cancer deaths were already clustering among cigarette smokers, and most of all among the heaviest smokers. Early results appeared in 1954¹⁰.

On January 11, 1964, a 10-member committee advising the U.S. Surgeon General released its report after reviewing more than 7,000 scientific articles⁹. Average smokers had a nine- to tenfold risk of lung cancer, and heavy smokers at least a twentyfold risk⁹. Risk rose with years of smoking and fell after quitting, and laboratory scientists confirmed the statistical link⁹.

Those features made the case. A healthy-user effect can plausibly produce a 30% difference. It’s very hard to imagine one producing a tenfold difference that climbs with every pack, drops when people quit and lines up with lab evidence.

Our companion article on correlation versus causation covers the checklist scientists use to judge evidence like this.

When a trial is impossible, unethical or asks the wrong question

Some questions can’t be settled by random assignment. No ethics board would approve assigning teenagers to smoke for 30 years. Trials that last decades are rare and costly, so most evidence on lifelong eating patterns comes from cohorts. Rare side effects may surface only after a drug reaches millions of people.

Trials also study particular people. In 2018, a team of researchers published a deadpan trial in The BMJ¹¹. Of 92 airplane passengers screened, 23 agreed to be randomly assigned to jump from an aircraft wearing either a parachute or an empty backpack¹¹. Nobody in either group died or was seriously hurt¹¹.

The catch: the volunteers’ aircraft sat on the ground, at an average altitude of 0.6 meters, about 2 feet, moving at zero kilometers per hour¹¹. The joke carries a serious point. A flawless trial in the wrong people answers the wrong question. The WHI faced a milder form of this critique, since its participants were older than most women who start hormones for hot flashes⁴.

Nature’s coin flip: the genetic shortcut

There’s a clever middle path called Mendelian randomization. At conception, each of us inherits a chance shuffle of our parents’ gene variants. Some variants predict how much alcohol people tend to drink¹². Because those variants are dealt at random, comparing people who carry them works a bit like a lifelong trial.

Researchers at Massachusetts General Hospital and the Broad Institute used this method in a 2022 study in JAMA Network Open, as summarized by the Harvard Gazette¹². The team studied 371,463 adults in the U.K. Biobank. In ordinary comparisons, people who drank lightly or moderately came out best for heart health¹². But they also exercised more, ate more vegetables and smoked less than nondrinkers, and adjusting for a few such habits shrank the apparent benefit¹².

The genetic analysis disagreed with the standard comparison. People whose genes predicted heavier drinking had more high blood pressure and coronary artery disease. Risk rose slightly between zero and seven drinks a week and much faster beyond that¹².

Federal experts reading mostly observational data still split. In December 2024, a National Academies review concluded with moderate certainty that moderate drinking was associated with lower death rates than not drinking, STAT reported¹³. Weeks later, a federal interagency report found no significant net survival benefit from drinking at any age, noting that some abstainers avoid alcohol for health reasons¹³.

The Gazette described the older USDA standard for low-risk drinking as under two drinks a day for men and one for women, and the genetic data showed risk rising even within those limits¹². The 2025–2030 Dietary Guidelines, released in January 2026, give no daily number. They say: “Consume less alcohol for better overall health”¹⁴.

Genetic studies have their own blind spots, since one gene can affect several traits. Treat them as a strong clue, not a final verdict.

Most of the time, the two designs agree

It would be easy to conclude that only trials count. The data say otherwise.

A 2024 Cochrane methods review gathered 47 reviews and overviews that compared randomized trials with observational studies of the same questions¹⁵. On average, the effect estimates differed by about 8%, which the authors described as no difference or a very small one¹⁵. They rated that evidence as low certainty¹⁵.

So clashes are the exception. Our reading of the cases in this article, not a finding of that review, is that they cluster where people choose an exposure that travels with health-consciousness: hormones, vitamins, supplements and moderate drinking.

That matters to anyone who values natural health. Much traditional knowledge grew from careful observation of real people over generations, and cohort studies are its formal cousin. The respectful response is neither to dismiss observation nor to treat every link as proof. It’s to ask which kind of evidence you’re looking at and what could be fooling it.

A side-by-side guide and six questions to ask

Feature	Observational study	Randomized trial
Who picks the exposure	The people themselves	Chance
Best for	Long-term habits, rare harms, things you can’t assign	Testing whether a treatment causes a benefit
Main weakness	Hidden differences between groups, including healthy-user bias	Short follow-up, selected volunteers, high cost
Example here	Doll and Hill’s study of British doctors	The WHI hormone trial
Fair headline verb	“Linked to”	“Caused,” “lowered” or “raised”

Use these six questions the next time a study makes news.

1. Who chose? If people picked the habit themselves, assume the groups differ in other ways.
2. Who was compared with whom? New starters versus similar non-starters is fairer than current users versus never-users.
3. How big is the difference? A 20% to 40% gap can come from bias. A tenfold gap rarely does.
4. Would health-conscious people choose this? If yes, suspect healthy-user bias.
5. Has a trial tested it? If a trial exists, give it heavy weight, then check who was in it.
6. Do different methods agree? Cohorts, trials and genetic studies pointing the same way is the strongest signal you’ll get.

What the evidence says about the examples

Claim	Evidence level	Basis
Smoking causes lung cancer	Strong evidence	Large, dose-dependent risks across many studies, confirmed in the lab910
Estrogen plus progestin prevents heart disease in women aged 50 to 79	Evidence says no, or unsafe	WHI trial found more heart disease, strokes and clots1
Vitamin E pills prevent heart disease	Evidence says no, or unsafe	HOPE and other trials found no protection5
Moderate drinking itself protects the heart	Not enough evidence	No trial; the link shrinks after adjustment, a genetic analysis points the other way and federal reviews disagree1213
Observational studies and trials usually give similar answers	Promising but early	Small average differences across many comparisons, but rated low certainty15

Questions readers ask

The bottom line

Watching people and assigning them by chance are both legitimate ways to learn. Observation proved that smoking kills, while the same method, applied to hormones and vitamins, mistook healthy habits for a protective pill. Trials fixed that, though trials have blind spots of their own.

When you meet a new study, ask who chose the exposure, how big the effect is and whether other methods agree. Those three questions catch most of the errors that fooled experts for years.

Related reading

• When expert advice deserves weight, now that patients have the right to decide
• Three better questions to ask than whether something is natural or synthetic
• Chocolate tracks with Nobel prizes, and other lessons in telling a link from a cause
• Who paid for the study matters, even when the science is real

Sources

1. Writing Group for the Women’s Health Initiative Investigators. “Risks and Benefits of Estrogen Plus Progestin in Healthy Postmenopausal Women: Principal Results From the Women’s Health Initiative Randomized Controlled Trial.” JAMA, 2002 (copy hosted by McGill University). jhanley.biostat.mcgill.ca
2. Fred Hutchinson Cancer Center. “NHLBI Stops Trial of Combined Hormone-Replacement Therapy.” News release, July 9, 2002. fredhutch.org
3. Hernán MA, Alonso A, Logan R, et al. “Observational Studies Analyzed Like Randomized Experiments: An Application to Postmenopausal Hormone Therapy and Coronary Heart Disease.” Epidemiology, 2008 (Harvard DASH repository). dash.harvard.edu
4. U.S. Food and Drug Administration. “FDA Requests Labeling Changes Related to Safety Information to Clarify the Benefit/Risk Considerations for Menopausal Hormone Therapies.” FDA, November 10, 2025 (updated February 2026). fda.gov
5. NIH Office of Dietary Supplements. “Vitamin E: Fact Sheet for Health Professionals.” National Institutes of Health, 2021. ods.od.nih.gov
6. Ioannidis JPA. “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research.” JAMA, 2005 (abstract via Europe PMC). europepmc.org
7. Coronary Drug Project Research Group. “Influence of Adherence to Treatment and Response of Cholesterol on Mortality in the Coronary Drug Project.” New England Journal of Medicine, 1980 (abstract via Europe PMC). europepmc.org
8. Han MK. “Medication Adherence in COPD: What Have We Learned?” Thorax, vol. 64, no. 11. thorax.bmj.com
9. National Library of Medicine. “The Reports of the Surgeon General: The 1964 Report on Smoking and Health.” Profiles in Science, NIH. profiles.nlm.nih.gov
10. Yeh RW, Valsdottir LR, Yeh MW, et al. “Parachute Use to Prevent Death and Major Trauma When Jumping From Aircraft: Randomized Controlled Trial.” BMJ, 2018 (abstract, Beth Israel Deaconess Medical Center). research.bidmc.org
11. Hampton T. “Alcohol May Not Benefit the Heart, Researchers Say.” Harvard Gazette (Massachusetts General Hospital News), March 25, 2022. news.harvard.edu
12. Cueto I. “Alcohol Health Effects: New Federal Report Says a Single Drink Daily Raises Risk.” STAT, January 14, 2025. statnews.com
13. U.S. Department of Health and Human Services and U.S. Department of Agriculture. “Dietary Guidelines for Americans, 2025–2030.” January 2026. cdn.realfood.gov
14. Toews I, Anglemyer A, Nyirenda JLZ, et al. “Healthcare Outcomes Assessed With Observational Study Designs Compared With Those Assessed in Randomized Trials.” Cochrane Database of Systematic Reviews, 2024. pubmed.ncbi.nlm.nih.gov