How to read a study: what a result can and cannot tell you

In a randomised trial, chance decides who gets the treatment, so the groups tend to be alike, even in things nobody measured (the larger the trial, the more reliably). In an observational study, the groups may differ in health, income or habits, and those differences can create a link of their own (confounding). The GRADE approach therefore starts randomised trials as high-quality evidence and observational studies as low-quality, then adjusts each for how well it was done.² Systematic reviews sit at the top of the classic pyramid, but they are better seen as a lens on the studies below them: a review of weak studies is still weak evidence.¹

The 1991 overview judged the protective effect “unlikely to be explained by confounding factors”.⁶ The randomised trial then found more heart disease with oestrogen plus progestin (hazard ratio 1.29, nominal 95% confidence interval 1.02 to 1.63), and its investigators advised against the combination for preventing heart disease.⁷ The two bodies of evidence did not test exactly the same thing: most of the earlier studies were of oestrogen alone, and the trial tested oestrogen plus progestin. But the observational studies could not rule out that the women who chose hormones differed from those who did not. The trial could.

A control group. Without one, natural recovery, or symptoms easing from their worst point (regression to the mean), can look like an effect. A placebo also captures the effect of expecting to get better.

Randomisation, kept hidden. Researchers who choose the groups may, without meaning to, give the treatment to those likely to do well. Randomisation prevents that only if the next assignment stays hidden (allocation concealment): in 250 trials, effects were exaggerated by 41% when it was inadequate and by 30% when it was unclear.⁸

Blinding. People who know what they got may report, or score, results more kindly. Trials that were not double-blind exaggerated effects by 17%.⁸

A plan published first. Since 2005 the journals of the International Committee of Medical Journal Editors have required trials to be registered in a public registry, at or before the start of enrolment, before they will consider them for publication.¹⁰ When 102 trials were compared with their protocols, 62% had changed, added or dropped a main outcome.⁹

Animal studies rarely report these protections. In most systematic reviews of animal research, fewer than half of the studies reported randomisation, and 0 to 3% a sample size worked out in advance.¹¹ In 2,671 animal studies of drugs collected for systematic reviews, 25% reported randomisation and under 1% a sample size calculation.¹²

A study’s statistical power is its chance of detecting an effect that really exists. Small studies miss real effects; and when they do find one, it is less likely to be real and tends to look too large, because only lucky results clear the bar for “significant”.^13,18 Chance is big in small groups: toss a fair coin 8 times and you get 6 or more heads in about 1 of every 7 tries (37 of 256 possible sequences).

At least 50 publications had described drugs that lengthened the lives of mice with ALS, a fatal nerve disease, yet besides riluzole none had worked in patients. Retested in properly sized studies, none of 8 gave a survival benefit, riluzole included.¹⁹ Check too what n counts: in one 2026 tendon study of BPC-157, each group’s 8 rats were split, so each test used 4 tendons per group.²⁰

Extra heartbeats after a heart attack are linked with sudden death, which is why the CAST drugs were expected to help; the reason for the excess deaths remained unknown.²¹ A treatment can improve a surrogate without improving, or while worsening, the outcome it is meant to predict.²² A claim built on a surrogate is a prediction, not a result.

Outcome trials close this gap for some medicines: the SELECT trial of semaglutide counted heart attacks, strokes and heart deaths, not only weight (see the worked example below).²³

Choosing which analyses to show because of the p-values they produce can give small p-values even when the treatment does nothing.²⁴ That is why the main outcome should be named in advance. And a trial whose main result is not significant may still be presented as positive: in 72 such trials, the conclusions of 42 abstracts (58%) contained spin.²⁵

SELECT followed its participants for a mean of 39.8 months. It was a double-blind, placebo-controlled trial paid for by the manufacturer.²³ We use it because it is large and well reported, not to recommend anything.

1. Without treatment: on placebo, 701 of 8,801 people had one of these events, about 80 in 1,000.
2. With treatment: on semaglutide, 569 of 8,803 did, about 65 in 1,000.
3. Absolute difference: 80 minus 65, about 15 in 1,000 (1.5 percentage points).
4. Relative difference: 15 out of 80, close to the 20% reported. (The hazard ratio of 0.80 also accounts for when events happened.)
5. Number needed to treat: 1,000 divided by 15, roughly 67 people treated for about three years for one to avoid an event. This is our arithmetic, and it applies only to people like those in the trial.
6. A rarer problem: if 8 in 1,000 would have had the event, 20% fewer means about 1.6 fewer in 1,000.

Risk researchers recommend absolute risks and counts of people (natural frequencies). Their standard example: a 25% reduction in breast-cancer deaths from mammography screening means 1 fewer death in 1,000 women.²⁶ The same goes for harms: “doubles the risk” of something affecting 1 in 10,000 means 1 extra case per 10,000.

Preprints. Among preprints posted between September 2019 and April 2020 and published in a journal by 30 April 2020, the conclusions of 7.2% of non-COVID-19 and 17.2% of COVID-19 abstracts changed, mostly not in substance; preprints never published were not studied.²⁷

Peer review. 607 BMJ reviewers received test papers with nine planted major errors each. On the first paper they found on average 2.58 of the nine, and about 3 on the later ones.²⁸

Press releases. Of 462 university health press releases, those about studies in animals, cells or computer models stretched the results to people 36% of the time; when they did, 86% of the news stories did too, against 10% otherwise.²⁹ Spin in press releases about trials went with spin in the paper’s own abstract.³⁰

A 2026 study of BPC-157 and Achilles tendons was the first test of this claim in living animals by a group outside Zagreb.²⁰

• In whom, how many? Rats with cut and surgically repaired tendons: 32 in four groups of 8, with 4 tendons per group in each test.
• Compared with what, and who knew? Rats with the same injury and no treatment; the pathologist was blinded and specimens were coded.
• What was measured, and how big? Breaking strength and tissue scores, both stand-ins. Strength and overall tissue scores were higher with BPC-157 but not statistically significantly; one collagen measure was significantly higher Animals(Evidence level: Shown in animal studies).
• Repeated, and who paid? It was itself the first independent test, in a related model; the authors declared no financial support.

So: a blinded, small, independent animal study, too small to rule a modest effect in or out, and not evidence of an effect in people. See the BPC-157 page.