When famous findings fall apart

In 2011 the social psychologist Daryl Bem published nine experiments in one of the field's most respected journals. They claimed evidence for precognition: sensing events before they happen. In one, students saw a list of words, tried to recall them, and only afterwards practised some of the words. They had recalled the practised words better, as if the later practice had reached back in time.

More than a thousand people took part, and eight of the nine experiments passed the standard statistical test. That was the alarm. Bem had used the ordinary methods of the field, and almost nobody believed his conclusion. If ordinary methods could prove the impossible, what else might they have proved? When three British researchers repeated the recall experiment three times and found nothing, the journal that had published Bem turned their paper away. It did not publish repeats of earlier studies.

What was the standard test Bem passed? Most psychology studies compare two groups and ask whether a difference between them is real or just the luck of who happened to land in each group. The answer comes as a p-value: the chance of seeing a difference at least this large if the thing being tested had no effect at all.

By long convention, a p-value below 0.05, one chance in twenty, counts as statistically significant and fit to publish. The catch sits in that one in twenty. Test something with no effect twenty times and, on average, one of those tests will still come out significant. That lucky hit is a false positive: evidence for something that is not there. One in twenty sounds safe, but only if each study makes a single test that was planned in advance. Many studies did not.

In 2011 the psychologists Joseph Simmons, Leif Nelson and Uri Simonsohn showed how easily honest researchers could fool themselves. Every study involves small choices: which of two measures to report, whether to test ten more people per group, whether to adjust for sex, whether to drop one of three groups. They called this latitude researcher degrees of freedom, and trying choice after choice until one works is now known as p-hacking.

Each choice looks harmless. But trying several and reporting the one that works gives chance many more tickets to win. In computer simulations, combining just four such choices raised the false-positive rate from 5 per cent to about 61 per cent. To make the point, they ran a real experiment and "showed" that listening to the Beatles song When I'm Sixty-Four left people nearly a year and a half younger.

Imagine twenty labs test the same idea, and the idea is wrong. Nineteen find nothing. One finds a significant result by chance. Journals, keen on clear and surprising discoveries, publish that one. The nineteen null results stay in a file drawer, unpublished, so readers see only the success. The psychologist Robert Rosenthal named this the file drawer problem in 1979.

The resulting skew in what reaches print is called publication bias, and incentives drive it. Careers are built on publications, and a surprising positive result is far easier to publish than a failure or a repeat of someone else's work. So the published record can fill up with findings that each look solid, while the failures that would put them in context are nowhere to be seen.

Many classic studies tested only a few dozen people. That matters because of statistical power: the chance that a study will detect an effect that really exists. Small samples are noisy, so a small study of a modest real effect will usually miss it. With twenty people in each group, a study of a modest effect finds it less than a quarter of the time.

Small samples also distort the hits. When a small study does reach significance, it is often because chance pushed its result unusually high that time. So the effects that get published from small studies tend to come out larger than the true effect. Run again with a big sample, the same effect shrinks, or vanishes if it was never there. The noise that made the first result exciting is exactly what the second study averages away.

How widespread was the problem? In 2015 the Open Science Collaboration, 270 researchers coordinated by Brian Nosek, published an answer in the journal Science. In the Reproducibility Project, they had repeated 100 studies published in 2008 in three leading psychology journals, using larger samples and, where possible, the original materials.

Of the originals, 97 had reported significant results. Of the repeats, only 36 per cent did. On average, the effects the repeats found were about half the size of the originals.

That did not mean the rest of psychology was false. A repeat can itself fail by chance, or copy the original imperfectly, and some effects were real but smaller than first claimed. What the project did show was that one significant finding in a good journal was far weaker evidence than most readers had assumed.

One of the most famous findings to be retested was about priming: the idea that a word or picture can nudge later behaviour without our noticing. In a 1996 study by John Bargh and his colleagues, students unscrambled sentences sprinkled with words linked to old age, such as Florida and wrinkle. As they left, they walked more slowly down the corridor than students who had seen neutral words.

In 2012 Stéphane Doyen and colleagues in Belgium repeated the study with more people, timing the walk with infrared sensors instead of a hand-held stopwatch. The slow walking disappeared. Then they led some of the experimenters to expect slow walking. Only then did it come back, and only with the experimenters who expected it. What the experimenters expected, not the words, may have produced the original effect.

The slow-walking study had a famous admirer. In his 2011 book Thinking, Fast and Slow, Daniel Kahneman presented priming research as a showcase of System 1, the fast, automatic mind that works beneath deliberate thought. In September 2012, as failed repeats piled up, he wrote an open letter to priming researchers: "I see a train wreck looming." He urged them to form a chain of labs, each rerunning a neighbour's study.

In 2017 he went further. Answering a blog analysis of his book's priming chapter, he wrote that he had "placed too much faith in underpowered studies". The irony was that his first paper with Amos Tversky, in 1971, had warned that researchers trust small samples too much. Surprising, elegant findings make good stories, and System 1 believes good stories easily. That pull works on scientists and readers alike.

Another celebrated idea was ego depletion: that self-control draws on a limited store, so using it up on one task leaves less for the next. In a 1998 study led by Roy Baumeister, hungry students who had to eat radishes while resisting fresh chocolate cookies later gave up sooner on an unsolvable puzzle than students who had been allowed the sweets. Hundreds of studies followed, and a 2010 review of 83 of them found a medium-sized effect.

Yet that review could count only published studies. In 2016, 23 labs ran one agreed procedure with 2,141 participants, having registered their plan and committed to report whatever they found. The effect came out close to zero. Defenders objected to the task used, so in 2021 a project of 36 labs, led by researchers who had long studied the effect, tried again. It also found almost nothing.

The Stanford Prison Experiment of 1971 is among the most famous studies in psychology. Philip Zimbardo split student volunteers into guards and prisoners in a mock jail, and the study was stopped after six days as the guards grew abusive. For decades it was told as proof that ordinary people turn cruel when handed a role.

It was never repeated in the same form. When the psychologists Alex Haslam and Stephen Reicher ran a partly similar prison study for BBC television, filmed in late 2001, the guards never settled into their role, and the prisoners eventually overpowered them. In 2018 the French researcher Thibault Le Texier, working from the Stanford archives, showed that the guards were briefed on how to behave. That points to demand characteristics: cues that tell participants what the experimenter wants to see.

The most direct fix is to decide before looking. In preregistration, researchers post a time-stamped plan online before collecting any data: the hypothesis, the sample size and the exact analysis. Choices made after seeing the results then stand out as departures from the plan.

Registered Reports, launched at the journal Cortex in 2013 by the neuroscientist Chris Chambers, go a step further. Reviewers judge the question and the method before the study is run, and a sound plan is accepted before the results are known. A null result is then as publishable as a striking one. The difference shows. A 2021 comparison found that the first hypothesis was supported in 96 per cent of standard psychology papers, but in only 44 per cent of Registered Reports.

Another fix is scale. In the Many Labs project, published in 2014, labs around the world ran the same 13 classic and newer effects in 36 samples totalling 6,344 people, then pooled the results. Ten of the 13 replicated, and one had only weak support. Two did not replicate, both of them priming effects: seeing a national flag did not make people more conservative, and images of money did not make them defend the social system.

One of the ten, anchoring, came back even stronger than first reported. That is the pull an arbitrary first number exerts on a later estimate, and Kahneman had helped establish it. Projects like this, with data and methods open for anyone to check, show that the crisis did not prove psychology empty. It was science correcting itself, keeping what survives and dropping what does not.

Trusting a claim only when others confirm it is an old idea. Muslim scholars of hadith, the reports of the Prophet Muhammad's words and deeds, judged each report partly by its isnād, the chain of people who had passed it on. One test was a narrator's precision, called ḍabṭ. Critics compared what a narrator reported from a teacher with what that teacher's other students reported. A narrator whose peers usually matched him was trusted; one who often contradicted them was not.

Critics also searched for corroboration, support for a report through other chains, which could strengthen one whose own chain was weak. A report carried by so many independent chains that they could not have agreed on a falsehood, called mutawātir, was beyond doubt. The methods are not statistics, and the scholars were weighing people, not experiments. But the instinct is shared: trust grows with independent confirmation.