The bitter lesson

Why should programs that know less keep winning? Sutton's answer is the falling price of computing.

In 1965 the engineer Gordon Moore noticed that the number of components on a chip was doubling every year, and in 1975 he revised the pace to every two years. The trend became known as Moore's law. Sutton points to its broader form: the cost of each unit of computation has kept falling exponentially, decade after decade. By one tally of real machines, buying a billion calculations a second cost nearly ten trillion times less in 2022 than in 1961, after inflation.

Because computing keeps getting cheaper, a method that can turn extra computing into better results grows stronger every few years without being redesigned. Knowledge written in by hand stays exactly as good as it was.

If general methods win, why do researchers keep building knowledge in? Sutton blames the timescale of research.

Most projects are planned as if the computing power on hand will stay the same. With computing held fixed, one of the few ways left to improve a system is to build in human knowledge, meaning what experts understand about the task. That usually pays off quickly, and it is satisfying to see your own understanding work.

But over a stretch only slightly longer than a typical project, far more computing becomes available. Meanwhile the two approaches compete for researchers' time: hours spent encoding expertise are hours not spent on methods that scale. Worse, a system tangled up with hand-built rules tends to be harder to scale up when the extra computing arrives.

For decades, many computer-chess researchers tried to capture the special understanding that grandmasters bring to the game. Then, in May 1997, IBM's Deep Blue beat the world champion, Garry Kasparov, by 3½ games to 2½, and it won mostly another way.

Deep Blue relied on search: looking ahead through possible moves, replies and counter-replies, and scoring where each line leads. With 480 custom chess chips it could examine more than 100 million positions a second. People still tuned its scoring rules, and learning played little part, but its strength came from how deeply it could search.

Many chess researchers were dismayed. They called the win mere “brute force”, said it was no general strategy, and complained that it was not how people play chess. Sutton opens his case with this story.

Go was a harder test. On a typical turn a chess player has about 35 legal moves; a Go player has about 250, so the tree of possible games explodes far faster. Early Go programs poured effort into human knowledge of the game to avoid searching, and that effort was overtaken once search was used well at scale.

The breakthrough came about twenty years after Deep Blue. In March 2016 DeepMind's AlphaGo beat Lee Sedol, one of the world's strongest players, four games to one. It combined search with learning: it first studied 30 million positions from games by human experts, then improved by playing against itself.

Its successor, AlphaGo Zero, was given only the rules. It learned entirely by self-play, starting from random moves, and after three days it beat the version that had defeated Lee Sedol by 100 games to 0.

Chess and Go point to what Sutton sees as the heart of the matter. Of all the techniques in AI, two kinds can put enormous amounts of computing to use: search and learning.

Search spends computing while a decision is being made. The more positions it can examine, the better its choice. Learning spends computing on experience. The more examples or games it can process, the better it gets, and self-play lets a program produce as much experience as it has computing to spare.

What the two share is that they scale: they keep improving as the available computing grows, even when it grows enormous. Knowledge written in by hand does not. A rule a grandmaster teaches a program is just as good, and no better, on a computer a thousand times faster.

Speech recognition tells the same story. In the early 1970s the US defence research agency, then called ARPA, funded a five-year programme to build systems that could understand spoken sentences. Several teams packed their systems with human knowledge of language: of words, of the sounds that make them up, and of the grammar that constrains a sentence.

A different approach treated speech as a matter of probabilities, learned from recorded examples rather than written down by experts. Through the 1980s one statistical tool, the hidden Markov model, spread until it dominated the field. It did far more computation than the rule-packed systems, and it won out.

Deep learning, which trains large layered networks of simple adjustable units on examples, later pushed further in the same direction, using even less built-in knowledge, even more computing, and huge collections of recorded speech.

Computer vision followed suit. For decades researchers decided by hand what a program should look for in an image: edges, simple 3D shapes, and later carefully engineered descriptions of small patches, such as the popular SIFT method published in 1999. These hand-designed features were the program's whole view of a picture.

In 2012 a University of Toronto team entered a deep neural network in the ImageNet contest, which asks programs to label photos across a thousand categories. It was given almost no rules about what to look for, and learned its own features from more than a million labelled photos. Allowed five guesses per photo, it missed 16.4 per cent of the test images. The best entry built on hand-designed features missed 26.2 per cent. Within a few years, most of the field had switched to learned features.

Across chess, Go, speech and vision, Sutton sees one story repeat. Researchers build their knowledge into a system. That helps at first, and it is satisfying. Then progress flattens into a plateau, and the built-in knowledge can even hold further progress back. Finally a breakthrough comes from the opposite direction: methods that scale up search and learning.

The success is bitter because it defeats the approach researchers favoured, one centred on human understanding and often modelled on how they believed their own minds work. Sutton's phrase for the mistake is building in “how we think we think”. Because the defeat stings, he argues, the lesson is only half absorbed, and the field keeps repeating it.

His point is not that knowledge never helps. It is that knowledge built in by hand has, again and again, been overtaken in the long run.

Sutton draws two lessons from this history. The first is the power of general methods that keep scaling with computing, and the two that seem to scale without limit are search and learning.

The second concerns what should be built into an AI system at all. The things a capable system must deal with, such as space, objects, other agents and symmetries, belong to an outside world that is endlessly complex, too complex for anyone to write down by hand. So rather than building in what we have discovered, Sutton argues, we should build in only meta-methods: methods that can find and capture that complexity for themselves. He wants AI agents “that can discover like we can”.

This is advice about how to engineer AI systems, not a claim about what a human mind is.

Six days after Sutton's essay, the roboticist Rodney Brooks, a former director of MIT's artificial intelligence laboratory, posted a reply called “A Better Lesson”. His main charge: successful AI still depends on human ingenuity, which has simply moved somewhere else.

His example was image recognition. The networks that won there have a front end designed by people so that an object is recognised wherever it appears in the frame. People also choose each network's design and how it is trained, and they label the millions of training images, which is knowledge poured in through another door.

Brooks also counted costs he felt Sutton ignored: training that only very large companies can afford, the energy it consumes, and a slowing Moore's law. The better lesson, he wrote, is to weigh the total cost of a solution, including all the human ingenuity it needs.

Another reply came in April 2019 from Max Welling, a machine-learning researcher in Amsterdam who called himself a strong believer in computation. His point was that the lesson depends on data as well as computing.

Sutton's examples, Welling noted, are narrowly defined problems with plenty of data. A Go program can generate its own games without end, and speech systems can train on vast recordings. There, methods that learn from data shine, because new cases resemble ones already seen.

Trouble starts when a system must extrapolate, handling situations unlike anything in its data. A self-driving car meets a long tail of rare events that no dataset covers well. Where data is scarce, Welling argued, human knowledge still has to fill the gaps, for instance through a model of how the world works.