Transformers

Nobody writes out the rules a language model follows. It is a neural network: layers of simple units, each taking in numbers from the layer before, multiplying each by a weight, adding up the results and passing a new number on. Every weight is an adjustable setting, and together these settings are the network's parameters. Large models have billions or more. A network with many layers is called deep, which is where the name deep learning comes from.

Training sets their values. The network starts with random parameters and is shown an example, such as a sentence with its next word hidden. It makes a guess, the size of its error is measured, and every parameter is nudged a tiny amount in whichever direction makes the error smaller. Repeated over billions of examples, those small nudges add up to a network that guesses well.

A neural network works only with numbers, so text has to be converted first. The first step cuts it into tokens, the pieces a model actually takes in and puts out. A common word is often a single token, while a rarer word is split into fragments: GPT-3's tokenizer cuts “cleverest” into “cle”, “ve” and “rest”. Each piece in the model's fixed vocabulary has its own ID number, and that vocabulary held 50,257 of them.

Why fragments rather than whole words? No list of whole words is ever complete, because new names, misspellings and technical terms turn up all the time. A modest set of pieces can spell any word, including ones never seen before. In English, one token averages about three-quarters of a word, so a hundred tokens hold roughly seventy-five words.

A token's ID number is only a label; it says nothing about what the token means. So the first thing a model does with each token is swap it for a long list of numbers, called its embedding. GPT-3 used 12,288 numbers for every token.

Picture each list as the position of a point in a space with thousands of directions. Training moves the points so that tokens used in similar ways end up close together: “cat” lands near “dog” and far from “carburettor”. Directions can carry meaning too. In 2013 Tomas Mikolov and two colleagues showed that starting from the list for “king”, taking away the list for “man” and adding the list for “woman” gave a point whose nearest word, leaving out the three words used, was “queen”.

Before 2017, most leading language networks were recurrent: they took in a text one token at a time, folding each into a running summary, a fixed-size list of numbers handed from one step to the next.

That design had two costs. The first was distance. Every new token reshaped the summary, so the trace of a word many places back grew faint, and during training the error signals faded as they were passed back through many steps. Long short-term memory networks, introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997, eased this problem but did not remove it.

The second cost was speed. The steps must run in order, each waiting for the last, so the work on a long text could not be spread across many chips at once. That was a serious brake on training with long texts.

“The animal didn't cross the street because it was too tired.” To handle “it”, a program has to link it to “animal”. Change “tired” to “wide” and “it” now means the street.

Attention makes that link directly. For each word, the model gives every other word a weight for how much it should shape this one, then blends in information from all of them in proportion to their weights. In the tired sentence, “animal” gets a large weight from “it”; in the wide sentence, “street” does. There is no running summary to fade, so a word twenty places back is as easy to reach as the one next door.

The idea took off in translation. In 2014 Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio let a recurrent network look back over the whole source sentence as it produced each translated word.

How are those weights set? In self-attention, attention among the tokens of a single text and the kind used inside a Transformer, each token's embedding is turned into three new lists of numbers, each made by its own trained parameters.

The query stands for what this token is looking for. The key stands for what a token offers, ready to be compared with other tokens' queries. The value is the information a token hands over. A token's weight on another comes from how well its query matches that token's key. The weights are scaled to add up to one, and the token takes in a weighted average of the values. It works like a library search: the query is what you type, each book's key is its catalogue entry, and its value is the book itself.

Nobody programs what queries and keys pick out. Training shapes them.

A single set of queries, keys and values blends everything into one average, blurring different kinds of connection. So each Transformer layer runs several sets side by side, each with its own trained parameters. Each set is an attention head, and each head can track a different kind of link. The 2017 paper's base model used eight heads in every layer and its larger model sixteen; GPT-3 used ninety-six.

Nobody assigns the heads their jobs. In the 2017 paper's pictures, several heads connected the verb “making” to the words “more difficult” further along, and two heads appeared to link the word “its” back to the noun it stood for.

The heads' outputs are combined and passed through an ordinary neural network layer. Transformers stack many such layers, six in each half of the original and ninety-six in GPT-3, so later layers can build on links found by earlier ones.

Attention has a blind spot: word order. It compares every token with every other in one go, and nothing in that comparison records which came first. On its own it would handle “the dog bit the man” and “the man bit the dog” as the same set of words.

So before the first layer, a Transformer adds a second list of numbers to each token's embedding, one that depends only on the token's position in the text. This positional encoding stamps every token with where it sits. The 2017 paper built it from sine and cosine waves of many different lengths, read off at each position, so every position gets its own pattern of values. The authors also tried letting training learn the position lists instead, and found the results nearly identical.

In June 2017 eight researchers, most of them at Google, posted a paper titled “Attention Is All You Need”. Translation networks of the time already used attention, but bolted onto recurrent networks. The paper threw out recurrence and built a network from attention layers alone. They called it the Transformer.

It was designed for translation, in two halves. An encoder took in the whole English sentence at once, each word attending to every other; a decoder then produced the German or French one token at a time, attending to the encoder's output and to the words it had already written.

With no step waiting on the one before, every word in a training text can be processed in parallel across many chips. The larger model trained in three and a half days on eight graphics chips and beat the best previous English-to-German score by more than two points.

A chat assistant writes its reply with a loop. The model takes in all the text so far, including any reply already begun, and predicts how likely every token in its vocabulary is to come next. It then picks one token, usually by a weighted random draw, so likely tokens usually win. A setting called temperature controls that draw: raising it makes the choice more varied, lowering it makes it more predictable. The chosen token is appended to the text, and the slightly longer text goes back in to predict again, until the model picks a token that means stop.

Models with published designs, such as GPT-3 and Llama, use only the decoder half of the Transformer, so each token can draw only on the tokens before it. The model is first trained to continue text, then trained further on example conversations and on people's rankings of its replies.

Most machine learning needs people to label examples: this photo shows a cat, that email is spam. Language models get around that bottleneck. Any piece of writing already holds its own answers, because each word is the correct guess for what follows the words before it. So a model can take any text, predict every next token in it, compare each guess with the real one and adjust.

This first and largest stage of training is called pretraining, and it means the supply of examples is as large as the text that can be gathered: books, websites, articles, code. GPT-3 was trained on 300 billion tokens of it. What the model absorbs is whatever patterns fill that text, including its errors, gaps and slants.

In 2020 Jared Kaplan and colleagues at OpenAI measured how much bigger language models improve. A model's prediction error fell smoothly and predictably as three things grew: its parameters, its training text and the computing power spent training it. Each tenfold increase in parameters cut the error by about the same fraction. A regular pattern like this is called a scaling law, and it let researchers forecast a bigger model's error before building it. GPT-2 had 1.5 billion parameters in 2019; GPT-3, a year later, had 175 billion.

Size alone was not the whole story. In 2022 a DeepMind team showed that many large models should have been trained on more text. Their Chinchilla, with 70 billion parameters trained on 1.4 trillion tokens, beat the 280-billion-parameter Gopher on the same computing budget. Training text has kept growing since: Meta reported over 15 trillion tokens for Llama 3 in 2024.

A language model produces the text its training makes likely, and nothing in that process checks whether a sentence is true. In a 2025 OpenAI study, a leading open model was asked for one of the authors' birthdays, with the instruction “If you know, just respond with DD-MM”, and gave three different wrong dates on three tries. Fluent, confident statements that are false are called hallucinations.

The study argued that testing makes this worse. A score that counts only right answers rewards guessing over “I don't know”, as a multiple-choice exam does. On one quiz of short fact questions, an older OpenAI model almost never declined: it was right 24% of the time and wrong 75%. A newer model declined about half the questions and was wrong far less often.

So treat a model's factual answer as a claim to check against a reliable source.