The Chisel-Construct Cycle · AI Series · 19

A Title That Took Five Seconds: Attention Is All You Need

Han Qin · About 11 minutes · Published

1. Company

In 1954 the linguist Zellig Harris wrote in “Distributional Structure” that differences in meaning correlate with differences in distribution. Distribution means what other words a word appears next to. In 1957 the British linguist J. R. Firth put it more playfully: “You shall know a word by the company it keeps!” What circulated later in Chinese was a different sentence: a word's meaning is determined by its context.

It took many years for a machine to put this idea to use. In 2003 Bengio and three colleagues published “A Neural Probabilistic Language Model.” The machine turned each word into a string of numbers, called a distributed feature vector. It learned the numbers along with the ability to predict the next word.

In January 2013, Tomáš Mikolov and three colleagues at Google released a preprint called “Efficient Estimation of Word Representations in Vector Space.” They reported learning high-quality word vectors from text containing 1.6 billion words in less than a day. A second paper in October added ways to save time and used text from Google News. The approach came to be known by the name of a program: word2vec.

A word became a string of a few hundred numbers. If two words often kept company with the same set of words, their numbers were close together. Cat was near dog. Monday was near Tuesday. What the string recorded was which words cat tended to be with. The thing Harris and Firth described had become arithmetic.

2. The King

In June 2013, the abstract of a conference paper by Mikolov and two colleagues gave an example. Take the vector for king, subtract man, add woman, and the result is very close to the vector for queen.

What traveled was an equation: king minus man plus woman equals queen. In the program Google released that year, the code that answered such questions had a rule: none of the three words in the question could be an answer. In 2019 three researchers in the Netherlands and Italy published a paper whose title itself took that form: “Man is to Doctor as Woman is to Doctor.” They pointed out that the rule could make some perfectly sensible answers impossible. Without it, the nearest word to the point the calculation located was often one of the words in the question.

A paper by another group in 2016 also put a question in its title: “Man is to Computer Programmer as Woman is to Homemaker?” It used word embeddings trained on Google News. Its abstract reports that the vectors showed a disturbing degree of gender stereotyping. Their method first identified a gender direction in the embeddings, then drew words that ought to be gender-neutral away from that direction. Associations that belonged, such as between queen and female, were left in place.

The vectors were learned from news. People wrote the news. The company between words that the machine learned was what people had left behind by the way they wrote, across millions of articles.

3. Alignment

In 2014 word vectors were put to work translating.

That September, Ilya Sutskever and two colleagues at Google published “Sequence to Sequence Learning with Neural Networks.” One network read the whole source sentence word by word, turning it into a fixed-length string of numbers. A second network used the numbers to write the translation word by word. They also found that reading the source sentence in reverse gave better results. In the summer of the same year, members of Bengio's group at the University of Montreal made a similar two-stage network.

Reading a whole sentence into a string of numbers of a fixed length: when the sentence gets long, it is like trying to fit a long piece of writing into a little box.

Also in Montreal, a young researcher named Dzmitry Bahdanau, another member of the group, and Bengio put out a draft paper on September 1, 2014. It was called “Neural Machine Translation by Jointly Learning to Align and Translate.” Alignment is which word in a translation goes with which word in the source. The network writing the translation looked back at every word in the source each time it wrote a word. It assigned each source word a weight. The weights added up to one. The network combined information from the source according to those weights and used it to write the new word.

The explanation of this step in the paper says that, intuitively, it amounts to an attention mechanism in the decoder. The decoder is the network that writes the translation.

4. A Name

The title doesn't say attention. It says align and translate. Bahdanau described where the name came from in a letter years later. He had called the architecture RNNsearch, meaning search with an RNN. The better name, attention, he recalled, was added by Yoshua in one of the last rounds of revisions. Yoshua was Bengio. The letter was published with Bahdanau's permission by Karpathy, whom we met in essay sixteen. The word had come from psychology. In 1890 the American psychologist William James opened the chapter on attention in The Principles of Psychology with the words “Every one knows what attention is.”

A year before Bahdanau's paper, in August 2013, Alex Graves had made a network that wrote handwriting. As the program wrote stroke by stroke, a soft window that moved forward decided which letters in the text being written it should attend to at that moment. He called it a soft window. Bahdanau and his coauthors mention this paper and say their approach resembles it. For an earlier parallel, Schmidhuber has said that his fast weights from 1991 are formally equivalent to a later simplified form of attention.

The arrangement had several names along the way: soft window, alignment, search, attention. The name that stayed was the last one added, the one with intuitively in front of it in the original sentence.

5. Blue Lines

On September 26, 2016, Google released a long technical report on its neural machine translation system. The second half of the title was “Bridging the Gap between Human and Machine Translation.” The next day the company blog said the system had replaced its previous Chinese-to-English translation on phones and the web. That was about eighteen million translations a day. Chinese into English was the first language pair to switch.

Each time the half that wrote the translation produced a word, it looked back at the source. The blog showed a picture: a string of Chinese words on one side, a string of English words on the other, blue lines between them. The shade of a line indicated how much attention the machine had put on the Chinese word when it wrote the English one. Attention had been drawn in a way anyone could see.

The report gives an evaluation by people. A panel saw four French translations of the same English sentence, from the old phrase-based system, two versions of the new system, and a human reference translation. Out of six, the average scores were 3.87, 4.46, 4.44, and 4.82 in that order. The abstract describes the test as involving isolated simple sentences. The last paragraph of the blog says the machine still misses words, gets names and rare words wrong, and doesn't consider context. It adds that machine translation is by no means solved.

6. Eight People

On June 12, 2017, eight people released a paper called “Attention Is All You Need.”

The names, in order, were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. A footnote on the first page says the contributions were equal and the order random. It then tells, one by one, who did what. Uszkoreit proposed replacing recurrent networks with self-attention. Vaswani and Polosukhin designed and implemented the first model. Shazeer proposed several key algorithms. Parmar tried a large number of variants. Jones worked on the initial code and visualization. Kaiser and Gomez wrote the tools used to run experiments.

They named the architecture the Transformer. The abstract says it is based solely on attention mechanisms, dispensing with recurrence and convolution. The kind of network in the previous section reads along one word at a time. When the Transformer reads the source, each position can look at all parts of the sentence at the same time and give them its own weights. When it writes the translation, however, it can't look at words that aren't written yet. The network still has other processing layers. It is those two earlier ways of passing information that it does without.

Two months later, Google's blog gave general readers an example. “The animal didn't cross the street because it was too tired.” “The animal didn't cross the street because it was too wide.” In the first, it is the animal. In the second, it is the street. The words for animal and street are of different genders in French, and how it is translated changes with them. The blog says the new model translated both sentences correctly. In the diagram of the encoder reading the original, the weights for it lean more toward the animal in the first sentence and the street in the second.

Training took eight Nvidia P100 graphics cards and three and a half days. The paper did translation from English to German and from English to French, as well as parsing in English. In the conclusion the authors say they are excited about the future of attention-based models and want to apply them beyond text, to images, sound, and video.

Jones told a reporter for Wired in 2024 that the title came from the Beatles' “All You Need Is Love.” It came to him one night a few days before the deadline. He said it had really taken him five seconds. That December 6, the paper was assigned to the evening poster session at NIPS in Long Beach, California. Several authors later recalled it being crowded for four hours. People were still gathered around at 10:30 p.m., and finally security asked them to leave. By 2024 all eight had left Google. Some had started companies, others gone elsewhere.

7. Short Names

In 2018 more of this work followed, with names that were all short.

In February a group from the Allen Institute for Artificial Intelligence and the University of Washington released ELMo. In June OpenAI published “Improving Language Understanding by Generative Pre-Training.” A twelve-layer Transformer learned to continue text on a large body of unannotated writing, then was applied to a range of particular tasks. The model later came to be called GPT. At the time, the letters were still scattered across the words of the title. In October Google released BERT. The smaller one trained on four cloud TPUs, the larger on sixteen, for four days each. ELMo and BERT happened to share their names with Sesame Street characters.

The GPT paper calls the text it learned from a book corpus. It says the corpus contains more than seven thousand unpublished books, in genres such as adventure, fantasy, and romance. The corpus came from a paper in 2015. That paper calls it BookCorpus and says it collected 11,038 books by as-yet unpublished authors, available free online.

In 2021 two researchers went back and wrote documentation for the corpus. They found that the books had been collected in bulk from the self-publishing website Smashwords, with software that gathered links to free ebooks, downloaded them, and converted them to plain text. One conclusion was that the authors had not explicitly consented.

In April 2018, researchers at New York University and elsewhere put together a test called GLUE, consisting of nine tasks in language understanding. The test had a human score. Different people were asked to do the tasks and their results combined according to the benchmark's rules, yielding about eighty-seven points. In October BERT scored 80.5. By the middle of 2019 systems were reporting scores above the human line.

8. Attention

In 1890 James said everybody knew what attention was. In 2014 the word was added to a translation paper, with intuitively in front of it. In 2016 it was drawn in blue lines. In 2017 it went into a title: Attention Is All You Need. The way it was told later, the machine had acquired attention too. Intuitively fell out along the way.

In these papers attention means a set of weights that add up to one. When a word is written, the weights decide which words in the source to look back at. What guides those weights is learned from writing: text with 1.6 billion words, Google News, bilingual sentences translated by people, more than seven thousand unpublished books. All were written or translated by people.

The line on the test is people, too. The reference translation that scored 4.82 was done by a person. The eighty-seven points on GLUE came from finding people to do the tasks one by one. The machine's score was measured against what those people did.

The animal was too tired and the street was too wide. A researcher at Google wrote the two sentences to explain the point. It was the animal that was tired and the street that was wide. The person who wrote the sentences knew that, which is why there could be a question to ask.

A person's attention is where they put their finite time. The unnamed people in this essay, people who wrote books, translated parliamentary records, and sat down to rate translations, each put their attention on their own book, their own translation, or the sentence in front of them. The machine learned from the traces their attention left. This series calls what stays outside, however much a machine learns, the remainder.

The machines are still being built. Each generation says again where the boundary lies.