The Chisel-Construct Cycle · AI Series · 21

How Much of the Internet Did GPT-3 Read?

Han Qin · About 14 minutes · Published

1. Unicorns

On February 14, 2019, OpenAI put a piece of writing on its blog. The first two sentences were written by a person. Scientists had found a herd of unicorns in a valley in the Andes where no one had ever been, and the unicorns spoke fluent English. A program wrote the long passage that followed. It named the species, described what the unicorns looked like, and invented a professor to lead an expedition. Paragraph after paragraph, it read like a news report.

The program was called GPT-2, the next version of the GPT in essay nineteen. It had one and a half billion parameters and learned from text on eight million web pages. It learned to do just one thing: look at the text so far and guess the next word.

Beside the sample the blog says it usually took several tries to get a good one. On topics the model knew well, about half the attempts were presentable. The blog also lists what often went wrong: a passage repeated itself, or the subject lurched somewhere else. Once the model wrote about fire burning underwater. Someone had picked the unicorn passage from several tries.

The same post says that, out of concern about malicious uses, the company would not release the trained model. It released only a smaller version. The blog calls the decision and the discussion around it an experiment. It says they weren't sure it was the right thing to do that day.

The headline in Britain's Guardian that day said a new generator of fake text might be too dangerous to release, according to its creators. The next day the World Economic Forum's site republished a piece whose headline said scientists had made an AI they thought was too dangerous to release. Farther down the line it became OpenAI said GPT-2 was too dangerous. Experiment, not sure, might, thought: as the story traveled, they fell away one by one.

2. Four Releases

The version released that February was the smallest. The medium version came out in May. On August 20, OpenAI released the one with 774 million parameters and wrote that at least five groups had already replicated GPT-2. That month two recent graduates, Aaron Gokaslan and Vanya Cohen, built a comparable model and put it online. They used about fifty thousand dollars' worth of Google Cloud credits. The headline in Wired said OpenAI had called its code risky and two graduates had built it anyway.

On November 5, the largest version, with one and a half billion parameters, was released in full, with code and weights. Weights are the long string of numbers worked out during training. They hold what the model has learned to do. The company called this the last step in a staged release. It said it had seen no strong evidence of misuse up to that point. From February to November: nine months, four releases.

3. Bitter

A year earlier, on May 16, 2018, OpenAI's Dario Amodei and Danny Hernandez had published “AI and Compute.” They picked a set of the largest AI training runs of the preceding years, estimated how much computation each used, and plotted the estimates. From 2012 to 2018 the amount rose by about three hundred thousand times, doubling every 3.4 months on average. The two ends of that measure were AlexNet from essay seventeen and AlphaGo Zero from essay eighteen, the one that played only itself.

On March 13, 2019, Richard Sutton of the University of Alberta posted a short essay, “The Bitter Lesson,” on his website. Its first sentence says the greatest lesson of seventy years of AI research is that general methods able to make use of computation ultimately work best, and by a wide margin. His examples are chess, Go, speech, and vision. Each time researchers began by writing what they themselves knew into a machine. In the short term it worked, and the people who wrote it were pleased. Years later, what won was the approach that didn't depend on that knowledge, but relied on search and learning and grew with the amount of computation. The bitterness belonged to the people who had written what they knew into the machine. The games in essays eleven and eighteen, and the line about firing linguists in essay fifteen, can all be set along the line Sutton draws.

Computation takes machines. Machines take money. On March 11, 2019, OpenAI announced the creation of OpenAI LP, a hybrid of a for-profit and a nonprofit, with a cap on returns. Investors in the first round could get back at most a hundred times what they put in. Anything above that would go to the original nonprofit. The announcement said the structure would make it easier to raise capital, buy computing power, and attract talent. On July 22, Microsoft invested a billion dollars and became its exclusive cloud provider.

4. A Law

In January 2020, Jared Kaplan and colleagues at OpenAI published “Scaling Laws for Neural Language Models.” They trained a large set of language models of different sizes and measured how badly each one guessed the next word. The number for how far off it was is called the loss. They found that for any one of three things, the number of parameters, the amount of text, or the amount of computation, if the other two weren't holding it back, multiplying that thing by ten reduced the loss by roughly the same proportion. From the smallest model to the largest was a span of six or seven orders of magnitude. The line was smooth all the way.

In the title, that is a law. There is a section in the appendix devoted to caveats. The authors say they have no firm theoretical understanding of any of the scaling laws they propose. They call the laws for model size and computation especially mysterious. The discussion says smooth improvements in loss might conceal changes in ability that look qualitative. It then cites the title of a paper by a physicist: “More Is Different.”

Two years later, in March 2022, Jordan Hoffmann and colleagues at DeepMind trained more than four hundred models, from seventy million parameters to sixteen billion, on five to five hundred billion tokens. A token is a small piece of text as the model reads it. An English word is often one or two tokens. They arrived at a different balance: double the size of the model and you should double the number of training tokens as well. They used that balance to train Chinchilla, a seventy-billion-parameter model. It used the same amount of computation as Gopher, which had 280 billion parameters, but got about four times as much text and outscored Gopher and GPT-3 on a set of tasks. Kaplan's balance favored a bigger model and less text. Hoffmann's had the two grow together.

5. A Hundred and Seventy-Five Billion

On May 28, 2020, a paper with thirty-one names on it went online. The first was Tom Brown's. The title was “Language Models are Few-Shot Learners.” The largest model in the paper was GPT-3. It had 175 billion parameters and read three hundred billion tokens during training.

Few-shot refers to the way it is used. You write what you want it to do, give it a few worked examples, then leave a blank for it to continue. No number in the model itself changes. The piece of text you give it is called a prompt. Give it instructions with no examples and it's zero-shot. Give it one and it's one-shot.

The paper describes an experiment. The participants, in the United States, were workers selected for good performance on Mechanical Turk, the platform from essay sixteen. Their pay was set at about twelve dollars an hour. They read news pieces of around two hundred words and were asked to tell which were written by reporters and which by a model. For the output of the model with 175 billion parameters, they got it right about fifty-two percent of the time, close to tossing a coin.

The limitations section lists its faults. A longer piece can repeat itself, contradict itself, or stop hanging together. It often gets ordinary physical facts wrong, for instance whether cheese melts in a refrigerator. The authors say simply scaling up this way of learning by predicting the next word is likely to hit limits.

On June 11 OpenAI opened an interface called the OpenAI API. The description was one line: text in, text out. At first you had to apply, get your use reviewed, then go through another review before going live. The model itself was not released. The reasons given were that it was too big and costly to run, that an interface could bring in revenue, and that keeping it in one place made it easier to manage misuse. What did training cost? Lambda, a company that rents graphics processors, converted the amount of computation to its rental rates at the time. On one card it would have taken 355 years, at a rental cost of about 4.6 million dollars.

6. Eight Versions

On September 8, 2020, The Guardian printed an opinion piece with the headline “A robot wrote this entire article. Are you scared yet, human?” The machine spoke in the first person. It wanted to convince humans they didn't need to fear it. Everything it knew, it said, it had learned by reading the internet.

An editor's note at the end says how the article came about. The editors had written the prompt: a short opinion piece of about five hundred words in simple language explaining why humans need not be afraid of AI. They supplied the first few sentences as well. Liam Porr, an undergraduate studying computing at Berkeley, entered the prompt. GPT-3 returned eight different pieces. The editors ran none of the eight as it was. They picked the best parts from each one, cut some sentences and paragraphs, moved things around, and made this article out of them.

It said it had read the internet. The GPT-3 paper lists five sources. The largest was web pages collected by a project called Common Crawl. Gil Elbaz started the nonprofit project in 2007. It collects only web pages open to people who haven't logged in, and the results are publicly downloadable. Being able to download something and having permission to use it for training are two different things. GPT-3 used forty-one months of the data, from 2016 to 2019, about forty-five terabytes of compressed plaintext, reduced to about 570 gigabytes after filtering. The other four sources were WebText2, two collections of books identified only as Books1 and Books2, and Wikipedia. The training run drew from the five in fixed proportions. Common Crawl accounted for sixty percent of the draw; Wikipedia, three percent. In terms of the amount drawn, that was less than half a pass through the Common Crawl text and about three passes through Wikipedia.

WebText2 was an enlarged version of GPT-2's WebText. WebText drew on outside links people had posted on Reddit. When a post had at least three karma, the team collected the main text of the page it linked to. The GPT-2 paper says this used the judgment of Reddit users to select pages other people found interesting or useful. By December 2017 more than eight million pages had made the cut. It deliberately left out Wikipedia entries to avoid overlap with common tests.

7. Emergence

In 2021 the numbers got bigger. In January Google's Switch Transformer reported 1.6 trillion parameters. The model was divided into many experts, with each word sent to just one to be processed. In April came Huawei's PanGu, two hundred billion. In June came version two of Wudao from the Beijing Academy of Artificial Intelligence, 1.75 trillion. In October, Microsoft and Nvidia, 530 billion. In December, DeepMind's Gopher at 280 billion, and Baidu at 260 billion. In April 2022, Google's PaLM at 540 billion.

In August 1972, the physicist Philip Anderson published “More Is Different” in Science. The ability to reduce everything to simple fundamental laws, he wrote, didn't mean you could start from those laws and reconstruct the universe. Go up a level in complexity and new properties appear. It takes new ways of studying them. He was talking about levels between fields like physics, chemistry, and biology. Fifty years later Kaplan and his colleagues borrowed the title for language models.

On February 9, 2022, OpenAI's Ilya Sutskever wrote online that perhaps large neural networks today were slightly conscious. Three days later Yann LeCun, whom we met in essay fourteen, answered with one word: no. He said even if you made slightly very slight and large very large, he still didn't agree.

In June that year Jason Wei and colleagues at Google put out “Emergent Abilities of Large Language Models.” Their definition was that an ability is emergent if it isn't present in smaller models but is present in larger ones. They put scores from many models and many tasks next to one another. On some questions, the scores stayed around chance when the models were small. Past a certain size they went up sharply.

In April 2023, Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo at Stanford published a paper with a question as its title: “Are Emergent Abilities of Large Language Models a Mirage?” Sometimes what looked sudden, they said, came from the scoring. If a sum with several digits has to be right in every digit to count, small models keep getting zero and start scoring suddenly at a certain size. Score how many digits they are off by instead and the line climbs steadily.

8. Large

The unicorn blog post in February 2019 already used large language models several times. In Chinese, the names in 2021 started out as super-large-scale pretrained models and super-large-scale intelligent models. Over the year they got shorter, until it was just large models.

In August that year a group of researchers at Stanford published “On the Opportunities and Risks of Foundation Models,” giving the class a different name. They said they chose foundation model to mark a central role that was critical yet incomplete. A foundation is what other systems are built on. It's also a thing you haven't finished building. What stayed when the name got around was the base you build on.

On March 14, 2023, OpenAI released GPT-4. The technical report said that in view of competition and the safety implications of these large models, it would no longer give details of the architecture, including model size, hardware, training computation, or dataset composition. The numbers had been public on the way from one and a half billion to 175 billion. Here they stopped appearing.

Where did the words come from? Someone looked that year. At the end of 2020 an open-source group called EleutherAI had assembled a corpus called The Pile. One part, Books3, drew from a collection of books on a site called Bibliotik. On September 25, 2023, Alex Reisner of The Atlantic made a searchable tool from the 183,000 books in that collection: enter an author's name and see which books of theirs are there. Two days later the Authors Guild wrote a guide for writers. What do you do when you've just found out your book was used to train AI?

That year people who wrote books went to court in groups. On June 28 novelists Paul Tremblay and Mona Awad sued OpenAI, alleging that their books had been copied without permission to train its models. On July 7 Sarah Silverman and two other authors filed separate suits against OpenAI and Meta. On September 19, the Authors Guild and a group of writers including George R. R. Martin, John Grisham, Jodi Picoult, and Jonathan Franzen sued OpenAI in New York. On December 27, The New York Times sued Microsoft and OpenAI.

There was movement on the side of the people posting links and voting on them, too. In April Reddit said it would charge for large-scale access to its data through its interface. Its chief executive, Steve Huffman, told The New York Times that Reddit's corpus was genuinely valuable, and there was no reason to give it away to some of the world's biggest companies. The same month Stack Overflow's chief executive, Prashanth Chandrasekar, said the questions and answers written by the community were raw material for large language models and the platform should be compensated. In August OpenAI announced its web crawler, GPTBot, and described how sites could refuse it in robots.txt, the little file that tells crawlers where not to go. A number of publishers subsequently shut it out.

The GPT-3 paper ends with acknowledgments. The last people it thanks are the millions who made the content that trained the model and the people who indexed it or voted on it. The author list has thirty-one names. The people who wrote the web pages, edited the Wikipedia entries, clicked an upvote on Reddit, or wrote the books are not among them.

The number of parameters rose from one and a half billion to 175 billion, and then it stopped being given. These people's words went in. The people stayed outside. This series calls the part that cannot go in, however big you make the thing, the remainder.

The machines are still being built. Each generation says again where the boundary lies.