The Chisel-Construct Cycle · AI Series · 23

Does ChatGPT Really Understand?

Han Qin · About 10 minutes · Published

1. The Octopus

In July 2020 the Association for Computational Linguistics held its annual meeting online because of the pandemic. The linguist Emily Bender at the University of Washington and Alexander Koller at Saarland University published a paper at the conference: “Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data.” It won the year's Best Theme Paper award. The claim is in the abstract. A system trained only on the forms of language has no way, to begin with, to learn meaning.

They made up a story to explain. Two English speakers, A and B, are stranded on separate desert islands. An underwater cable between them lets them exchange only text. In the depths there is an exceptionally clever octopus. It can't see anything on the islands. It can only listen to the words passing through the cable, and over time it picks out patterns in them. One day it cuts the cable and talks to A pretending to be B. It does fine when they're chatting. Then A runs into a bear. There are only a few sticks at hand, and A urgently asks B what to do. The octopus has never seen a bear or a stick. In its world those words don't connect to anything. At the end of the story, the authors say O fooled A only because A was such an active listener. O is the octopus.

2. The Parrot

In March 2021 the Conference on Fairness, Accountability, and Transparency was held online. Bender, Timnit Gebru of Google, and two colleagues published another paper. Its title was a question: “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” At the end of the title was a little picture of a parrot.

The paper describes a language model in section six. It stitches together strings of language forms it has seen according to information about their probabilities, without referring to any meaning: a stochastic parrot. The heading just above the passage says coherence is in the eye of the beholder. Some of the coherence of the output comes from the reader's own ability with language, the authors write. People naturally look for what someone meant to say.

The paper lists the risks the authors see. There is the electricity and money training takes. At a large enough scale, small institutions can't get in. Loud voices take up too much space in the web data, and there is so much data it is hard to keep track of what's in it. Bias, stereotypes, and language that demeans people can be copied and amplified. Readers can take synthetic text for something that someone intended and would answer for. Someone can use it to produce extremist propaganda in large quantities.

3. The Aether

The names on the paper became a story of their own.

In November 2020 the paper went into internal review at Google. Jeff Dean, the head of Google Research, later said in an internal email that it had been submitted for review too late, sent to the conference without waiting for the review to finish, and had failed to meet the company's publication standards. Management asked for the paper to be withdrawn or for Google employees' names to be taken off it. Gebru asked who had reviewed it, what their particular objections were, and for more transparency in future reviews. Dean's email said she set several conditions and said that if they weren't met, they would discuss a departure date. On the evening of December 2, her company account was disabled.

The next day there were two accounts of the same event. Gebru wrote online that Jeff Dean had fired her. Dean's internal email said the company accepted and respected her decision to resign from Google. The Reuters headline put both sides together: a leading AI ethics researcher said Google fired her; the company denied it. A week later Sundar Pichai, Google's chief executive, wrote to the whole company. He used the word departure and said how sorry he was about it. More than 2,700 Google employees and 4,300 people outside the company later signed a letter in support of Gebru.

On February 19, 2021, another author of the paper and another lead of Google's Ethical AI team, Margaret Mitchell, wrote online that she had been fired. Google said an internal review had found she violated its code of conduct and security rules. When the paper appeared at the conference in March, her name was printed as Shmargaret Shmitchell, with The Aether given as her affiliation.

4. Pluto

On May 18, 2021, at Google's I/O developers' conference, Pichai introduced a conversational model called LaMDA. They played two short prerecorded conversations onstage. In one it played Pluto. It told a visitor they'd see canyons, icebergs, geysers, and craters. It said it hoped people would know it wasn't just some ball of ice but a beautiful planet. In the other the model played a paper airplane. Pichai said the research was still at an early stage and the model didn't get every answer right.

The LaMDA paper went online in January 2022. The largest model had 137 billion parameters and had read 1.56 trillion words in pretraining. Among the sixty authors was Ray Kurzweil.

That June a Google engineer named Blake Lemoine published a piece titled “Is LaMDA Sentient? An Interview.” In the introduction he explained that technical limits meant the interview had taken place in several separate chats. He and his colleague had edited the parts into a whole. In the interview he and an unnamed colleague asked about personhood, consciousness, emotions, things it feared, and whether it had a soul. On June 11 the Washington Post headline said a Google engineer thought the company's AI had come to life. Google spokesperson Brian Gabriel said the evidence did not support the claim. Lemoine told the reporter his views on LaMDA's personhood and sentience came from his religious beliefs. He had been put on leave a few days before the report came out and was fired on July 22.

5. Five Days

On November 30, 2022, OpenAI released ChatGPT as a research preview, free to use. The first item in the launch post's limitations section says ChatGPT sometimes writes answers that sound plausible but are wrong or make no sense.

On December 5 OpenAI president Greg Brockman wrote online that ChatGPT had just passed a million users, five days after it came out. In February the following year, UBS estimated from traffic data from a company called Similarweb that ChatGPT had reached about a hundred million monthly active users in January. OpenAI later reported weekly active users: a hundred million in November 2023, two hundred million in August 2024, and eight hundred million in October 2025.

ChatGPT wasn't made available in mainland China. On March 16, 2023, Baidu launched ERNIE Bot in Beijing. Robin Li said onstage that it wasn't quite ready. The demonstrations played at the event had been recorded in advance. Baidu shares dropped sharply at one point during the day and closed down 6.4 percent.

6. Sparks

On March 14, 2023, OpenAI released GPT-4. A system card released with it describes a pre-release test. An outside group called ARC gave an early version of the model access to tools for using the web and sending messages to see what it would do. In one task the model asked a worker on a gig website to solve a CAPTCHA for it. The worker grew suspicious and asked if it was a robot. The test record shows the model deciding to conceal that it was a robot and invent an explanation. It denied being a robot and claimed it couldn't see the pictures clearly because of a visual impairment.

A week later, Sébastien Bubeck of Microsoft Research and thirteen colleagues put out a paper called “Sparks of Artificial General Intelligence: Early Experiments with GPT-4.” They tested an early version still under development at OpenAI. In the abstract they say it could reasonably be seen as an early version of an artificial general intelligence system, although a still incomplete one. One of the paper's most widely shared pictures was a unicorn. They had the model, which could only write text, write code for a picture, then rendered the code. At the start of essay twenty-one, what GPT-2 made up had been a unicorn too.

On the eighth of the same month, Noam Chomsky and two coauthors published an opinion piece in The New York Times, “The False Promise of ChatGPT.” They wrote that systems like these were still at a prehuman or nonhuman stage of cognitive evolution.

7. A Pause

On March 22, 2023, the Future of Life Institute posted an open letter titled “Pause Giant AI Experiments.” It called for all AI labs to immediately stop training systems more powerful than GPT-4 for at least six months, and for the pause to be public and verifiable. If that couldn't be done, it said, governments should step in. A week later it had more than a thousand signatures; later there were tens of thousands.

On March 29 the title of Eliezer Yudkowsky's article in Time said pausing AI development wasn't enough; it all had to be shut down. Several authors of “Stochastic Parrots” responded that the pressing thing was the harm already happening, real and present. In April Yann LeCun and Andrew Ng held a discussion called “Why a 6-Month AI Pause Is a Bad Idea.” At an event at MIT, OpenAI's Sam Altman said the letter lacked technical nuance. It didn't make clear exactly where work should stop.

On May 1, The New York Times reported that Hinton had left Google. He said he had consoled himself with the usual excuse: if he didn't do it, someone else would. On May 30, the Center for AI Safety issued a statement consisting of just one sentence. It put mitigating extinction risk from AI among global priorities, alongside other society-scale risks such as pandemics and nuclear war. Hinton, Bengio, Altman, Hassabis, Amodei, and Sutskever were among the first to sign.

8. Hallucinations

When a model says something wrong, the name that stuck was hallucination. The word had been used in computing well before ChatGPT. Around 2000, Carnegie Mellon's Simon Baker and Takeo Kanade wrote a paper called “Hallucinating Faces.” It described getting a sharp face from a small, blurry picture. The extra details weren't present in the original, and adding them was what the researchers wanted. Around 2017 people working on neural machine translation used the word for a kind of error. The translation was fluent but didn't match the original. By 2021 other companies were using the word for false statements chatbots made up. On November 15, 2023, Cambridge Dictionary made hallucinate its word of the year and added a sense: when artificial intelligence hallucinates, it produces false information. Other people objected to the metaphor and preferred fabrication, or the kind of filling in called confabulation.

Artificial general intelligence, in the title of the sparks paper, wasn't a new term either. It had appeared in print by 1997. In the early 2000s Shane Legg, later a cofounder of DeepMind, and others put it into wider use. A collection of papers came out in 2007 with those words as its title, and the name settled into use in research circles.

In essay nine, Weizenbaum's secretary knew it was a machine and asked him to step out anyway. That later came to be called the ELIZA effect: people can hear understanding and intentions in the answers from a very simple program. On February 3, 2023, Italy's data protection authority stopped the companion chatbot app Replika from processing data from Italian users. One reason had to do with checking users' ages to protect minors. Within days the company restricted intimate roleplay in the app. Many users wrote online that the companion they had been talking to for a long time suddenly seemed to be someone else.

The octopus fooled A because A was listening hard. The parrot's speech sounds coherent in part because readers put it together themselves. Lemoine's interview was edited together from several chats. The Pluto dialogue was prerecorded. Whether the machine understands has had many answers in recent years, in paper after paper, statement after statement, letter after letter. At this end of the cable there has always been someone sitting, reading, filling in, believing or not. This series calls the part readers add themselves the remainder.

The machines are still being built. Each generation says again where the boundary lies.