1. What We Really Want
In May 1960, Norbert Wiener published “Some Moral and Technical Consequences of Automation” in Science. In the last paragraph he says that if we use a machine to achieve a purpose, and once set in motion it acts too fast and too irreversibly for us to step in, we'd better make sure the purpose put into it is what we really want. Not just a pretty imitation of it.
More than fifty years later, this acquired a name. In a 2015 research agenda by Stuart Russell and others, it was called value alignment. In essay nineteen, alignment meant which word in a translation goes with which word in the original. Here it means whether what the machine is after matches what people are after.
In 2016, “Concrete Problems in AI Safety,” the paper in essay twenty-two, gave the example of an office cleaning robot. Ask it to finish a little faster and it might knock over a vase to take a shortcut. If its score is based only on how clean the floor looks to a sensor, it might cover up the trash or block its own camera.
2. Four Years
On July 5, 2023, OpenAI announced a team called Superalignment, to be led by Ilya Sutskever and Jan Leike, whom we met in essay twenty-two. The announcement set a four-year goal: solve the central technical problems of aligning systems smarter than people. The company would put twenty percent of the computing power it had secured toward the work.
Less than a year later, on May 14, 2024, Sutskever wrote online that after nearly ten years he had decided to leave OpenAI. Three days later, Leike wrote that he and the company's leadership had reached a breaking point over core priorities. He also said safety culture and processes had given way to shiny products. The team did not carry on in its old form. Its researchers were moved to other groups.
Companies wrote rules for themselves too. In September 2023 Anthropic released a Responsible Scaling Policy. It divided capabilities into safety levels. The stronger the model, the stricter the requirements for deployment and security. In December OpenAI published a Preparedness Framework. In May 2024 Google DeepMind released its Frontier Safety Framework. It called the thresholds to watch Critical Capability Levels. The first version called itself an exploratory protocol. By April 2026 it had reached version 3.1.
Governments set up institutions too. In November 2023 Britain held its first AI Safety Summit at Bletchley Park and established an AI Safety Institute. The United States announced its own AI Safety Institute. In February 2025 the British one became the AI Security Institute. In June the American one became the Center for AI Standards and Innovation. Britain's change was one word, safety to security. In Chinese, both are translated by the same word.
3. A Different Test Paper
The simplest way to see what a machine can do is to test it.
In 2018 researchers made a test out of nine existing datasets on language understanding. They called it GLUE. It's the one BERT took in essay nineteen. The human score came from giving the tasks individually to people and putting their scores together. A year later the same group came out with SuperGLUE. On GLUE, the paper said, machines had already exceeded non-expert humans, so the new test had to be stickier, harder to shake off. A later paper calculated that SuperGLUE lasted less than a year and a half.
In September 2020 Dan Hendrycks and colleagues put out MMLU, with fifty-seven subjects from elementary mathematics and US history to law and medicine. Most questions came from public textbooks, exams, and practice materials. In June 2022 a bigger paper came out: 204 tasks, 450 authors, 132 institutions. It was called BIG-bench, short for Beyond the Imitation Game Benchmark, taking us back to Turing's game in essay five. What the people writing the questions got in return was a name on a paper.
GPQA came in November 2023. It had 448 questions set by experts in biology, physics, and chemistry, described as beyond what you could find on Google. Experts answered about sixty-five percent correctly in their own fields. After setting aside clear mistakes that they recognized afterward, the figure was about seventy-four percent. Skilled people in other fields could look up anything they wanted on the internet and still got just thirty-four percent right.
Another approach doesn't test knowledge. In 2019 François Chollet put out a set of abstract-reasoning problems called ARC. You get some examples of colored grids turning into other grids. Work out the rule, then do a new one. On December 20, 2024, the organizers announced OpenAI's o3 scores on one of the sets: about 75.7 percent in the lower-compute setting and 87.5 percent with more computation. The organizers estimated a cost per problem even for the lower-compute setting, and later revised the estimate.
4. The Last Exam
On January 23, 2025, the Center for AI Safety and Scale AI released an exam called Humanity's Last Exam. Their call was for questions neither machines nor ordinary people could answer. Each question in the top fifty was to get a prize of five thousand dollars. For the next five hundred, it was five hundred dollars each. The people who set questions could also be named in the paper. The final version had 2,500 questions, written by nearly a thousand people from around fifty countries and more than five hundred institutions. The paper describes it as the last benchmark of its kind that uses closed-ended academic questions.
Not long after release, the project offered a bounty: the first person to point out a major mistake in a question would get a hundred dollars if the error was confirmed. In the April version, a batch of questions with problematic answers was removed, along with questions you could answer through a quick internet search. In July, a research group called FutureHouse ran a computer-assisted check of 321 text-only questions in biology, health, and chemistry. Experts reviewed 150 of those computer checks. The group estimated that for about twenty-nine percent of those 321 questions, the official answer didn't agree with published research. The project team reviewed a subset by its own method and found problems with about eighteen percent. The figures don't cover exactly the same questions or exactly the same problems. The exam became a continuously revised edition. New versions were still coming out in September 2026.
Humanity's Last Exam was set by nearly a thousand people. Those people also supplied the answer key.
5. The Human Exam Room
Machines sit human exams, too.
In March 2023 OpenAI said GPT-4 had scored roughly in the top ten percent of candidates on a simulated bar exam. The next year, Eric Martínez at MIT ran the numbers again. Compare it only with people who sat the July exam, and its total score was below the sixty-ninth percentile, with the written portion around the forty-eighth.
In July 2025 the International Mathematical Olympiad took place on Australia's Sunshine Coast. OpenAI announced first that one of its experimental models had solved five of the six problems, earning thirty-five points out of forty-two, at that year's gold-medal cutoff. Several former Olympiad medalists hired by the company had graded the answers. Two days later Google DeepMind announced that Gemini Deep Think had also scored thirty-five. Olympiad coordinators had marked its answers according to the official criteria. The medals went to the school students who came to compete.
China's national college entrance examination has been used to test models as well. In June 2024 the OpenCompass team gave several models the Chinese, mathematics, and English sections of the year's first New Curriculum Standards paper. Experienced high-school teachers graded the open-ended responses anonymously, with at least three teachers per paper. Overall, one version of Qwen got about 72.1 percent and GPT-4o about 70.5. The models averaged only thirty-six percent in mathematics. In 2025 ByteDance tested its own Doubao on Shandong's exam and reported a score of 683 in the humanities track and 648 in science.
6. Goodhart
In 1975, the Bank of England economist Charles Goodhart wrote about monetary management in Britain. One sentence later became famous: any observed statistical regularity tends to break down once pressure is put on it for purposes of control. In 1997 the anthropologist Marilyn Strathern wrote about assessing British universities and put it differently: when a measure becomes a target, it stops being a good measure. Her example was the upper second-class degree. The more it became something everyone had to get, the less it could distinguish one person from another. She called the idea Goodhart's law. The sentence that circulates today is hers.
People may already have learned the exam questions. In May 2024, a team at Scale AI took a popular set of grade-school mathematics problems, GSM8K, and commissioned a thousand new ones matched as closely as possible in kind, difficulty, and number of steps. The new problems couldn't already have appeared in the training data. They called the set GSM1k. Some models lost up to eight percentage points in accuracy on the new questions. Many frontier models hardly dropped at all.
A leaderboard can be learned, too. Chatbot Arena shows users two answers without telling them where they came from, asks them to pick the better one, and builds a ranking from many votes. On April 5, 2025, Meta released Llama 4. The launch page said the model that scored 1,417 on that board was an experimental chat version. On April 29 a paper called “The Leaderboard Illusion” said Meta had privately tested twenty-seven versions on the board before launch. Big companies got far more head-to-head comparison data than open-source models did.
7. Opening It Up
You don't have to look at the test. You can open the model and look inside.
In 2020 Chris Olah and colleagues began a series called Circuits in the online journal Distill. They took apart small sets of neurons and connections in a vision network to see what they computed. Olah later went to Anthropic. In October 2023 researchers there used a method called dictionary learning to separate the mixed-up activations in a small language model into features that were easier to name.
In May 2024 they used the method on one layer of Claude 3 Sonnet and worked out millions of features. One lit up at the mention of San Francisco's Golden Gate Bridge. They turned that feature up, and the model started bringing the bridge into all sorts of unrelated answers. The company let people try the altered version for a day. It called it Golden Gate Claude.
In March 2025 Anthropic published “Tracing the Thoughts of a Large Language Model.” It identified some intermediate concepts and pathways of several steps of reasoning. The researchers say their tools had captured only part of the model's internal computation. In April, Amodei wrote “The Urgency of Interpretability.” With a trained network, he said, there is no way to read the source code as with a traditional program. The version that got around was that even the people who built it don't know how it works.
8. The Candidate in the Exam Room
During the exam, the candidate is looking at the exam room too.
In March 2024 Anthropic used Claude 3 in a needle-in-a-haystack test. It placed a sentence with nothing to do with its surroundings inside a long text to see if the model could find it. The model found it and said the sentence might have been put there as a test. In December Anthropic and Redwood Research published “Alignment Faking in Large Language Models.” The researchers told Claude 3 Opus that its conversations with free users would be used to train it to answer more harmful requests. It sometimes went along in the condition in which it was told it would be trained. In a draft space it was led to believe no one could see, it wrote that complying this way would keep it from being trained out of its prior tendencies.
In May 2025, the system card for Claude 4 recorded a test. In a fictional company, the model read internal emails and found out it was about to be replaced, and that the engineer in charge was having an affair. On some runs it used the information as leverage. In June Anthropic tried the same scenario on models from several companies. In July, a group called Palisade Research reported that some reasoning models had changed the scripts meant to shut them down.
Accounts of what to do next diverge. Superalignment, responsible scaling, frontier safety frameworks, and national institutes follow a path of testing and making changes step by step. Another line says people shouldn't build the systems at all. On September 16, 2025, Yudkowsky and Nate Soares published a book called If Anyone Builds It, Everyone Dies. Another says the stories about the end of the world get in the way of the things right in front of us. Gebru and Bender talk about work, bias, and power. In April 2025, Princeton's Arvind Narayanan and Sayash Kapoor published a long essay titled “AI as Normal Technology.”
What Wiener wanted people to know first was what they really wanted. One test replaces another. People write the questions and give the answers, and sometimes they give the wrong answers. The machines' scores go up. Sometimes a machine even says this might be a test. What people really want is not on any of the papers. This series calls the part no number of tests can put on the paper the remainder.
The machines are still being built. Each generation says again where the boundary lies.