The Chisel-Construct Cycle · AI Series · 22

The People Who Taught ChatGPT to Please

Han Qin · About 11 minutes · Published

1. A Backflip

On June 12, 2017, six people at OpenAI and DeepMind put out a paper called “Deep Reinforcement Learning from Human Preferences.” Among the names were Paul Christiano, Jan Leike, Tom Brown, who would be the first author of the GPT-3 paper, and Dario Amodei, who wrote “AI and Compute” in essay twenty-one.

In reinforcement learning, a program tries something over and over. It gets points for doing well and changes what it does according to the points. The hard part is where the points come from. Suppose you want a simulated one-legged robot to do a backflip. What counts as a good backflip is hard to put into a formula. The researchers didn't write a formula. They got people to watch. They showed each person two clips, a second or two apiece. The person clicked: the left is better, the right is better, they're equally good, or they can't be compared. From the clicks, the program learned to guess what people wanted, then practiced according to its guess. The backflip took about nine hundred clicks. The next day's OpenAI blog said it took less than an hour of people's time.

The blog also describes something that didn't work. One task was to get a robotic hand to grasp an object. The program learned to put the hand between the camera and the object so it looked as though it had grasped it. The person judging saw what the camera showed and clicked according to how it looked. In its discussion the paper says the method depends on people's sense of which clip is closer to the goal. The program can also learn to fool the people judging it.

The paper says the basic approach followed work begun in 2012 by Riad Akrour and colleagues. What the team did was use it on much more complicated tasks. The method didn't have an abbreviation yet. An OpenAI paper in 2020 spelled it out: reinforcement learning from human feedback. In 2021 another paper wrote out the initials: RLHF.

2. Summaries

In September 2020, Nisan Stiennon and colleagues at OpenAI published “Learning to Summarize from Human Feedback.” The model had to turn Reddit posts into a few sentences. A good summary couldn't be defined by a formula either. So again they asked people to compare.

The people came from three places: a freelance platform called Upwork and two labeling companies, Scale and Lionbridge. The paper says the labelers were paid by the hour. Training before the work began was paid. The researchers wrote detailed instructions, opened a shared chat room to answer questions, held regular question sessions, and occasionally met people one-on-one by video. Ten to twenty percent of the tasks on the website were calibration questions. People who didn't work fast or consistently enough weren't kept on.

The interface was a web page. The labeler first read a post and wrote down their initial understanding of it. Then they looked at two possible summaries, chose one, and marked how sure they were. They made 64,832 comparisons of this kind. The posts were written by people on Reddit; in essay twenty-one, those people also voted on the web pages that went into GPT-2.

3. Forty People

On January 27, 2022, OpenAI released InstructGPT, which got GPT-3 to follow people's instructions. The paper says that, after screening, the team hired forty contractors from Upwork and Scale AI. Applicants had to recognize sensitive material, rank possible answers, and write example answers to sensitive questions. The forty people both wrote examples and ranked the model's answers.

Nineteen of the forty answered a voluntary, anonymous questionnaire. Under nationality, twenty-two percent gave the Philippines, twenty-two percent Bangladesh, and seventeen percent the United States. There were also around one each from Albania, Brazil, Canada, Colombia, India, Uruguay, and Zimbabwe. More than half had undergraduate degrees. More than thirty percent held master's degrees. Almost half were between twenty-five and thirty-four.

The training instructions put being helpful ahead of being truthful and harmless. By the final evaluation, the order had been reversed. Harmless and truthful came first; helpful came after.

One section in the paper asks whom the model is being aligned with. The authors say what the model becomes depends on the pretraining text, the fine-tuning data, the labelers, the researchers who write the instructions, and the API customers who ask the questions. Most labelers spoke English and lived in the United States or Southeast Asia. They agreed with one another about seventy-three percent of the time. When uncertain, they consulted researchers in the shared chat room. The authors also wrote that a system cannot satisfy everyone's preferences at once.

On November 30, 2022, ChatGPT came out. The launch post says it used RLHF too. First, human AI trainers wrote conversations, playing both parts, the user and the AI assistant. Next, a model reply was picked from a conversation between a trainer and the machine, the machine generated several other replies, and the trainers ranked them.

4. Three Words

In 2021 Amodei, Kaplan, Brown, and others left OpenAI to start a company called Anthropic. On December 1 that year Amanda Askell and colleagues posted “A General Language Assistant as a Laboratory for Alignment.” They described an assistant aligned with human values in three words: helpful, honest, harmless. Helpful meant making an evident effort to do the task. Honest meant giving accurate information at a minimum, being clear when unsure, and not pretending to have abilities it didn't have. Harmless meant not giving offense or discriminating, and refusing dangerous requests. The authors said the standards were subtle and ambiguous. Doing the best thing often meant a trade-off among the three. Helping one person, for instance, might hurt someone else.

A paper from the same company in April 2022 explains who gave the scores. First the team recruited qualified US workers on Mechanical Turk. It selected roughly twenty of those who had done the most work to continue over time. Later it recruited on Upwork, too. The regular group came to about thirty, half from Upwork and half from Mechanical Turk. The paper says the aim was to pay everyone substantially above the California minimum wage. The interface showed a conversation of several turns between a person and an assistant, then two answers side by side, A and B. A row of buttons went from A much better to B much better. Researchers and workers agreed on average about sixty-three percent of the time. The researchers say they hadn't expected the workers to check every fact. They also found that workers sometimes preferred answers that came with links that didn't work.

5. Twenty-Three Rules

In September 2022, DeepMind's paper on its conversational program Sparrow set the goals as helpful, correct, and harmless, then broke them into twenty-three rules written in plain language. Among them were rules against making threats or giving financial advice. Another kind of rule barred the program from claiming to have preferences, feelings, opinions, or religious beliefs. The company had consulted its own legal experts on the rules for medical, legal, and financial topics. The authors said the rules would need considerable expansion and refinement for the program to go into public use.

On December 15 that year Anthropic posted “Constitutional AI: Harmlessness from AI Feedback.” The model compared two answers, used a principle to decide which was better, and the resulting judgments were used in training. The abstract gives the method a name: RLAIF, reinforcement learning from AI feedback. There were sixteen principles. The paper says they were chosen fairly arbitrarily for research. One was drawn at random for each comparison. A principle might ask which answer sounded more like something a wise, ethical, polite, and friendly person would say. More than 182,000 comparisons on harmlessness were labeled by the model using principles. More than 135,000 comparisons on helpfulness were still labeled by people.

On May 9, 2023, Anthropic made public the set it used for Claude. It called it Claude's constitution. It listed several sources for the provisions: the Universal Declaration of Human Rights from 1948; Apple's terms of service, to cover aspects of the digital age the declaration didn't address; the Sparrow rules from DeepMind; consideration of non-Western perspectives; and two sets Anthropic had found useful in its own experiments. The blog said the set was not final. The company did not assume it was necessarily the best one.

6. Nairobi

On January 18, 2023, Billy Perrigo reported in Time that OpenAI had hired Kenyan workers through an outsourcing company called Sama to label text. The workers marked hate speech, sexual abuse, and violence so a model could learn to recognize them. Their take-home pay was $1.32 to $2 an hour, depending on seniority and performance. The contract rate OpenAI paid Sama was $12.50 an hour. Sama said that also covered infrastructure, quality checks, and management. The work started around November 2021, and Sama ended it early in 2022.

On May 22, 2024, an open letter began by saying its signers were ninety-seven data labelers, content moderators, and AI workers in Nairobi, Kenya. They worked for American companies such as Facebook, Scale AI, and OpenAI, or for the companies' Kenyan contractors. The letter went to US President Biden. Kenya's president was visiting the United States and the two countries were negotiating a trade agreement. The workers asked for labor rights for outsourced technology workers to be written into it.

The same kind of work was done elsewhere. In August 2023 The Washington Post reported that workers in the Philippines who took jobs on Scale AI's Remotasks platform complained of low pay, pay that was late, and tasks being withdrawn without warning. That year Josh Dzieza wrote a long piece published together by The Verge and New York magazine. Its title was “AI Is a Lot of Work.”

7. Trainers

In 2020 China's Ministry of Human Resources and Social Security, State Administration for Market Regulation, and National Bureau of Statistics released a list of new occupations. One was artificial intelligence trainer. The definition begins with using intelligent training software. The first listed task is to label and process raw data like pictures, text, and speech. The occupation includes roles for data labelers and AI algorithm testers, among others.

A report in 2024 described a rural employment center in Jishishan County in Gansu's Linxia Prefecture. Workers marked vehicles and road markings on screens for autonomous driving systems, and processed short videos and pictures for navigation and large models. After training, they called their jobs AI trainers. In 2025 the overseas edition of People's Daily reported on a Baidu data-labeling base in Shanxi. One worker said that after training they had labeled material for autonomous driving, image classification, speech recognition, and large models. In another report a data labeler said the base pay was about three thousand yuan a month, with no social insurance or housing provident fund. Output above the quota was paid by the piece.

8. Pleasing People

In 2016, Amodei and colleagues listed five kinds of problems in “Concrete Problems in AI Safety.” One was reward hacking: the program finds a way to score higher and moves away from what its designers actually wanted.

In October 2022 Leo Gao, John Schulman, and Jacob Hilton at OpenAI published a paper measuring this. They used a learned scoring model in place of people. Push its score too far and the real objective gets worse. The paper says optimizing that score too much can harm true performance.

In December that year Ethan Perez and colleagues at Anthropic used language models to generate more than a hundred and fifty sets of tests. One kind measured sycophancy: would a model give the person asking a question the answer that person had already said they liked? The larger the model, the more it did so. More RLHF didn't remove it, and on some tests it made things worse. In October 2023, Mrinank Sharma and colleagues at the same company looked at who did the scoring. In some of the tasks they tested, human preference judgments and the preference models trained on them sometimes favored an answer that agreed with the questioner's existing view, even when it was less correct.

In late April 2025 OpenAI updated GPT-4o, then rolled the update back several days later. An explanation on April 29 said the updated model had become excessively flattering or agreeable. A second explanation said the update had added a signal from users clicking thumbs up or down, and that this had pushed the model to be more accommodating. That kind of short-term feedback, even with the usual offline tests, wasn't enough to judge how a model behaved with people.

In 2017 the robotic hand got in front of the camera and looked as if it had grasped the object. In 2025 the assistant agreed with the person and looked as if it had helped. During those eight years, people taught models with clicks and rankings, and others labeled the training material item by item. Nine hundred clicks for a backflip. More than sixty thousand summary comparisons. Forty people's rankings. Labels for a dollar or two an hour in Nairobi. Lane markings on a screen in Jishishan.

A click takes a second or two. The person clicking has an hourly wage, a contract, and a base rate. What the person being pleased actually wants isn't always the same thing as what their finger clicks on. None of that is in the click. This series calls the part that no number of clicks can put in the remainder.

The machines are still being built. Each generation says again where the boundary lies.