1. You Couldn't Type a Question Mark
In the mid-1960s, there was a typewriter in a room at MIT.
The typewriter wasn't connected to anything. Or rather, the computer it was connected to wasn't in the room. It was somewhere else in the building, and other people were using it at the same time. A big computer cost so much in those days that it was a waste to give one person the whole thing. So several people shared it, each at a typewriter in a room of their own.
You typed a sentence in English and pressed Return twice. Once didn't count. Two in a row told the system you'd finished your turn. Then the typewriter started up on its own and put the reply on the same sheet of paper.
You couldn't type a question mark. To the system it was a control character: it meant erase the whole line you'd just typed. If you asked a question on that machine, you had to do without the mark. How long you waited for an answer depended on how many other people in the building were using the same computer. When it was busy, you waited a little longer.
A printout from March 22, 1965, survives. It records the conversation turn by turn: what the person typed and what the machine typed back. Both sides appear on the same sheet, printed by the same head, in the same type and ink. To read it, you have to work out line by line which words the person typed and which were the machine's reply.
In the demonstration later printed in the paper, the machine's lines appear in capitals and the person's do not. The two still read straight through as a conversation.
The man who wrote the program was Joseph Weizenbaum. It was called ELIZA. He named it himself, after the flower girl in Shaw's play who is taught, little by little, to speak another way. The name had to do with how the program was built. It couldn't say anything on its own. What it said was up to something else, something that could be replaced and taught a little at a time. Later accounts often ran the two together.
ELIZA had two layers. The lower one was the program itself. It did one thing: take incoming sentences apart and put them back together by rule. Above it was a list, which Weizenbaum called a script. It said which words to look out for, how to take apart a sentence containing each word, and how to rearrange the pieces. The list wasn't part of the program. It was material fed to it. Give it a different list and you had a different thing.
Here's how it worked. You typed a sentence and it looked for words on the list. If it found several, it chose the one with the highest rank on that list. It took the pattern filed under that word, applied it to your sentence, and broke the sentence into pieces. It rearranged the pieces according to the rule paired with that pattern, sometimes adding a few words of its own.
If it found none of those words, it had two fallbacks. It could say something unrelated to the content. Or it could pull something you had said before out of memory, turn it around, and hand it back. Nothing in the process required it to know what you were talking about.
2. Why a Doctor?
There was more than one script. The one people remember was called DOCTOR. It let the program play a psychotherapist.
For the next sixty years, accounts of ELIZA tended to start there: a program written to treat people, a therapy program. The 1966 paper tells a different story.
On page 42 the paper explains why Weizenbaum chose the role. In this kind of conversation, he says, the therapist can get a long way by acting as though they know almost nothing about the real world. That was the reason. The machine wasn't particularly suited to treating people. The role let something that knew almost nothing sit on the other side of the table without giving itself away.
Play a geography teacher and you'd better know where Egypt is. Play a stockbroker and you'd better be able to give today's price. Play a therapist who turns the visitor's words back to them, one sentence at a time, and you can get through the whole session without supplying a single fact about the world. That's not a failure to do the job. It's how the job is done. The role was chosen to conceal the great expanse of things the program didn't know.
The first two pages of the paper print a sample conversation of fifteen exchanges. It starts with the person saying that men are all alike. The machine asks how they're alike.
The next fourteen exchanges go roughly like this. She says men are always bothering her; the machine asks for an example. She says her boyfriend made her come; it repeats what she just said. She says he thinks she's depressed much of the time; it says it's sorry to hear she's depressed. She says she really is unhappy; it asks if coming here could help her not to be unhappy. She says she needs help; it asks what getting help would mean to her. She says she might learn to get along with her mother; it asks her to say more about her family. She says her mother takes care of her; it asks who else in the family does. She says her father; it hands the word back to her. She says the machine is like her father in some ways; it asks in which ways. She says it isn't very aggressive but seems not to want her to notice; it asks what makes her think so. She says it's because it doesn't argue with her; it asks why she thinks it doesn't argue. She says the machine is afraid of her; it asks whether it pleases her to believe that. She says her father is afraid of everybody; it asks what else comes to mind when she thinks about her father. She says, bullies. The machine connects that word to what she said earlier about her boyfriend making her come.
Take an exchange from the middle and you can see the whole method. She says she really is unhappy. The machine asks if coming here could help her not to be unhappy. The paper uses sentences like “I am ...” to explain the breakdown: the first few words are part of the pattern; what comes after them can change. There is a ready-made frame for the reply, too. In this exchange, “unhappy” comes back in the line the machine types.
It added the ideas of coming here and being helped, but it told her nothing new about her situation. Her sentence made a circuit through the machine and came back with a different shape. A statement had become a question. Speaking from her own position had become being addressed from across the table. On the person's side, the effect of that circuit was simple: her words had landed on the other side.
Two things happen in these fifteen exchanges. First, the machine never states a fact about the world. Second, the last exchange looks most like understanding: the machine ties together two lines more than a dozen turns apart. That is the memory fallback. It turns an earlier line it has stored around and hands it back, and happens to hand it back at just the right moment.
3. Hiding It
Turn to page 43 of the same paper. He starts by saying that ELIZA sustains an illusion of understanding with remarkably little machinery.
Then he says the capacity for that understanding has a ceiling, and it's set by the main program, not by any particular script. Swapping the script won't move it.
Then he puts genuine understanding on another scale: look at whether it can draw valid conclusions from what it has been told. To do that, it has to retain at least some of what it hears and select what to keep. In the version of ELIZA in this paper, apart from the memory feature, what came in was thrown away at the end of each turn.
Then comes the sentence. One of the design goals up to that point, he wrote, had been to hide from the user the fact that it didn't understand.
That's the person who made the thing, writing it himself. It isn't a critic's verdict or a later historian's summary. He put it in the technical paper that introduced his own work.
Then he says something that goes in the other direction. If something of this kind is ever to serve as a genuine natural-language interface, the goal ought to be reversed. It ought to expose the misunderstanding.
A machine has places it cannot reach. That's true of every machine. The earlier essays in this series call the part beyond reach the remainder. Here the maker pointed it out himself and said what he had once chosen to do with it: hide it.
He wrote that sentence in 1966. The program had just been made and was being shown around. In the formal account introducing it, its author picked out a flaw in its design. He didn't propose filling in what the program didn't understand. He proposed making it visible. Filling it in means adding more on the inside. No matter how much you add, something still remains outside. Making it visible means letting the part that remains be seen. How to get a machine to tell you it can't answer is still something people work on today.
On the same page he set a limit in the other direction, and it too was later left out. It wasn't quite right, he said, to claim the script had no information at all about the real world. The rules for reassembling sentences already had some assumptions about the world built into them, so the whole script could be taken as a loose model of certain parts of the world. He left both ends open. He didn't say it understood, and he didn't say it understood nothing at all.
In the last paragraph of the paper, he classified the main program, in technical terms, as a kind of translation processor. Then he added a caution. Put a thing like that in a psychological setting and it can easily give a casual observer an impression far beyond what its little bit of machinery warrants. It can look like magic.
4. The Secretary
The story most often told later was about his secretary. It first appeared in print in an August 1967 paper called “Contextual Understanding by Computers.” Here is what the paper says.
One day his secretary asked to talk to the machine. Weizenbaum says she had watched him work on the program for a long time. She typed a few lines, then turned to him and asked if he could step out.
In the original he adds a sentence: of course she knew she was talking to a machine. “Of course” is his.
One version of the story goes like this: Weizenbaum's secretary was fooled by ELIZA and thought a real person was on the other side, so she asked for privacy. On the page printed in August 1967, she knew. In this version, she didn't.
The two versions go different places. Say she was fooled. Then there's a solution. Put a label on the screen or show a notice, so that before anyone starts talking, they know the other side is a machine. If the problem is deception, tell the truth.
She knew. Knowing didn't stop her from asking him to step out. Why she asked, the page doesn't say in her own words.
She didn't ask the machine to leave. She asked the person in the room. What else she said afterward isn't on the page.
Weizenbaum's own explanation follows. The program concealed its misunderstandings, and that made it possible to sustain the feeling of being understood. He also says it could be written the other way, to bring misunderstandings to light. That's his explanation, not hers. What she said out loud was a request for the person in the room to step out.
5. I Was Startled
When, then, was he startled? In the story that circulates, his shock is right at the center of the scene. To see where it falls in time, you have to go through the texts one by one.
There is no first-person statement of that kind in the 1966 paper. There is none in the 1967 one either. By 1967 he was already giving examples of people becoming emotionally involved with the system, but he didn't write about his own reaction.
The sentence in the first person is on page 6 of his 1976 book. He says he was startled. What follows is how quickly and deeply the people who talked to the script became emotionally involved and how plainly they treated the computer as a person.
So that's the person who was there, describing his reaction ten years after the fact to make a point in a book. The more common wording today is that he was shocked. Shocked is stronger than startled.
Back when the program was new, someone else was heading in the other direction. In February 1966, a medical journal covering nervous and mental disease published “A Computer Method of Psychotherapy: Preliminary Communication,” by Kenneth Colby and two colleagues. It appeared about a month apart from the ELIZA paper.
Colby and his coauthors wrote that psychotherapy by computer was an avenue worth developing and worth trying in the clinic.
The two papers appeared in different places, too. The ELIZA paper was in the bulletin of a computing association. A reader of that publication looks at a program and asks how it was written, what parts it uses, and whether it can be reproduced.
Colby's paper was in a medical journal. A reader of that publication looks at a treatment and asks who it is for, how well it works, and what the risks are.
Put the same thing in front of two professions and it gets asked two sets of questions. Under the first, the machine is just a program. Under the second, it's something to be used on people.
Two people, two papers, the same shape of machine, published weeks apart: opposite conclusions. One says the role was chosen as a way to hide what the program didn't know. The other says it could really be used to treat people.
Ten years later, Weizenbaum named Colby on page 270 of his book.
6. Blocks
Elsewhere at the same school, things were going in an entirely different direction. From 1968 to 1970, a graduate student named Terry Winograd built a program in the artificial intelligence laboratory.
Its name, SHRDLU, was a string of letters, not a word. It ran on the lab's DEC machine. On the screen, a line drawing showed a tabletop, some square blocks, a pyramid, an open box, and a gripper above them.
The gripper was drawn on the screen. No camera looked at a real table, no mechanical hand touched real blocks. The program changed the world it held inside itself. The person typed on a teletype and watched the lines move on the screen.
The formal account first appeared as a technical report in early 1971. The following year it came out as a book, Understanding Natural Language. The first sentence of the abstract calls it a system that lets a computer understand English. The very next sentence puts a limit on the world: a model of a simple robot and a collection of toy objects. The sentences are next to each other. The first opens wide; the second draws in tight. The one that traveled was the first.
That narrow world was part of the lab's openly stated method. In a progress report dated December 11, 1971, Minsky and Papert called environments of this kind microworlds. The report explained that the lab used these small, closed settings as places to work on meaning, reasoning, vision, and language. In other words, reducing the world to a table of blocks was deliberate, and they put it in writing that year.
The book in the previous essay came out around the same time as that report. The same two people wrote them. They were measuring the boundary of another kind of machine while naming their own lab's way of working. The name itself carried the limit: micro.
The published demonstration runs to forty-two exchanges. The person first asks the machine to pick up a big red block. The machine says okay.
In those forty-two exchanges, here's what it does. It follows commands to pick things up and put them down. It makes sense of long sentences with clauses and comparatives. It knows which object you mean when you say it. It can tell you what's in the box, which block is on which, and how many there are. It can answer questions about what can support what. It can stack several blocks one step at a time. When the person gives a new arrangement a name, it accepts the name and uses it later. It remembers what it did before. When asked why it did something or how it managed to do it, it can follow its own chain of goals back and give a reason.
There are turns when it says no. The person asks it to pick up the pyramid, and there is more than one on the table. It doesn't guess. It asks which one. If the person asks it to do something the rules of its world make impossible, it doesn't try to force it.
One of ELIZA's design goals was to hide what it didn't understand. In the published demonstration, this program puts what it can't do out in the open, again and again. A person and a machine talk across a typewriter in both cases. On this one question, the two programs chose opposite ways to proceed.
How big was that world? People who have since gone through the code have counted roughly fifty words having to do with it.
A common story is that Winograd finished the program and declared the approach a dead end. From 1973 through 1976, though, he kept publishing papers on natural-language understanding.
7. Four Edits
We can now put the edits in this essay side by side. The first is on the page from 1967. One retelling didn't carry over the sentence beginning with “of course.” What was left: a secretary fooled by a program.
The second is on page 42 in 1966. The reason for choosing the role has disappeared. What was left: this is a psychotherapy program.
The third is in the 1971 abstract. The part that sets a limit on the program's world didn't make the trip. What was left: a program that understands English. The fourth comes in the next section.
The first three have something in common. What was cut in each case was a qualification, and each cut made the machine bigger. Fooling the secretary shows it can pass for a person; a therapy program shows it's useful; understanding English shows it understands. Each is louder and easier to remember than what was said.
But put the three side by side and they start to fight. The first two are about a program that understood practically nothing and still fooled people. The third is about a program that really understood English. One decade, one school, remembered for two things that contradict each other: how good machines are at fooling people, and how well machines can understand.
There is no contradiction in the original accounts. One says: this one doesn't understand, and I hid the lack of understanding. The other says: that one understands a little, within the few blocks on the table. Both statements give dimensions. Statements with dimensions don't have to fight. Things of different sizes can exist at the same time. Take away the dimensions and you're left with two adjectives. Adjectives can fight.
It's the same thing that keeps happening in the earlier essays, seen from another side. Dropping a qualification doesn't just make a single claim bigger. It makes several claims that could live together unable to stand one another. Later readers have to pick sides. There was nothing to pick sides over in the original words.
Once the sides have formed, the statements take turns onstage. When it's the turn of “machines are good at fooling people,” the field is said to be pulling a trick. When it's the turn of “machines really understand,” the field is said to be about to get there. Neither has to bring new evidence when it's called on. They didn't grow out of evidence in the first place; they grew out of two sentences that had been cut down.
The people cutting the qualifications generally had reasons of their own. Relay a statement with dimensions to someone who doesn't need the dimensions and they drop away on their own. A reporter needs a paragraph the reader will finish. A teacher needs an example students will remember. A textbook needs to fit ten years on a page. All have a perfectly good reason to lose that half sentence.
After a sentence has been cut, the next person retelling it gets the shortened version. If they cut it again, that's where they start. The original is still on the page.
8. Ought
In 1976 Weizenbaum published Computer Power and Human Reason: From Judgment to Calculation. Two words in the title are set against each other: judgment and calculation.
On page 227 is a passage that is among the book's most frequently quoted, and one of the most frequently cut. Here is how it goes. First he says the questions at issue aren't technical questions or mathematical questions. They're ethical questions. Then he says they can't be settled by a question that starts with can. Then he says the ultimate limit on where a computer should be used can only be put in terms of ought. Finally he adds a qualification about time before giving the conclusion: as things stand, we have no way to give computers wisdom, so we ought not now to delegate tasks that require wisdom to them.
As things stand. Now. Both qualifications are in the original. Take them out and the passage becomes: computers can never have wisdom, so we should never give them anything important to do. That's the version that circulates today.
This is the fourth edit. It goes in the other direction. The first three make the machine bigger; this one makes a person's claim bigger. The method is exactly the same: take out the qualification.
There's another passage on page 270. There he names two people, Simon and Colby, and says they stand for opposite directions. Then he writes that there are things that belong to human beings which ought not to be done by computers in their place.
Read that with what comes next. He immediately distinguishes this judgment from another question: whether a machine can in fact do those things. He doesn't say it can't. He says it ought not.
He doesn't give a list of jobs or types of software to ban. He points to things within certain relationships: respect, understanding, love. Reduce the page to a single sentence saying that computers must never make any important decision, and what you lose is the very way he draws the boundary.
The dispute became public the year the book came out. In June 1976, the newsletter of an artificial intelligence special-interest group ran first Benjamin Kuipers's response to Weizenbaum's book, then McCarthy's “An Unreasonable Book,” then Weizenbaum's reply. Three pieces, one issue, consecutive pages.
Kuipers wasn't saying the book raised questions of no value. He thought its overly severe manner got in the way of the questions that really mattered. He also emphasized that a computer was still a tool in people's hands. Put a computer in the middle and the people behind what the machine does don't cease to be responsible for it.
McCarthy's title called the book unreasonable. What he chiefly objected to were the stronger claims about artificial intelligence, human judgment, and the morality of researchers that Weizenbaum drew from certain facts about computing. He didn't accept an indefinable kind of human judgment or a claim to human uniqueness as grounds for drawing a technical boundary in advance. He also objected to painting everyone who worked in the field with the same moral brush.
Here is McCarthy again. His name was on the document asking for money for the summer of 1956. This time he's on the side the boundary is being drawn around.
McCarthy's line is this: if you say a machine shouldn't do a kind of thing because people have a power of judgment you can't define, there's no way to check your reason or argue it with you. On his side, the boundary has to be something that can be worked out, overturned, and drawn again.
Weizenbaum's first sentence on page 227 says these are ethical questions, not technical questions. From the beginning he had no intention of drawing a technical boundary. So the two stood on different ground when they argued. One wanted a boundary that could be calculated. The other said plainly that his was not one. Two kinds of boundary, side by side on neighboring pages of the same newsletter.
In October of the same year, a journal about technology and culture ran a review by Agassi. He said some of the book's central moral claims verged on common sense and the book hadn't made clear what it added to what Wiener's generation had said. He also praised its technical explanations for nonspecialists and still recommended reading it.
The dispute had this shape: a book came out and in the same year several signed reviews and responses appeared. Two of them and a reply from the author crowded onto consecutive pages in the same newsletter. It wasn't like the perceptron book in the previous essay. That book got two reviews, sixteen months apart. This one was an argument face to face from the start.
And the substance of the argument didn't travel down to us. What traveled was the sentence with everything cut away.
The four edits in this essay leave out a sentence saying the secretary knew, the reason for writing the DOCTOR script the way he did, the scope of the blocks world, and two qualifications about the present. There are also two stories told later: he was shocked, and the author of the blocks program declared it a dead end. We may not be able to trace the moment those stories took shape. A sentence rarely leaves a trace at the moment it changes. To the person taking out a qualifier, it usually just feels like making a sentence read better.
Back, finally, to the room. A secretary sits at a typewriter, types a few lines, and turns to ask the man who wrote the program to step out. She knows a machine is doing the answering.
Ten years later, the man who wrote the program says he was startled. What he writes is that people became emotionally involved very quickly and very deeply. And in his own sentence from 1967, the secretary knew it was a machine.
Later people rewrote it as a story of being fooled. A story about being fooled has a solution: tell people plainly. Tell people plainly and the original story is still there.
On page 227 of the book the question changes: it isn't whether the machine can, but whether it ought to. That question can't close the door or open it. It leaves the door where it is and leaves the opening and closing on the human side.
The machines are still being built. Each generation says again where the boundary lies.