The Chisel-Construct Cycle · AI Series · 10

How a Report Became a Winter

Han Qin · About 23 minutes · Published

1. A Committee's Name

In October 1963, Leland Haworth, the director of the US National Science Foundation, wrote to the National Academy of Sciences. He wasn't writing just for his own organization. He represented three: the National Science Foundation, the Department of Defense, and the Central Intelligence Agency. Together they asked the academy to set up an advisory committee to see whether something was worth spending money on any longer. They'd been paying for it for a decade or so and wanted to know what they'd bought. It was time to look at the accounts.

In April 1964, the academy's president, Frederick Seitz, set up the committee. Its name was the Automatic Language Processing Advisory Committee, or ALPAC. There was no “translation” in the name. Later people usually called it the machine-translation committee.

It had six members. John Pierce of Bell Telephone Laboratories chaired it. The other five were John Carroll of Harvard, Eric Hamp of the University of Chicago, David Hays of RAND, Anthony Oettinger of Harvard, and Alan Perlis of Carnegie Tech. Pierce handed the report to Seitz on August 20, 1965. It came out in print in 1966. Letters passed between those dates. On July 27, 1966, Harvey Brooks was still writing to Pierce about wording and what the report's publication might do to funding.

The printed report was called Language and Machines: Computers in Translation and Linguistics. With its appendices it ran to 124 pages. Something the report does in those 124 pages would later be erased entirely: it distinguishes between names.

It used at least five terms for things in the same field: automatic language processing, mechanical translation, machine translation, computational linguistics, and machine-aided translation. The five terms covered different things in those 124 pages. You couldn't swap one for another. The report gave machine translation a strict definition: from machine-readable source text, an algorithm produces a usable translation without human translation or editing along the way. Machine-readable meant punch cards, punched paper tape, or magnetic tape. A keyboard operator had to key in printed material character by character.

By that definition, the operational system used by the Air Force’s Foreign Technology Division didn't count as machine translation. A person had to go over its output and revise it. So in those 124 pages it was called mechanical translation. A committee writing for three funding bodies went to the trouble of sorting out five names. In the later story, the five became one.

2. A Reading Test

The committee didn't just write down its judgment. It ran a test. The test was for people, not machines.

The same set of Russian technical material was made into three English versions. The first came straight out of a machine. No one touched it. The second came out of the machine and was edited by a person. The third had been translated by a person from the start. People were asked to read the texts and then answer questions about them. Their reading speed was recorded, too.

Here are the results for the physics texts. Relative to the human translation, readers of the unedited machine version were 10 percent less accurate and 21 percent slower, and their comprehension scores were 29 percent lower. With human editing, the three shortfalls were 3 percent, 11 percent, and 13 percent.

The comprehension score for unedited machine translation was about seven-tenths of the score for human translation, not zero. The committee itself said so in the body of the report. Most of the time you could puzzle out what it meant, but reading was slow and hard and you could be led astray. So it distinguished two things: being able to work out the general meaning, and getting a translation good enough to use.

There was another test in an appendix to the same report, which the committee separately commissioned Carroll to design. He didn't ask whether the translation was good. He broke quality into two questions. One was intelligibility: does this English sentence make sense on its own? The other was fidelity: does it carry over what the original says? To judge intelligibility, the raters looked only at the English sentence under review. They consulted the original or a reference translation to get an indirect measure of fidelity.

There were two groups of eighteen, and both groups rated both qualities. The Harvard undergraduates, who did not know Russian, used a carefully prepared English translation for comparison. The native English speakers with a good command of scientific Russian compared the sentence directly with the Russian. They didn't mark answers right or wrong. They graded on a nine-point scale. The sentences to be judged were randomly mixed among sentences from other translations so the judges wouldn't get a sense of the pattern as they read through them.

This was how it was done in 1965. A committee wanted to judge a line of technical work. What it came up with was a scale: break a vague term into two things you can rate separately, find thirty-six people, give them nine points to choose from, and shuffle the samples.

A test and a judgment are two different things. With a judgment, other people can only agree or disagree. A test can be run again. The questions are there, the material is there, the way of scoring it is there. Anyone who disagrees can find another thirty-six people and do it over. The disagreement becomes something that can be tested again.

Something like that happened once before in this series. A book used a number called the order to measure which things a class of machines couldn't decide. This time what was measured was different. It wasn't the machine; it was the person sitting there reading. How much can a person understand of what comes out of the machine, and how long does it take?

3. A Hundred and Nine Days

One appendix has operating figures from the Air Force’s Foreign Technology Division system. Its average output from June through September 1964 was 103,146 Russian words a day. It averaged 7,569 words per computer hour. A typical working day for the computer lasted thirteen hours.

A hundred thousand words a day. And the same appendix gives another number. From the arrival of the material to the delivery of a usable translation, the average was a hundred and nine days. For high-priority material, forty-four days.

A machine that does a hundred thousand words a day, and a customer who waits a hundred and nine days. A Russian physics article comes in. The person who needs it waits three and a half months for an English version they can read. In those three and a half months, the machine's part is measured in hours. Put the two numbers side by side and they say something simple: the machine wasn't the slow link.

The appendix also tells you where the slow part was. Postediting and resetting the type together made up more than seventy percent of the total cost. Keying the material into the machine accounted for eleven percent. In other words, the machine's part took almost no money or time. What took money and time was the work of people before and after the machine.

The reading test measured the quality of the translation. These accounts measured something different: how much of the work between producing a translation and being able to use it had to pass through human hands. The answer was most of it.

Behind the judgment of 1966, then, there was a reading test and there were these accounts. What was at issue was not whether the machine got a calculation right but whether the whole production line paid. And on that line the machine happened to be the fastest and cheapest part. Saying a thing doesn't pay and saying the machine that does it is no good are two different things. The second is what traveled.

4. The Sentence and Its Footnote

The most famous sentence in the 124 pages is on page 19. It says, in effect, that machine translation for general scientific text has not yet come about, and is not in sight in the short term. That sentence was later shortened to: machine translation doesn't work.

There's a footnote at the bottom of the same page that says something else. It grants that with simple or preselected texts, experiments can produce very good machine output. Then it explains why that doesn't settle the question: such results have little practical value and limited theoretical significance.

So here is the whole claim. For general scientific text, there is none now, and none in sight in the short term. With simple, selected texts it can be done, but it isn't useful. There are two qualifications, one about the kind of text and the other about time. Both are on the same page. The version that traveled lost both.

Then there is the money. An appendix records cumulative support up to that year: about $6.58 million from the National Science Foundation, $1.31 million from the Central Intelligence Agency, and $11.9 million from the Department of Defense. Together, about $19.8 million.

There is something else in the same table. The Air Force line shows about $2,045,000 in fiscal 1964 and $680,000 in fiscal 1965. In a year it fell to a third of what it had been. And both numbers come before 1966. The line didn't start going down only after these 124 pages were printed. At least for the Air Force, a drop had already taken place before publication.

There is one other thing wholly absent from the later accounts. Pierce sent a letter with the report. What he proposed was not stopping, but spending. He recommended supporting four or five first-rate centers at $600,000 to $700,000 each per year, about $2.5 million to $3 million a year in all. The purpose was explicit: basic computational linguistics, not short-term translation production.

A report remembered for cutting funding to nothing went out with a letter recommending annual spending of a few million dollars. And what Brooks worried about in his letter of July 27, 1966, was precisely that this point would be lost. He wrote that to take the report as a recommendation to shut down computational linguistics too would be to go beyond what the committee meant. What the committee wanted to reduce was funding aimed at translation in the near term. What he worried about is what happened.

The position in those 124 pages comes down to four points. First, there is no machine translation of general scientific texts today and none in sight in the short term; it can be done with simple, selected texts, but it isn't useful. Second, there is no urgent shortage of translation in the United States at present. The problems to be addressed are quality, speed, and cost. Third, large outlays aimed at translation in the near term should be cut. Fourth, basic computational linguistics should be funded, at a few million dollars a year divided among four or five centers.

Three of the four points deal with how to spend money, and two of those are about continuing to spend it. What traveled was the first half of point one.

5. Thirty-Nine Days

The second report appeared across the Atlantic seven years after the first. On September 20, 1971, Brian Flowers, the chair of Britain's Science Research Council, wrote to James Lighthill about reviewing research in artificial intelligence. Lighthill didn't work in the field. He was a mathematician whose work was in applied mathematics, fluid mechanics, and what later came to be called aeroacoustics.

The council asked him for an individual review. Individual here describes the form: not a committee's collective conclusions but an outside reader's account of what he saw. The form meant something else, too. It didn't call for agreement. A committee has to talk until its members can put their names to the same draft. Someone writing alone doesn't have to talk it over with anybody.

He recorded thirty-nine full working days on the task. At £12.60 a day that came to £491.40. Thirty-nine days and less than five hundred pounds.

What he wrote took up twenty-one pages and was titled “Artificial Intelligence: A General Survey.” Completed in 1972, it was printed by the council in 1973. Twenty-one pages. The other report ran to 124; its appendices alone held a rating experiment with thirty-six judges, operating figures on throughput, and a funding table spanning more than ten years. This was one long piece by one man who had spent thirty-nine days looking.

The reports took different forms and were different in size. Later they were set beside each other in one story as though they were the same kind of thing. One killed machine translation; the other killed artificial intelligence.

The backbone of Lighthill's twenty-one pages is a classification. He didn't divide the field as we do today. He used three letters and gave each a name of his own.

Category A was Advanced Automation: automation for a particular and clearly defined use, with a machine doing a specific job in place of a person. The point is specific. It wasn't about building something that could do everything.

Category C was Computer-based Central Nervous System Research: using computers to study, simulate, and model the central nervous systems of humans and animals, for the purposes of psychology and neurophysiology.

Category B sat in the middle. Its original name was Bridge. The bridge was work that tried to join the practical automation of A to the nervous-system research of C. It included, in particular, efforts to build robots with broader intellectual abilities, general problem solving, and certain tasks in language and vision.

Later, B was widely written up as robotics. That drops the word bridge, although that word is the whole logic of the classification. Lighthill's point wasn't that robots didn't work. It was that a kind of work justified itself by claiming to join the two ends, and he wanted to know whether the bridge actually reached across. It wasn't just robots he put on the bridge. He put general problem solving and some work in language and vision there, too.

6. Four Responses in the Same Book

What is quoted most often from the twenty-one pages is a general verdict. To paraphrase it: discoveries so far in every part of the field had failed to make the kind of major impact initially promised. That last part is the qualification. The point of comparison is what was promised at the start. Cut it and you get: the field has produced nothing.

Yet the same piece praised both A and C. It recognized that worthwhile methods already existed for specific automation problems and that computer models had scientific value for research on the nervous system and cognition. Its criticism was concentrated in one place: extending those achievements into a general claim about B.

There's another place on page 17. Lighthill clearly praises the blocks program from the previous essay. Then he immediately adds a proverbial reservation: one successful case is not enough to prove that a general line of work has succeeded. Extract the praise and Lighthill endorsed the program; extract the reservation and he dismissed it. Neither is what that page says.

Now look at how the twenty-one pages were printed. The council didn't publish them on their own. In 1973 it published a collection called Artificial Intelligence: A Paper Symposium. Lighthill's twenty-one pages come first. Then there are four contributions offering different views, by N. S. Sutherland, Roger Needham, Christopher Longuet-Higgins, and Donald Michie, on pages 22 through 45. Four responses, bound into the same book as the piece they answered, the page numbers in sequence.

The later story sometimes says the British researchers never got a chance to answer. The answers are in print from page 22 onward.

The four don't all take the same position. Needham starts by saying that he substantially agrees with Lighthill's conclusions. His two pages and a bit add qualifications and details rather than rejecting the conclusions. Longuet-Higgins accepted artificial intelligence as a scientific tool for understanding human mental activity and objected to the title and description of C leaning too heavily toward the brain's hardware. One scientific value of the field, he stressed, was that it made models of cognitive processes precise enough to be tested directly.

Michie's eight pages are the most direct. He objects that Lighthill lacks professional qualifications in the field, that he took too little time, and that he consulted too narrowly. He also defends the research in category B. The objection about time goes back to those thirty-nine days.

The following year McCarthy published a formal review in an artificial intelligence journal. His brief remark was that most researchers in the field would at once lose intellectual contact with Lighthill. The first thing McCarthy objected to was the classification. As he saw it, dividing the field into A, B, and C shuffled parts of artificial intelligence that had already succeeded into other fields. What worked went to advanced automation, what worked went to neuroscience, and what artificial intelligence got to keep was the unfinished bridge in the middle.

This idea has appeared before in the series. Essay seven looked for something: had anyone between 1957 and 1970 put in black and white that once a machine masters an ability it no longer counts as intelligence? We didn't find that sentence in the publications from those years. Here, in a piece from 1974, the claim has a specific object. It doesn't say that's how people are in general. It says this table does it. When it lands on paper this time, it isn't an observation about human nature. It's an objection to a particular document.

There was activity on the American side as well. On May 16, 1973, Nils Nilsson and ten others from Stanford Research Institute, Stanford, MIT, and the University of Southern California sent a joint letter to Flowers as chair of the council. Lighthill hadn't spent his thirty-nine days seeing just a handful of British researchers, either. The names in his list and acknowledgments include Brooker, Needham, Gregory, Longuet-Higgins, Michie, Minsky, McCarthy, Nilsson, and Winograd. He saw these people and then wrote the twenty-one pages.

7. A Mirage

In 1973 the BBC recorded an episode of a series called Controversy. The proposition that day was that the general-purpose robot is a mirage. The venue was the Royal Institution in London. Four people were onstage: Lighthill, Michie, McCarthy, and Richard Gregory.

The video survives. The date of the debate is still unclear. Later records give four answers: May 9, June, June 14, and July 4. There is video, but no official transcript.

Lighthill's central concern was still the problem of category B. A program or robot that succeeds in a tightly restricted environment has not thereby shown that the same method can cope with a rapidly growing number of possible states and combinations of actions in an open, complex world.

On page 8 of his twenty-one pages he sets out the whole argument, using theorem proving. It goes like this. People started with high hopes for theorem proving. The number of possible lines of deduction rises quickly as the problems get bigger. The growth in the number of paths cancels out the gains from faster computers. So we can't expect the next generation of faster hardware to scale the same general search method up to problems of real-world size.

The passage is often boiled down to: Lighthill thought computers weren't fast enough. He was saying precisely the opposite: greater speed wouldn't help, because the things to be searched were multiplying much faster than the machines were speeding up.

Michie's argument onstage was that the difficulty of large, general problems wasn't a reason to write off the work on robotics, learning, and problem solving between A and C. The point of building an actual robot, as he saw it, was to test theory, not to put on a show. McCarthy made the same point he wrote up the next year: start by challenging the classification. Gregory, a researcher in perception and psychology, talked about vision, perception, and where models of human cognition meet the work with machines.

This debate left something the previous essays haven't had: image and sound. In the first nine, the physical evidence to hand was paper: manuscripts, printouts, journal pages, conference records. From the 1973 debate we have the four faces, their tones of voice, the pauses and interruptions, footage of a robot shown on the day. For the first time in the series, we can see the faces of the people arguing about what machines can do, and hear them talk.

Faces and voices, but no date. That's new in this series. In the previous nine, what was often missing was the scene: what the room looked like, how the machine sounded, who was there. The date was usually something we could pin down. This time it's the reverse. We can clearly see all four faces, who interrupted whom, who paused for a moment. And four records give different answers about which day it all happened.

What survives and what gets lost doesn't depend on which matters more. It depends on who had a machine recording and who had a sheet of paper to take notes. In the room in 1973 there was a camera. There was no clerk taking minutes.

8. The Name of a Season

Now lay out what we have in this essay.

1966, United States. A report on translation, 124 pages, six people, a reading test comparing three translations, a sentence-rating study with thirty-six judges, and a covering letter recommending a few million dollars' spending a year.

1969, United States. A law. The Mansfield Amendment was incorporated into the military procurement authorization for fiscal year 1970 and signed into law on November 19, 1969. It required research financed with the authorized defense funds to have a direct and apparent relationship to a specific military function or operation. The text contains no artificial intelligence, no machine intelligence, no robots, no machine translation. Not a word. The limit it sets is on the relationship between the purpose of the spending and the military task.

1973, Britain. A twenty-one-page review by one person, thirty-nine days, £491.40, bound ahead of four responses.

Around 1976, United States. A concentrated effort called Speech Understanding Research came to an end. It had run since about 1971, roughly five years. Its goal was to get usable information from continuous speech with a limited vocabulary and multiple speakers. Carnegie Mellon's system called Harpy was one of its main results. In the later story, those five years often turn out to have produced nothing.

Four events. Two countries. Translation, military spending, artificial intelligence, speech. Four kinds of records: an academy's inquiry, a law passed by Congress, a review commissioned by a research council, a research program reaching its end. Later the four events were made into one and given a name. The name was winter.

It wasn't a name from the time. It appears nowhere in the documents from those ten years, 1966 to 1976: not in the papers above, not in the collection, not in the letters, not in the video, not in the law.

In 1984 the American Association for Artificial Intelligence held a panel at its annual conference. The transcript appeared in its publication the following year under the title “The Dark Ages of AI.” The title says Dark Ages. The moderator also said some people were already calling his worst-case scenario an AI winter. He meant the future. One printed source this essay can trace for the name being turned back on the earlier period is a 1993 history of artificial intelligence. The book traces the term to the 1984 panel.

That's the order. Four things happened from 1966 to 1976. In 1984 some people used winter for something that had not yet happened. Later books used the word for the years of those four events. It gradually became the name of a period in textbooks. Once a period has a name, that name reaches back and rewrites the time it covers.

The edit in this essay differs from the ones before it. There, the cuts took qualifiers out of sentences: the Navy expected, or reject, of course she knew, as things stand. Each cut took a few words from one sentence. Here what's cut is the boundaries between four events. Take out those boundaries and you get a single event. A single event has to have a shape. So it was given the shape of a season.

The figure of a season gives you four things in one word.

First, it gives you a common cause. A season is the weather. The weather comes from outside; the people indoors don't decide on it. So once it's called winter, you don't have to ask who signed which piece of paper, or what they had to go on.

Second, it gives you a common reach. Winter doesn't make only one room cold. So translation funding in America, laboratories in Britain, and a speech program in America all find themselves automatically under the same sky.

Third, it gives you a period of time. Winter starts and ends. Now you can mark off a period, from this year to that, even though the lines around it were drawn later.

Fourth, it gives you an ending. Winter passes. So as soon as you use the word, you've also announced that spring is coming, and that forecast has been used many times since. Four things, one word.

First look at when the name arrived. Late arrivals have happened several times in this series. The machine got its name from someone else in a review the following year. The name that put those theses together came thirty-one years later. The method was named for a person who was no longer alive.

This time the name is farther removed. The earlier names were for things still there. Someone could point at the thing and say that's what it's called. This time the name is for a stretch of time that had already been over for a decade or two. What people had when they named it was summaries of a few documents from that time, and those summaries had been through one round of cuts already. Name a thing and you can still look back at the thing. Name a time that has passed and you have to go through the records one by one. No one thing is the season.

Now put those four things back in the documents of their time.

The cause isn't the same. A 1969 law concerned the use of military funds. It had nothing to do with the 124 pages from 1966. And the Air Force line in those pages had already dropped before the report went to press.

The reach isn't the same. The collection holds four responses on pages 22 through 45, printed at the same time as the piece they answer. McCarthy's 1974 review appeared all the same in a journal actually called Artificial Intelligence. In 1974 Michie published a book called On Machine Intelligence. All those names are in print, in public.

The period isn't the same. The four events span a decade, in two countries, each with its own history.

They have even less in common when it comes to their endings. What happened afterward differs with each. Speech research didn't stop. The program ended, but the work went on. Some of the British centers closed.

There are three gaps, all in the same place: between a judgment and a sum of money. After the 124 pages came out, how much did each body actually spend in 1967, 1968, and 1969? We haven't found accounts that can be joined up with the report's own table. For Britain, we haven't found a table for 1973 to 1975 that gives project names, prior funding, amounts cut, dates of termination, and who made the decisions. And for the end of the speech program, this essay hasn't found a document giving the signature, the day, and the reason.

The judgments survive, in print, complete with page numbers. Where the money went is not fully told by those documents. And the later story is all about money. A season's name fits neatly in the gap. It explains the place where the accounts don't join up: no need to look up who signed. It got cold.

The name gets rid of a few other things on the way. The thirty-six judges and their nine-point scale, gone. The hundred and nine days and the seventy percent of the cost, gone. Pierce's covering letter, gone. The four responses on pages 22 to 45 of the collection, gone. The thirty-nine days and £491.40, gone. These things have something in common. They all give dimensions. A season doesn't come with dimensions.

Back, finally, to the room in 1965. People sit down to read three English versions of the same Russian material: one straight from a machine, one edited by people, and one translated by people from the start. When they finish they answer questions about what they've read. Someone notes how quickly they read and how many answers they got right. What was measured wasn't the machine. It was how much the people understood, how long they took, and where they were led astray. Is the machine any good? That was made into a different question: can a person use what it produces?

Later that event was remembered as a sentence: machine translation doesn't work. Later still, the sentence too was remembered as a season. And a season has no people in it. A season is the weather. The weather doesn't ask who was reading inside, or who signed which piece of paper.

The machines are still being built. Each generation says again where the boundary lies.