The Chisel-Construct Cycle · AI Series · 15

“Every Time I Fire a Linguist, the Recognition Rate Goes Up”

Han Qin · About 14 minutes · Published

1. Friends

In May 2004, a conference in Lisbon on language resources and evaluation invited Frederick Jelinek to speak. The title he gave the talk was “Some of My Best Friends Are Linguists.”

The title borrows an old English turn of phrase. When people want to deny that they're prejudiced, they say some of their best friends are such and such. People who say that usually have something else already on the record.

In Jelinek's case, the line that came down was this: every time he fires a linguist, the recognition rate goes up a notch.

A remembrance written by a colleague in 2011 gives a different version: whenever a linguist leaves the group, the recognition rate goes up. The piece says he liked to amuse people with this quip from 1988.

Someone who had sat in the audience in Lisbon remembered the talk in 2012 and gave the popular version instead: every time I fire a linguist.

A verb and a subject separate the versions. In one, a linguist leaves the group. In the other, I fire a linguist. The first says a person has left, but not who made the decision. The second has a person taking action.

2. A Phone Call

Jelinek wrote about how the group began in a 2009 recollection. In 1971, IBM's Watson Research Center in Yorktown Heights, New York, set up an internal task force that included John Cocke and Herman Goldstine. It recommended that the research division form a group called Continuous Speech Recognition. Joe Raviv got the group started early in 1972.

That summer Jelinek was still at Cornell. He'd been teaching there since the fall of 1962, working on information theory and communication theory. He called Raviv, who told him the company was getting into speech recognition. Jelinek took leave from Cornell and came over. A few weeks later he took over the direction of the group.

Members came from several places. By his account, only four of them knew about speech to begin with. Jim and Janet Baker, a married couple, joined later. In September of 1972 a man named Robert Mercer arrived, a freshly graduated physicist, as Jelinek put it.

The company also brought in three people from linguistics on a temporary basis: Fred Damerau, Stan Petrick, and Jane Robinson. Jelinek later wrote that around the time the Bakers arrived, the three returned to their original group. The word he used was returned. The three linguists who left had returned to where they had been before.

The group was called Continuous Speech Recognition from the start. Continuous meant the way people ordinarily speak, one word running into the next without a pause. That was one of the words in the name in 1972.

3. Pauses

Thirteen years later the group had a machine that could take dictation in real time. The title of its paper at the 1985 conference was “IBM PC Based Large-Vocabulary Isolated-Utterance Speech Recognizer.” It used an IBM personal computer with two signal processors made by the company inside the case. The vocabulary was five thousand words; the basic list drew on common words from office memos and business letters. A new speaker had to talk to it for about twenty minutes first, so it could learn that person's voice. In another paper the same year, Jelinek specified how to speak to it: a short pause was required between words.

In 1986 the vocabulary grew to twenty thousand. Two more processors went into a separate expansion box, and there was a speech workstation for studying how people used their voices to write documents. In a paper that year the system had a name: Tangora.

Jelinek explained it in 2009. They named the system Tangora after Albert Tangora, who had set a typing record at a business exhibition in New York in 1923. He typed 8,840 correctly spelled words in an hour, or 147 a minute. A machine for writing documents by voice took its name from the fastest typist.

The group's name said continuous in 1972. The machine in 1985 asked people to pause after each word, and the paper's title said isolated utterances. The word in the group's name had another twelve years to wait. In October 1997 a speech-technology trade publication tested two dictation programs meant for the general public, Dragon Systems' first version of NaturallySpeaking and an IBM product still in trials. The magazine's term was continuous speech. The Bakers had gone on to start Dragon Systems themselves.

In the 1985 paper, Jelinek describes his approach as an information-theoretic formulation rather than an expert-system formulation. The parameters were estimated directly from the data. The approach he set next to his was called expert systems. As essay twelve noted, 1985 was just when the enthusiasm for expert systems was running highest.

4. A Cipher

In March 1947 Warren Weaver of the Rockefeller Foundation wrote to Wiener, who appeared in essay four. He compared translation to cracking a code. One line imagines that the text was actually written in English and had only been encoded in unfamiliar symbols. He was looking at Russian.

In July 1949 Weaver wrote the idea into a memo just called “Translation.” He wrote it in New Mexico and sent it to about thirty people he knew. As essay ten noted, the United States spent money on machine translation in the years that followed. A 1966 evaluation saw no immediate prospect of usable machine translation and recommended concentrating research on subjects such as computational linguistics and machine-aided translation. What the report said and how money was allocated afterward are two different accounts.

The group at Yorktown Heights took up translation around 1987. Jelinek recalled that none of the people at the center of the work was an expert on French. He names Bahl, Peter Brown, Mercer, and himself. They later took a French immersion course.

From June 12 to 14, 1988, Pittsburgh held a conference on theoretical and methodological issues in machine translation. The group submitted “A Statistical Approach to French/English Translation.” That same year the proceedings of the International Conference on Computational Linguistics in Budapest included their “A Statistical Approach to Language Translation.” The journal version appeared in Computational Linguistics in June 1990 as “A Statistical Approach to Machine Translation.” The long June 1993 paper was called “The Mathematics of Statistical Machine Translation: Parameter Estimation.” In five years the titles went from French/English translation to language translation, then machine translation, then settled on statistical machine translation.

The 1990 paper makes translation a problem in probability. Given a French sentence, find the English sentence most likely to have been its source. This divides the problem in two. How likely is the English sentence on its own? That's the language model. How likely is it to turn into this French sentence? That's the translation model. In 1947 Weaver's letter said that the text was English to begin with and only the symbols had changed. Put the two statements beside each other, forty-three years apart: one is a metaphor and the other an algorithm.

Jelinek's recollection from 2009 also reproduces a review. The group submitted its first results to the 1988 computational-linguistics conference, he says, and received an anonymous review dated March 1, 1988. One line said that brute force with a computer wasn't science. Their paper is in that year's conference proceedings.

5. Translators

These papers used records of debates in the Canadian Parliament to learn to translate. Parliament keeps the records in both English and French. As Jelinek later made a point of explaining, a member's remarks in one language are translated into the other.

In each sentence pair the machine learned from, on one side are the words the member spoke, and on the other are the words a translator supplied. The papers don't say how many translators Parliament had in those years, or who translated which passage.

More than twenty years earlier, Brown University had made what people call the Brown Corpus. It drew on English prose published in the United States in 1961: five hundred samples of around two thousand words each, about one million words in all. The work of compiling it took place in 1963 and 1964.

A university's corpus archive still gives the names of the people who did it. The project was led by W. Nelson Francis and Henry Kučera. The original corpus was put together by Anne Robb Taylor, Loretta Felice, Mary Lois Marckworth, Henry Hall Peyton Jr., and Robert Staudte Jr. The programmer was Andrew Mackie. A million words, each to be selected by someone from the printed page and then entered into the machine.

In 1992 John Godfrey, Edward Holliman, and Jane McDaniel published a corpus of telephone speech called Switchboard. It contains about twenty-four hundred calls and 543 speakers. The two people on a call usually didn't know each other, and the subject of their talk came from a list of topics prepared in advance. Two strangers spoke on the phone about something on a list. The call was recorded so a machine could learn how people speak.

6. Trees

Between 1989 and 1992 the University of Pennsylvania put together an annotated English corpus called the Penn Treebank. By the June 1993 paper it had more than four and a half million words. Each word had a part-of-speech tag, and for more than half the words the outline of the sentence structure had also been drawn.

The machine came first and the person came after. To tag the parts of speech, a probabilistic tagger written by Kenneth Church at Bell Labs made a first pass. Then people corrected it word by word. To draw the structure of a sentence, a parser supplied a tree and a person repaired it.

The project ran a comparison early on. There were four annotators, all with graduate training in linguistics. Each practiced correcting the machine's tags for fifteen hours, then practiced tagging from scratch for six hours. Tagging a thousand words from scratch took forty-four minutes on average. Correcting the machine's tags took twenty. Drawing the trees was much slower. After weeks of practice the annotators were still getting through just a few hundred words an hour. With a simplified skeleton, an experienced annotator could do 750 to more than a thousand in an hour.

The annotator looking at the screen didn't have a bare sentence. In the tree the parser had produced, there were fragments it couldn't attach, with a question mark beside them. The annotator pressed the mouse button next to the question mark, dragged the fragment to the parent node where it belonged, and released the button to attach it. Wrong material could be deleted, and the interface adjusted the indentation. Part-of-speech tags were done in a text editor familiar to programmers. Move the cursor to the mistagged word and type the correct tag. The software checked that the tag was a valid one. Two manuals said what to tag and how: Beatrice Santorini's 1990 guide to tagging parts of speech and her 1991 guide to bracketing sentence structures with Mary Ann Marcinkiewicz.

The paper describes the project as having a statistical orientation. The people sitting at screens putting sentences together one question mark at a time had been trained in linguistics.

By June 1993 the trees were available to members of an organization called the Linguistic Data Consortium.

The people who left the group in 1972 were three linguists. Twenty years later linguists were sitting at the other end. The trees the statistical machine would learn from were trees people had drawn, one at a time.

7. The Test Papers

In February 1991, Charles Wayne described two programs he oversaw at a workshop on speech and natural language. He divided the spoken-language program into two parts: large-vocabulary speech recognition and spoken-language understanding. He wrote that the program stressed objective performance evaluation.

The program from 1971 to 1976 in essay ten was called Speech Understanding. By 1991, recognition and understanding were two parts, written down side by side in the same document.

Several shared sets of problems were used for testing. One was called Resource Management: queries and commands spoken aloud. Another was called the Air Travel Information System: questions phrased spontaneously. The recordings came from subjects speaking to a system that answered questions about flights. It was a simulation, the kind called Wizard of Oz at the time. The one behind the scenes, answering for the machine, was a person. To find out how people would talk to a machine, the researchers first had people talk to a person pretending to be one.

Ten organizations received funding to take part, including companies, universities, and national laboratories. One was the Bakers' Dragon Systems. The National Institute of Standards and Technology prepared the tests and did the scoring.

There were tests on the written side, too. The second Message Understanding Conference, in 1989, used naval operational messages. The third, in 1991, used news reports translated and edited by the Foreign Broadcast Information Service. A machine was to take a passage of text and fill in a database form with what had happened. Machine translation was to be tested as well. Wayne specified testing on previously unseen, naturally occurring text.

In Toulouse in July 2001, at the annual meeting of the Association for Computational Linguistics, Michele Banko and Eric Brill of Microsoft presented “Scaling to Very Very Large Corpora for Natural Language Disambiguation.” The test was on a small thing. Three words in English sound alike but are spelled differently: to, two, and too. The machine had to choose the right one in a sentence. They trained four kinds of learner, one a perceptron of the kind discussed in essay eight. The training text grew from a million to a billion words. The test set was a million words of The Wall Street Journal.

The curve kept rising and hadn't leveled off at a billion words. The authors added a qualification straight away: adding training data has a cost. Their conclusion advises the field to spend more effort building annotated corpora and less on comparing algorithms on small ones. The later claim that more data beats better algorithms is often hung on this paper. That sentence isn't in the paper.

8. Leaving

The statements on the other side have dates of their own.

Chapter two of Chomsky's Syntactic Structures, published in 1957, has two strings of words. One is Colorless green ideas sleep furiously. The other reverses the order. Neither makes sense, but the first is grammatical and the second is not. He uses the two to discuss whether being grammatical and what a simple statistical model approximates are the same thing.

In 2000 Fernando Pereira published “Formal Grammar and Information Theory: Together Again?” He used a simple statistical model trained on newspaper text to work out the two strings again. The grammatical one was about two hundred thousand times as probable as the reversed one. The abstract argues that two traditions long treated as incompatible, formal linguistics in Chomsky's line and information theory in Shannon's, should be reconnected after forty years. The title ends with a question mark.

At a discussion for MIT's 150th anniversary in 2011, Pinker asked Chomsky what he thought of probabilistic models. Chomsky began by saying there had been some successes and many failures. What he went on to criticize was taking the approximation of data without analyzing it as a standard for success. In the same year Norvig responded with “On Chomsky and the Two Cultures of Statistical Learning.” That October Church published “A Pendulum Swung Too Far.” Set the statements beside one another and each says something different.

Go back to the line. In the version that went around, a linguist leaving the group became I fire a linguist. A verb and a subject were changed. After the change, the joke had a person taking action. What Jelinek wrote about those three people in 1972 was that they returned to their original group.

More than a verb had changed. The line was about people leaving and a number going up. It said nothing about where the numbers came from. The machine learned to translate from parliamentary records; the lines of French were translated by translators. It counted English in newspapers and books; a few people at Brown had picked out those million words and entered them one by one. It learned sentence structure from a treebank; people trained in linguistics had sat at screens and put those trees together, one question mark at a time. To learn how people talk to machines, it used conversations in which a person had sat on the other side pretending to be one.

Language was counted. What was counted was what people had written, translated, tagged, and said out loud. This series calls the share that remains outside no matter how much counting is done the remainder. Each time the figure went up a notch, there was the work of a group of people underneath it. Most of their names did not travel with the number.

The title of the talk in Lisbon was “Some of My Best Friends Are Linguists.”

The machines are still being built. Each generation says again where the boundary lies.