The Chisel-Construct Cycle · AI Series · 12

The Program That “Beat the Doctors”

Han Qin · About 19 minutes · Published

1. How Certain Are You?

The Stanford project team's book prints a terminal demonstration from the 1974 version of the program. Someone is teaching the program a rule.

The rule goes like this. Suppose a patient's culture comes from a particular site; the Gram stain gives a particular result; the organism has a particular shape under the microscope; it doesn't require oxygen; and it entered the body by a particular route. Then there is strong evidence that it is a Bacteroides organism. Five conditions, one conclusion.

The person teaching it types in the conditions and the conclusion. When the program gets to the conclusion, it asks a question: how certain are you? It doesn't ask the expert to enter a decimal. It gives him a scale, one to ten, a scale people decided on. The expert types nine. The program prints the rule as it understood it, splitting one condition into two to make five in all. It writes nine as 0.9 and asks him to confirm.

From the start the rule has a number: 200. The program puts its conclusion this way: there is strongly suggestive evidence, 0.9, that the organism is Bacteroides.

The 0.9 is a score an expert gave to the link between those five conditions and that conclusion. It says how strong he thought the link was. It comes from someone pointing to a place on a scale of one to ten. The program has rules of its own for combining scores, not the rules of probability. Reading 0.9 as ninety percent came later.

A person's judgment becomes a notch on a scale, then a decimal, then something people later read as a probability. Each time it changes form, it gets a step farther from the person. This essay is about how people spent those twenty years putting what they knew into machines, and what happened once it was in.

2. The Atomic Weight of Carbon

Start ten years earlier. In 1965 a project began at Stanford that would come to be known as DENDRAL. Its formal paper appeared in 1969 in the fourth volume of Machine Intelligence: “Heuristic DENDRAL: A Program for Generating Explanatory Hypotheses in Organic Chemistry.” The authors were Bruce Buchanan, Georgia Sutherland, and Edward Feigenbaum.

It had a specific problem to solve. Give it a molecule's composition and its mass spectrum: the signals from its fragments, sorted by mass-to-charge ratio, after the molecule has been broken up. It looks for small structures that should be present and others that shouldn't. From these clues and rules of chemistry, it lists possible structures. Then it works out what spectrum each candidate might produce, compares it with the spectrum in hand, and ranks the candidates.

On May 21 of the same year, the Journal of the American Chemical Society ran two papers in one issue, the first two in a series called “Applications of Artificial Intelligence for Chemical Inference.” In 1969, in a journal of chemistry, there were the words artificial intelligence in a title. The year was right in the middle of the decade later called the first winter.

The program ran on a DEC computer in the Stanford Artificial Intelligence Laboratory. It was written in a list-processing language. The authors' names fall into two groups. The people writing programs and devising heuristics included Lederberg, Feigenbaum, Buchanan, and Sutherland. The people working on mass spectrometry and organic chemistry included Djerassi, Duffield, and Robertson. What had to travel between them was what chemists knew and the program didn't.

Early on, such knowledge went straight into the code. The atomic weight of carbon was written into a function called weight. A constant of nature, written into the steps of a calculation. To change it you had to change the program.

As the amount of knowledge grew, that wouldn't do. The project moved chemical properties out of the code and into data structures that were easier to change. The people who had done the work later summed up the experience: getting the knowledge of a field into a program had become a bottleneck. This is the first appearance of the word bottleneck here. It doesn't mean that the machine was slow to calculate. It means the knowledge was slow to get in.

DENDRAL was later moved to another computer and made available through a remote time-sharing facility. Chemists in American universities and industry could use it throughout the 1970s and into the early 1980s.

As for the claim that it was the world's first expert system: in 1969 it called itself Heuristic DENDRAL; in the chemistry journal it went under the banner of artificial intelligence. The title calling it the first expert system for forming scientific hypotheses came in 1993, in a retrospective written by people who had been part of the project.

3. One Name and Three Names

Around 1972, a few people from Stanford's medical school and a few programmers began working together. At first the project didn't have a name.

Only when they needed to write a grant application in 1973 did they have to name a program that was already running. Stanton Axline, a physician, suggested the ending shared by many antibiotic names in English, the mycin in streptomycin and erythromycin. They tried to make an acronym and couldn't find one they liked. They settled on MYCIN.

The name was improvised. In the proposal the funders saw three other names, all in capitals: CONSULTATION PROGRAM; INTERACTIVE EXPLANATION CAPABILITY; ACQUISITION OF JUDGMENTAL KNOWLEDGE BY COMPUTER. Consultation, explanation, acquisition of judgment. None says doctor. The program would stand by and suggest what to do, say why it had suggested it, and learn from an expert how to make suggestions.

It hadn't looked like that at first. The initial proposal was for an antibiotic prescription monitor to watch what drugs the hospital was sending out. To do that it would need data from the pharmacy, the clinical lab, and the bacteriology lab. They were on three separate computer systems. Rather than connect the three, the project shifted to a consultation system that talked directly with a physician. What kind of program it turned into was decided by how hard it was to connect things. Connecting three systems was too hard, so it asked a person instead.

Its scope was narrow. At first it handled only antimicrobial treatment for bacteremia. Later it was extended to infectious meningitis. Because bacteria aren't the only causes of meningitis, the knowledge base took in some viruses and fungi as well.

The situation it was meant to face had another feature. In the first twenty-four to forty-eight hours after a patient was admitted, lab results were often incomplete. The task was not necessarily to identify one pathogen from the first few results. It was to cover the plausible, dangerous possibilities. Covering them is not the same as identifying them.

By the project's 1984 account, the mature knowledge base had about 450 rules and about a thousand facts in tables, and could consider about 120 pathogens.

4. Six Steps

How did the rules get in, one by one? Edward Shortliffe was the first to handle this for MYCIN. He elicited judgments about infectious disease from physicians Stanley Cohen and Stanton Axline while discussing how to represent them with Buchanan and others. They chose rules of the form if this, then that, for an explicit reason: physicians could read them, find mistakes, and make changes. The form of the rule was chosen for the people who had to read it.

In the mid-1970s the work acquired a name. Knowledge engineering, knowledge-based systems, expert systems: these terms became common in the group around 1975. In the team's 1984 book, knowledge acquisition is defined as extracting knowledge from human experts, textbooks, databases, and experience, then transferring and transforming it. Usually there was someone between the expert and the program. That person was called a knowledge engineer.

The team set down six steps for how it actually worked. First, the expert says what should be added or changed. Second, the knowledge engineer revises the knowledge base. Third, they run the old cases again. Fourth, the two of them look at what went wrong. Fifth, the expert tries new cases. Sixth, they look at the results and go back to step one. The authors emphasize how long a single turn through the loop takes. It is one of the main problems.

Often the hardest part was not the loop itself but the time before it could start. The knowledge engineer had to learn the expert's language and categories; the expert had to come to understand what the program could express. Only then did the number of rules start to grow quickly. Once it did, the rules still had to be tested and revised one by one. The two people had to learn to speak to one another first.

It was much the same with DENDRAL. Buchanan talked back and forth with mass spectrometry experts, turning judgments they had never written down into usable heuristics. Both the experts and the knowledge engineers had to put in a great deal of time. Much of it, early on, went into changing code, reconsidering categories, and running cases all over again.

Later the project built tools intended to let the experts inspect and revise the knowledge base on their own. One was called TEIRESIAS, after the blind seer of ancient Greece. The retrospective says the infectious-disease collaborators didn't come to use the full set of acquisition and debugging tools in their everyday work. The tool existed, but the person in the middle was still needed.

5. Eight Hundred Ratings

On September 21, 1979, the Journal of the American Medical Association published “Antimicrobial Selection by a Computer: A Blinded Evaluation by Infectious Diseases Experts,” with Victor Yu as first author. The title contains neither diagnosis nor defeat nor artificial intelligence.

The evaluation was designed step by step. It assessed one thing: what antimicrobial treatment to use in acute infectious meningitis before the pathogen was known.

First, ten meningitis cases were taken from a county hospital affiliated with Stanford and written up as case summaries. Each summary went to eight people who independently recommended treatment: five Stanford infectious-disease faculty members, a senior infectious-disease fellow, a senior resident, and a senior medical student. Added to these were MYCIN's recommendation and the treatment the patient had actually received. There were now ten recommendations for each case.

Second, eight infectious-disease experts from outside Stanford were asked to rate them. All eight had published on treating meningitis. The ten recommendations for each case were put in random order and given a standard format. Raters were not told who made each recommendation or that one came from a computer. Each rater first wrote down how they would treat the case. Then they put each anonymous recommendation in one of three categories: equivalent, an acceptable alternative, or unacceptable. Ten cases, ten recommendations, eight raters. Eight hundred ratings in all.

The results went like this. Out of eighty ratings for each source of recommendations, MYCIN's were acceptable fifty-two times, or sixty-five percent. Faculty member A had fifty. Faculty member B and the fellow had forty-eight each. Faculty member C and the actual treatment had forty-six each. Faculty member D had forty-four. The resident had thirty-six. Faculty member E had thirty-four. The medical student had twenty-four. For the five faculty members the average was fifty-five and a half percent, with a range from forty-two and a half to sixty-two and a half.

Count instead the cases in which a majority of raters accepted the recommendation. MYCIN had seven out of ten; the treatment actually given also had seven out of ten. Then there was a stricter measure: had the recommendation missed any treatable pathogen? None for MYCIN, none for the actual treatment. Among the five faculty members, three missed none and two missed one each. The fellow missed one, the resident one, and the medical student three.

The later story says MYCIN beat doctors at diagnosis in a blind test. The paper assessed drug selection, not diagnosis. It covered ten cases of meningitis. A stack of anonymous recommendations was rated. There was no head-to-head contest between a machine and a doctor.

The table has ten rows. On the first measure MYCIN comes first. On the second it ties with the treatment the patients received. On the third it ties with several faculty members. There is no row called doctors. Sixty-five percent versus about fifty percent comes from taking the first row on its own and rolling five other rows into one average.

6. Why No One Used It

The highest-scoring recommendations came from a program that was not put to use at the bedside. That isn't something outsiders said. In 1984 the project team devoted a section of its book to why MYCIN was never used routinely.

These are the reasons they gave. The kind of computer needed to run it was still too expensive. Its infectious-disease knowledge wasn't broad enough. The interface wasn't good enough. For meningitis and bacteremia alone, it didn't make economic sense.

There was also the matter of where it could run. The program was written in a dialect of a list-processing language and depended on the kind of operating system available at Stanford and at the remote time-sharing facility. Wards and doctors' offices didn't have that equipment. Rewriting a large and complicated system for a small computer was an undertaking the authors said they had no incentive to take on.

Then there were the human reasons. Who would be legally responsible? What about doctors' professional pride? Would the program's ability to explain its reasons be enough to make doctors accept it? The team looked back and wrote that they had once equated good performance with adoption. They came to think that was naive.

The 1979 blind evaluation asked whether experts could accept the recommendations it made. That was something one could measure, and it had been measured. Whether a doctor would use it at the bedside was a different question. They hadn't measured that.

The answer to the question they had measured was taken as the answer to the one they hadn't. We have seen this in essay five and again in the previous essay: a hard question has its place taken by one that is easier to ask. This time the people who made the program did the substituting. Five years later they were also the people who said so in a book.

Between a program that wins the highest score across eight hundred ratings and one that sits in a hospital ward, almost none of the things the team listed is a rule. They are the price of a computer, whether the room has the equipment, who takes responsibility, and how doctors see themselves.

There was something else. After about 1977, the project's medical collaborators began one by one to do other work. The infectious-disease knowledge base more or less stopped growing on a large scale. Work on explanation, knowledge acquisition, and tools derived from MYCIN went on. Putting medical knowledge into it stopped as the physicians left.

There was no clean date when MYCIN came to an end. It stopped a little at a time. First people left; then the knowledge stopped growing. Rule 200 was still in the knowledge base. So was 0.9. The people on the medical side who supplied the numbers went off, one by one, to do other things.

7. Seven Hundred and Seventy-Two Rules

Around the same time, there was a program that did go into use. The proceedings of the first annual conference of the American Association for Artificial Intelligence in 1980 included a paper by John McDermott called “R1: An Expert in the Computer Systems Domain.” R1 was the name in the title. The company later called it XCON.

It did a plain job. DEC sold a new model of computer, the VAX-11/780. A customer put in an order listing the components they wanted. The program read the order, drew plans for where the parts should sit in the cabinets and how they should be wired, and checked what was missing from the order. The technician got the plans and used them to assemble the real machine. DEC used a DEC machine to configure DEC machines. It ran on a large DEC computer and was written in a language for production-rule systems.

The first stage began in December 1978 and took about four months. DEC staff spent five or six days teaching McDermott how to configure systems. He also read two manuals. The first version had fewer than two hundred domain rules. The second stage took about four months more. DEC experts kept looking at what it produced and pointing out what was wrong or missing. The amount of domain knowledge nearly tripled, in the paper's words.

In October and November 1979, fifty real orders were used in a systematic test. Six configuration experts took an hour or two per order to review them. They found twelve kinds of errors, all of which were corrected.

By June 1980, the program had 772 permanent rules, 480 of which the paper calls configuration rules. It knew about 420 kinds of components; a typical system contained around ninety components. Since January 1980 it had processed more than five hundred orders. DEC's manufacturing department used it regularly, and it was beginning to be integrated with sales order entry. In the fifty-order test, a single order triggered about 1,056 rule firings on average and used about two and a half minutes of computer time.

It stayed in use and kept growing. In 1989, two DEC employees, Barker and O'Connor, wrote in Communications of the ACM that it had grown from about 250 rules looked after by two people to more than ten thousand rules and a maintenance team of more than sixty people. Two people became more than sixty.

For a knowledge system that goes into use, putting the knowledge in is not something you do once and finish. It needs an organization to keep watch. That organization keeps getting bigger.

A figure often given is that it saved DEC twenty-five million dollars a year. The 1989 retrospective gives a different figure: officials estimated that this family of configuration systems saved about forty million dollars a year net.

8. Zero Point Nine

In August 1984, the American Association for Artificial Intelligence met in Austin for its annual conference. A reporter named George Johnson was there, and in November he wrote a long piece about it. The conference he recorded looked like this: blue-gray light from terminals; demonstrations with loudspeakers on the exhibition floor; everywhere, people recruiting for companies. More than three thousand attended. About two-thirds weren't from universities.

At one stand, a company called its product a knowledge-engineering environment for building expert systems. The brochure said artificial intelligence had finally come of age. Another brochure said its makers had built a better brain. The cover of BusinessWeek that year announced the arrival of artificial intelligence. The association handed out bumper stickers saying artificial intelligence was real.

Feigenbaum told the reporter that venture capital was always looking for the next hot thing; in 1980 and 1981 it had been biotechnology, and now it was artificial intelligence. In 1984 those words were something to put on a sign, not something to avoid.

The reporter did something else. He set the sales pitches beside how researchers spoke in the technical sessions. The latter kept adding qualifications. Essay seven put pieces written years apart side by side, asking what qualifications each carried and how many were left later on. This time it was the same conference, the same week, and someone who was there wrote down both ways of talking side by side.

There are figures for the money. In February 1985, the US Congress's Office of Technology Assessment issued a report that collected market figures from the time. In 1983 industry had spent about sixty-six to seventy-five million dollars on AI products. The same page said at least fifteen of America's biggest companies were expanding work on expert systems; a dozen or so new companies had been founded in the past few years to make such products; and venture capital had put at least sixteen million dollars into new companies of that kind. The market for machines sold specifically for AI development was estimated at about fifty million dollars. There was also a large figure for the future: by the early 1990s the market might reach two and a half to eight and a half billion dollars.

Governments came in too. In 1982, Japan's Ministry of International Trade and Industry set up the Institute for New Generation Computer Technology to work on fifth-generation computing. High outside estimates put the project's total funding at up to one to one and a half billion dollars over the next eight to ten years. Annual expenditure at first went from about two million dollars in 1982 to twelve million in 1983 and twenty-seven million in 1984. The institute had fifty-two people.

Britain's Alvey Programme was a response to Japan's fifth-generation project. Its AI component was called Intelligent Knowledge-Based Systems. The government committed about 310 million dollars and industry was expected to add 230 million. Intelligent Knowledge-Based Systems was allocated roughly thirty-nine million for the first five years, of which about thirty million was for universities. About 150 people were working in the field in Britain in 1982. The plan was to increase that number by half in two or three years and double it in five.

In the United States, the Defense Advanced Research Projects Agency, DARPA, began its Strategic Computing program in fiscal 1984. It came to about six hundred million dollars over five years.

The same document used artificial intelligence and expert systems in its headings. In the same chapter it warned that some of the commercial enthusiasm might not be justified. The people advising Congress were wary of claims about how useful and how broadly applicable the current generation of expert systems was. The sign and the warning about the sign were in the same chapter of a government document.

Back to the number in this essay. What people tried to do over these twenty years could be said very simply: put what people know into machines. What chemists know, what doctors know, what configuration engineers know. What the surviving material shows is how the putting in was done.

At first the atomic weight of carbon was fixed in the code. Later, because it had to be changed, it was moved out. A doctor's judgment was first asked for as a notch on a scale of one to ten, then stored as a decimal. It took a long time to go once around the six-step loop, and the hardest part was the beginning, when two people had to learn each other's language. A configuration program went from having two people look after it to more than sixty. In a consultation program that scored highest, the knowledge stopped growing after the doctors left.

These facts are all about one thing. Knowledge isn't something you can put in and be done with. Once it's in, somebody has to look after it. The records say who was there: knowledge engineers, maintenance teams, and experts coming back again and again. The more rules inside the machine, the more people beside it.

A thing that has been built always has a boundary. There is always something left outside. This series has called that part the remainder. Over these twenty years the remainder had a particular face: it was the person who still had to sit beside the machine.

Rules can be copied in and counted: 772, 450, more than ten thousand. Why a particular rule was written that way, why 0.9 came from a nine and not an eight, isn't something that can be copied in. It stays with the person who gave it the nine. With the person there the rule can change. After the person goes, the rule remains and the 0.9 remains. The one who can explain it is no longer beside the machine.

Back to the terminal demonstration. A program asks a person how certain he is. It gives him a scale from one to ten. He types nine. The program writes down 0.9.

Later people read 0.9 as ninety percent. They read eight hundred ratings as a defeat for doctors. They read a knowledge base that stopped growing after doctors left as an AI doctor used in hospitals. They read a program with more than sixty people looking after it as a machine that saved its company twenty-five million a year. Each reading moved the person who said nine a step farther away.

The machines are still being built. Each generation says again where the boundary lies.