The Chisel-Construct Cycle · AI Series · 14

Who Invented Backpropagation?

Han Qin · About 15 minutes · Published

1. Four Pages

On October 9, 1986, the British weekly Nature published a four-page paper called “Learning representations by back-propagating errors.” There were three names on it, in order: David Rumelhart, Geoffrey Hinton, and Ronald Williams.

The first sentence of the abstract describes a new learning procedure, back-propagation, for networks of neuron-like units. New is the authors' word.

Partway through the explanation is a sentence saying that Parker had independently discovered a variant of the procedure and that Yann LeCun had independently discovered another. So the four pages contain both things: the word new, and a sentence that says it was also discovered independently.

There are four references at the end: a book by Rosenblatt, one by Minsky and Papert, a paper LeCun had presented in French in Paris the year before, and the eighth chapter of a two-volume work that had just come out. Several names that appear later in this essay aren't in those four references: Kelley, Bryson, Dreyfus, Linnainmaa, Werbos.

There were a few small experiments. One was symmetry. Six switches could each be on or off, for sixty-four possible patterns. The network had to say whether a pattern was symmetrical from left to right. Only two units sat between input and output. One round went through all sixty-four patterns; the example took 1,425 rounds. Another experiment used family trees. Two imaginary families were set out as individual relationships. Who was whose father, who was whose aunt: 104 in all. The network was trained on a hundred, with four held back to see whether it could supply them on its own. It had to learn relationships between people. The four pages also made clear that the procedure didn't guarantee finding the best set of numbers.

2. Working Backward

Another network made around the same time, by Terrence Sejnowski and Charles Rosenberg, learned to read English aloud. It was called NETtalk, as in a network that talks. It saw seven letters at a time, as though through a window only seven spaces wide. It had to say how the letter at the center of that window should be pronounced. Once it had done one, the window moved along one space. There were 203 units at the input, eighty in the middle, and twenty-six at the output. The connections between units had more than eighteen thousand numbers that could be adjusted. The network didn't produce any sound. It produced codes for speech sounds and stress. Those went into a DEC speech synthesizer, and out came a sound you could hear.

The technical account says the weights were adjusted after each word according to how closely the calculated pronunciation matched the correct one. Weights are those eighteen thousand and more adjustable numbers. How much one unit affects the next depends on the number on the connection between them.

The four pages in Nature give the following order. Start at the input and work forward, layer by layer, calculating the state of each unit. At the end compare the result with the desired answer to calculate the error. Then work backward from the final layer, one layer at a time. If each number inside changes a little, which way will it move the final error, and by how much? Work that out, then adjust the numbers in steps of the prescribed size. It is not the same discrepancy being passed backward unchanged.

One thing isn't in the network: the correct pronunciation. That is given in advance. Each time the network changes, it changes toward the answer it was given. In the family-tree experiment, the right answers came from whoever made up the family trees. In the symmetry experiment, whoever set the problem decided what counted as symmetric.

3. Flight Paths

In May 1960, the American Rocket Society held its semiannual meeting in Los Angeles. Henry Kelley, of the Grumman Aircraft Engineering Corporation, presented a paper called “Gradient Theory of Optimal Flight Paths.”

He was calculating the best way for a vehicle to fly: how to make full use of its flight performance, how to move from one trajectory to another, how to get there in the least time. There were constraints all along the way. The vehicle had to obey its own dynamics. He called his procedure the gradient method, or steepest descent, literally proceeding in the direction that goes downhill fastest. There are no networks or talk of learning in the paper.

In April 1961, Harvard held a symposium on digital computers and their applications. Arthur Bryson's paper was “A Gradient Method for Optimizing Multi-Stage Allocation Processes.” Multi-stage allocation means doing something in several steps and choosing how to allocate things at each step for the best overall result. Eight years later, Bryson and Yu-Chi Ho published Applied Optimal Control. Later accounts of the early history put this book next to Kelley's paper.

In 1962 a mathematics journal published Stuart Dreyfus's “The Numerical Solution of Variational Problems.” Here, too, the problem is finding the best path from a large set of possible ones. People who reconstructed the early history say that this paper used just the chain rule to organize the calculation of the gradient from the back to the front. The chain rule is from calculus. If one quantity affects another through several stages, the overall effect along that path is the product of the effects at the separate stages.

He appeared in essay eleven. His name is in the acknowledgments of the RAND paper from 1965, along with several others. He was Hubert Dreyfus's younger brother. More than twenty years later the two wrote a book and articles together. The philosopher wrote about what machines couldn't do. A mathematics paper his brother wrote in 1962 was later placed in this line of work.

Flight, allocation, and a problem in the calculus of variations. All three looked for the best route through a large set of possibilities. At the time the procedure went under the name gradient. It found trajectories. Nobody called it learning.

4. Round-Off Error

In June 1967, Shun-ichi Amari published “A Theory of Adaptive Pattern Classifiers” in a journal on electronic computing. He called one kind of adjustment the probabilistic descent method and also discussed steepest descent. His fields were pattern classification and statistics. The problem was how a machine could sort the things it received into categories.

In 1970, at the University of Helsinki, a student named Seppo Linnainmaa submitted a master's thesis. It was in Finnish. In translation, the title was “The Representation of the Cumulative Rounding Error of an Algorithm as a Taylor Expansion of the Local Rounding Errors.”

A computer does arithmetic with a finite number of digits. Some calculations have to round down or up at the end. Linnainmaa wanted a practical way to calculate how these little errors add up in the result of a long computation. He was concerned with an arbitrary computer program, not a network. There are no neural networks in the thesis, and it doesn't have the name backpropagation. People doing automatic differentiation later called this kind of procedure reverse accumulation. It didn't have that name when he wrote the thesis.

5. The Word Learning

In 1974, Paul Werbos submitted his doctoral thesis at Harvard: Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. His field was behavioral science and he wanted to make predictions. In the thesis he called the efficient procedure for calculating derivatives dynamic feedback.

On page 22 of chapter two, he refers to functions corresponding to model neurons. He goes straight on to say that the easiest way to calculate those derivatives is to use dynamic feedback. Farther on, the thesis discusses the brain and the idea of passing derivatives backward.

In 1981, at a conference on system modeling and optimization in New York, Werbos presented “Applications of Advances in Nonlinear Sensitivity Analysis.” In later accounts of the early history, this is the time he applied the procedure explicitly to neural networks.

In 1982 the office that handles technology licensing at Stanford University received an invention disclosure called “Learning Logic.” The inventor named was David Parker. In this room, a procedure for calculating derivatives was an invention waiting to be licensed. In 1985 he issued a technical report at a center for computational research in economics and management at MIT. It had a longer title: “Learning-Logic: Casting the Cortex of the Human Brain in Silicon.”

In 1985, at a meeting on cognitive science in Paris called Cognitiva 85, Yann LeCun presented a paper in French. In translation it was called “A Learning Procedure for Asymmetric Threshold Networks.”

Around September 1985, Rumelhart, Hinton, and Williams issued a numbered technical report in San Diego called “Learning Internal Representations by Error Propagation.” The following year that title became chapter eight of the two-volume work. The name they gave the adjustment was the generalized delta rule. In Nature, the error propagation of the title became back-propagating errors.

Line up the names used at the time: gradient method, steepest descent, probabilistic descent, dynamic feedback, learning logic, learning procedure, error propagation, generalized delta rule, backpropagation. People don't agree about which of them count as the same thing.

The names show one clear dividing line. At the aircraft company the procedure found a flight path. In Helsinki it worked out an error; in behavioral science at Harvard it made a prediction possible. The word learning doesn't come in until the procedure goes into a network of neuron-like units: learning logic in 1982; a learning procedure and learning internal representations in 1985; a new learning procedure in 1986. Related arithmetic is called optimization when it is applied to a flight path, analysis when it is applied to an error, and learning when it is put in a network.

Whether a machine can learn has been a boundary since Samuel's checkers machine in essay seven. This time it falls on arithmetic that calculates derivatives backward. What takes the arithmetic across the line is a name.

6. Two Volumes

In July 1986, MIT Press published two volumes collectively titled Parallel Distributed Processing: Explorations in the Microstructure of Cognition. The subtitle of the first was Foundations, and the second was Psychological and Biological Models. The first put Rumelhart first, then McClelland and the Parallel Distributed Processing Research Group. The second put McClelland first.

The books sold fast. The first printing of six thousand copies sold out before the formal release. Two years later, an estimated thirty thousand copies were in print. The Dreyfus brothers recorded both numbers in their 1988 article about making a mind or modeling the brain, the one in essay eleven. Stuart Dreyfus appears twice in this essay. In 1962 he wrote a mathematics paper later put in the early history of the procedure. In 1988 he and his older brother counted the copies of the book.

On January 4, 1987, The New York Times ran a review of the two volumes called “The Connections of Cognition.” That year the books were also reviewed in Science. The review in Nature was called “Only Connect!” Britain's Times Literary Supplement reviewed them on June 12. A book about how a network adjusts itself to arrive at an internal representation had made it into the newspaper book pages.

This line of work had more than one name at the time. The books called it parallel distributed processing. People also called it connectionism, neural computation, or neural network research. The name that came to be most widely used was connectionism.

In 1988, McClelland and Rumelhart published a manual with models, programs, and exercises in the title. Tucked in at the back were two floppy disks with programs for the networks. A reader could put the disks in a personal computer and run a network at their own desk, propagating the effects of the error backward, adjusting one round after the next.

7. In Its Present Form

The final paragraph of the four pages in Nature begins by saying the learning procedure, in its present form, isn't a plausible model of learning in the brain.

The paragraph goes on. Using it on a range of tasks shows that moving downhill by gradient descent can produce interesting internal representations. So it is worth looking for a more biologically plausible way to do gradient descent in neural networks. A paper announcing a new learning procedure draws a boundary in its last paragraph. The procedure is called learning, but in its present form it is not learning in the brain. The authors themselves draw that boundary.

On January 12, 1989, Nature ran a commentary by Francis Crick called “The recent excitement about neural networks.” Crick had helped discover the double-helix structure of genetic material. The last sentence of the abstract says that unfortunately most of these networks are unrealistic in several important respects.

There was an argument going on along another line in the same period. In 1988 volume twenty-eight of Cognition published two long articles. One, by Fodor and Pylyshyn, asked whether such networks could account for the compositionality of thought. If you can think that John loves the girl, you can also think that the girl loves John. The other, more than 120 pages by Pinker and Prince, examined a model in the two-volume work that learned the English past tense. They asked whether its representations, training material, and way of generalizing could explain rules in language. Also that year, Minsky and Papert's book from essay eight appeared in an expanded edition. The epilogue critiqued this same new set of adjustment rules.

These passages draw different boundaries. The authors say the procedure in its present form is not learning in the brain. Crick says these networks are unlike real brains in some important respects. Fodor and Pylyshyn ask whether such networks can account for the structure of thought. Three boundaries in three places, each asking a different question.

In that same year, 1989, the procedure was already in use. At Bell Labs, Yann LeCun and six colleagues published “Backpropagation Applied to Handwritten Zip Code Recognition” in the fourth issue of the first volume of the new journal Neural Computation. The data were images of digits, each sixteen by sixteen grayscale pixels. There were 9,298 images, 7,291 used to train the network and 2,007 held back to test it. The digits, the paper says, came from US mail that passed through the post office in Buffalo, New York. The paper did not record whose hands wrote the digits.

At Carnegie Mellon, Dean Pomerleau made a network that could follow a road, and put it in the university's experimental autonomous navigation vehicle. The picture from the camera went into the network as thirty by thirty-two points, brightness values from the blue channel. There were twenty-nine units in the middle and forty-five possible steering directions at the output, from a sharp turn left through straight ahead to a sharp turn right. The network trained mostly on simulated road images and was then tested on real roads. The paper reports that under some outdoor conditions it could follow the road.

Reading English, recognizing the digits on an envelope, driving along a road. The tests asked whether it pronounced things correctly, recognized them accurately, or stayed on the road. No test asked whether it was like a brain.

8. Independently

Who first discovered the procedure is still disputed. One person who has said the most about it is Jürgen Schmidhuber. Starting in 2014, he issued several versions of a chronology on his website. In 2020 he issued it again as a half-century retrospective, “Who Invented Backpropagation?” In his account, Linnainmaa, a master's student in Finland, first published the modern form in 1970. Kelley's paper of 1960 was a precursor. Then came Bryson, Stuart Dreyfus, Werbos, Parker, LeCun, and Rumelhart and his colleagues, one after the other. On his website Schmidhuber publicly asked three later Turing Award winners to respond to his claims point by point.

On the other side, an April 2020 article reported an answer Hinton had given in an online forum. He said he had never claimed to have invented backpropagation. David Rumelhart had invented it independently after people in other fields had already invented it. His team's contribution, Hinton said, was to demonstrate clearly that the procedure could learn interesting internal representations, and bring that result to many people's attention.

On October 8, 2024, the Nobel Prize in Physics went to John Hopfield and Hinton for foundational discoveries and inventions making machine learning with artificial neural networks possible. These statements have different places of their own. Set them beside one another.

The word independently has been attached in two places. In 1986, the four pages in Nature attached it to Parker and LeCun: the procedure was new, and two other people had independently discovered variants. New was in the foreground, independently was an aside. In Hinton's answer in 2020, independently went with Rumelhart. People in other areas had invented it first; he had invented it independently later. Here the other fields come first, and independently describes his own side. In thirty-four years the word moved once. One of the original signers moved it.

Yet the sentence that traveled furthest is another one: in 1986 three people invented backpropagation and made machines learn like brains.

Two things have to be cut for that sentence to work. One is the line in the middle of the four pages that says it was also discovered independently, and the rooms behind it that don't appear in the references: an aircraft company, a symposium at Harvard, a mathematics journal, a master's thesis in Helsinki, the technology licensing office at Stanford. The other is in its present form, in the last paragraph, and the whole sentence saying this is not a model of learning in the brain. Both are qualifications. Neither carries the story. With them gone, what remains is an invention, three inventors, and a machine like a brain.

There is one other thing that isn't in the story, whether you cut anything or not.

In the experiments in this essay, the error measures how far the machine's answer is from the right one. The right pronunciation, the right digit, the direction to steer, who is whose aunt in the family tree: these are all in place beforehand. The procedure for working backward doesn't decide what counts as right. Other ways of learning can work without someone giving a right answer for each problem. But they still need a criterion that tells the network which way to adjust. This series calls what always remains outside the remainder.

Each of the nine thousand and more digits from the post office in Buffalo came from a hand. The people whose hands they were have not been recorded. They wrote the numbers because they wanted a letter to get where it was going.

The machines are still being built. Each generation says again where the boundary lies.