The Chisel-Construct Cycle · AI Series · 17

What Is Deep About Deep Learning?

Han Qin · About 10 minutes · Published

1. Four Names

In July 2006 two papers appeared in print almost at the same time. One, in the seventh issue of volume eighteen of Neural Computation, was “A Fast Learning Algorithm for Deep Belief Nets,” by Geoffrey Hinton, Simon Osindero, and Yee-Whye Teh. Its abstract describes a fast, greedy algorithm that can learn deep directed belief networks layer by layer. Immediately after that comes a condition: the top two layers must form an undirected associative memory.

The other appeared in Science on July 28. Hinton and Ruslan Salakhutdinov called it “Reducing the Dimensionality of Data with Neural Networks.” Their approach was to pretrain a multilayer network one layer at a time, then fine-tune it using the way of working backward from the error described in essay fourteen. Neural networks is right there in the title.

That winter, at the Neural Information Processing Systems conference, NIPS, Yoshua Bengio's group from the University of Montreal presented “Greedy Layer-Wise Training of Deep Networks.” The abstract starts with deep multilayer neural networks. When it explains the trouble with them, its words are that gradient-based optimization from a random starting point appears to often get stuck in poor solutions. Appears and often are both there. At the same meeting Yann LeCun's group from New York University presented “Efficient Learning of Sparse Representations with an Energy-Based Model.” The word deep isn't in the title.

Four papers, four ways of saying it: deep belief nets, neural networks, deep networks, sparse representations. That year there was no one name they all used.

Deep learning had appeared in the machine-learning literature before. In August 1986, at the American Association for Artificial Intelligence's annual conference, Rina Dechter presented a paper about constraint satisfaction called “Learning While Searching in Constraint-Satisfaction-Problems.” It uses deep learning and sets it against shallow learning. By deep she meant how far back a conflict was recorded during a search. It had nothing to do with networks of many layers.

An adjective is used somewhere first, then borrowed for something else. It's not the first time this has happened in the series.

2. Those Years

In 2004 Hinton and several colleagues secured support for a new program at the Canadian Institute for Advanced Research. It was called Neural Computation and Adaptive Perception. Bengio and LeCun were part of it too. Hinton directed it until 2013. In August 2006 the program ran a summer school in Toronto.

In December 2007 there was a workshop in Vancouver called the NIPS 2007 Workshop on Deep Learning. Later papers cite Hinton's talk there under the title “How to Do Backpropagation in the Brain.” The last paragraph of the 1986 paper in essay fourteen says that the procedure in its present form is not a model of how the brain learns. Twenty-one years later, the same person stood in front of a workshop called Deep Learning and talked about how the brain might do it.

The name grew a little at a time. In 2009 deep unsupervised learning appeared in the title of a paper by Andrew Ng's group at Stanford. In 2011 a conference workshop called Deep Learning and Unsupervised Feature Learning stated that it wanted to bring together people interested in both. By 2012 a Google paper called the line of work unsupervised feature learning and deep learning in its text. A paper that year on speech recognition used deep neural networks in the title.

LeCun later half-jokingly spoke of their group as the so-called deep learning conspiracy. He also said one paper had been rejected by CVPR in 2011 and accepted by ICML the next year. Before that, a long string of papers using convolutional networks had been turned down.

3. Graphics Cards

Graphics cards originally drew pictures on a screen. In February 2007 Nvidia released a set of programming tools called CUDA. Programs that did other kinds of calculation could now use the cards directly without going the long way around through a graphics interface.

In June 2009 Andrew Ng and two of his students, Rajat Raina and Anand Madhavan, published “Large-Scale Deep Unsupervised Learning Using Graphics Processors.” The slides for their talk list the card: an Nvidia GTX 280, with one gigabyte of memory and 240 cores, then about $250. Compared with an ordinary dual-core computer, it trained one kind of network seventy-two times as fast and another about fifteen times as fast. In a multilayer network, a layer could be trained in about a day.

Near Lugano, Switzerland, is an artificial intelligence research institute called IDSIA. In 2010 Dan Cireșan, Jürgen Schmidhuber, and colleagues used graphics cards to run ordinary backpropagation. On handwritten digits they got thirty-five in ten thousand wrong. In a paper for IJCAI in July 2011, they wrote that with graphics cards a large convolutional network could be trained in days, not months.

That same year a recognition contest on German road signs had more than fifty thousand images. The group won with an error rate of 0.56 percent. The people tested for comparison averaged 1.16 percent. In 2012 the group won a contest to identify neural structures in electron microscope images, and another to spot dividing cells in breast cancer tissue sections.

4. Two Graphics Cards

At 11 p.m. GMT on September 30, 2012, submissions closed for that year's Large Scale Visual Recognition Challenge from essay sixteen. Three people at the University of Toronto entered as SuperVision: Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Their network had sixty million parameters, 650,000 neurons, five convolutional layers, and three fully connected layers. It later came to be called AlexNet, after its first author.

It trained on two Nvidia GTX 580 cards, each with three gigabytes of memory. The network was too large to fit in one card's memory, so they split it in two. Each card held half, and the two cards communicated between certain layers. A run took five or six days.

Where were the cards? In 2023 Hinton told a reporter for The New Yorker that the two cards had sat humming in Krizhevsky's bedroom. The same recollection mentions a substantial electricity bill, paid by Krizhevsky's parents.

SuperVision combined predictions from seven networks. Its top-five error rate in the competition was 15.3 percent, against 26.2 percent for the best of the other teams. On October 12, at a workshop in Florence, Italy, SuperVision was scheduled to speak from 5:05 to 5:30 in the afternoon. That afternoon has been retold in many different ways.

In December the paper came out in the conference proceedings under the title “ImageNet Classification with Deep Convolutional Neural Networks.” Neural networks is still in the title.

In the discussion the authors describe an experiment. Take out any of the middle convolutional layers and, when only the network's first choice is scored, accuracy drops about two percentage points. Depth, they then say, really is important for achieving their results. In the same section they note that they did not use unsupervised layer-by-layer pretraining this time, the step used in the two papers from 2006.

5. Tianjin

In October 2012, at Microsoft Research Asia's Computing in the 21st Century conference in Tianjin, Rick Rashid, the head of Microsoft Research, gave a demonstration. He stood onstage speaking English. First the English words appeared on the big screen overhead. Then they became Chinese words. Then a synthesized voice read the Chinese aloud. The voice was made to sound like Rashid's own. Rashid himself couldn't speak Chinese. The audience applauded.

On November 8 he wrote on the company blog that systems like these used to get about one word in four or five wrong. Now it was about one in seven or eight, a reduction in errors of more than thirty percent. In the same paragraph he added that they were still far from perfect.

That month a paper with eleven authors appeared in a signal-processing journal: “Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups.” The four groups were the University of Toronto, Microsoft Research, Google, and IBM. In one result from the Microsoft group, the word error rate fell from 27.4 percent to 18.5. Rashid's demonstration used work from this line of research.

6. Cats

In late December 2011, Quoc V. Le, Jeff Dean, Andrew Ng, and five colleagues put a paper online. It appeared at ICML the following year as “Building High-Level Features Using Large Scale Unsupervised Learning.” The network had nine layers and about a billion connections. It was given ten million little images, two hundred by two hundred pixels, taken from videos on the internet. No one had labeled what was in them. There were a thousand machines with sixteen thousand processor cores, and training took about three days.

The question the researchers started with was whether unlabeled images alone could yield a component that recognized human faces. They found one unit in the network that responded especially strongly to human faces. They ran further comparisons, moving the faces, changing their size, and turning them to see whether it still responded. In the abstract the sentence about cats says the same network was also sensitive to other high-level concepts, such as cat faces.

The story that traveled was that Google's network taught itself to recognize cats. The paper said sensitive. It said cat faces. It said such as.

7. The Auction

In 2012 Hinton and his two students started a company called DNNresearch. The company was the three of them.

The NIPS conference that December was held at a casino hotel by Lake Tahoe. In 2021 the reporter Cade Metz reconstructed those days from interviews. In room 731 of that hotel, the three-person company was put up for auction. Google, Microsoft, Baidu, and the London company DeepMind bid. At about twenty-two million dollars, Hinton called a pause. When bidding resumed Microsoft dropped out, leaving Google and Baidu. At twenty-four million, a senior executive in Beijing took over from Kai Yu as Baidu's bidder. At forty-four million dollars Hinton stopped taking bids. Google got the company. From 2013 Hinton worked at Google while staying at the University of Toronto.

In January 2014 Google also acquired DeepMind, which had started in London in 2010. Some reports at the time gave the price as four hundred million dollars, others as four hundred million pounds.

Three people in a company. The bid was for those three people. In the room, one of them got to say where the price would stop.

8. Deep

In November 2012 John Markoff wrote a long article in The New York Times, remembered under the headline “Scientists See Promise in Deep-Learning Programs.” He described the Tianjin demonstration, the English words on the screen, the Chinese words, Rashid's voice. He interviewed Hinton, too. Hinton told him they had decided early not to make money from the approach but to spread it as widely as they could.

That same year LeCun and Bengio organized a new conference. Its first meeting the next year was in Scottsdale, Arizona. It was called ICLR, the International Conference on Learning Representations. It received sixty-seven papers and accepted twenty-three. Neither deep learning nor neural networks was in the name.

The story most often told later is that neural networks had acquired a bad reputation, so the name was changed to deep learning. Yet the papers they printed at the time, in Science in 2006 and on ImageNet in 2012, both say neural networks in their titles. In essay thirteen, the name artificial intelligence was said to have gone into hiding. The story of a name being changed is like the story of the name being put away: many people tell it; few leave anything in writing.

Things get left out of this essay's story too. Appears to often in 2006; for achieving our results in 2012; sensitive and cat faces in the Google paper; still far from perfect in Rashid's account. Take those out and what remain are a few clean sentences: only deep networks work; the machine has learned to recognize cats; the machine can translate what you say into Chinese.

Deep in 1986 was how far a search went in recording conflicts. In 2006 it was how many layers a network stacked up. In 2012 it meant that with a middle layer taken out, accuracy dropped about two points if only the first choice counted. Later it became a name. Once an adjective becomes a name, the particular thing it used to describe, layer by layer, moves behind it.

And there was the afternoon in Tianjin. A machine turned a man's English into Chinese, using his own voice. The man onstage didn't speak a word of Chinese. His voice could be made again. Who wanted those words spoken, and to whom, was still up to the person onstage. This series calls the part that cannot be built into the machine the remainder.

The machines are still being built. Each generation says again where the boundary lies.