Defining IntelligenceSTUART RUSSELL is a professor of computer science, director of the Center for Intelligent Systems, and holder of the Smith-Zadeh Chair in Engineering at the University of California, Berkeley. He is an adjunct professor of neurological surgery at the University of California, San Francisco....
[2.7.17]
My field of work is artificial intelligence, and since I started I’ve been asking myself how we can create truly intelligent systems. Part of my brain is always thinking about the next roadblock that we’re going to run into. Why are the things we understand how to do so far going to break when we put them in the real world? What’s the nature of the breakage? What can we do to avoid that? How can we then create the next generation of systems that will do better? Also, what happens if we succeed?
Back in ’94 it was a fairly balanced picture that the benefits for mankind could be enormous, but there were potential drawbacks. Even then it was obvious that success in AI might mean massive unemployment. And then there was this question of control. If you build systems that are smarter than you, it’s obvious that there’s an issue of control. You only have to imagine that you’re a gorilla and then ask: Should my ancestors have produced humans?
From the gorilla’s point of view that probably wasn’t such a great idea. In ’94 I would say that I didn’t have a good understanding of why exactly we would fail to achieve control over AI systems. People draw analogies, for example, between gorillas and humans or humans and superior alien civilizations, but those analogies are not exact because gorillas didn’t consciously design humans and we wouldn’t be consciously designing alien civilizations.
What is the nature of the problem and can we solve it? I would like to be able to solve it. The alternative to solving the control problem is to either put the brakes on AI or prevent the development of certain types of systems altogether if we don’t know how to control them. That would be extremely difficult because there’s this huge pressure. We all want more intelligent systems; they have huge economic value.
Bill Gates said that solving machine-learning problems would be worth ten Microsofts. At that time, that would have come out to about $4 trillion, which is a decent incentive for people to move technology forward. How can we make AI more capable, and if we do, what can we do to make sure that the outcome is beneficial? Those are the questions that I ask myself.
Another question I ask is: Why do my colleagues not ask themselves this question? Is it just inertia? That a typical engineer or computer scientist is in a rut? Or are they on the rail of moving technology forward and they don’t think about where that railway is heading or whether they should turn off or slow down? Or am I just wrong? Is there some mistake in my thinking that has led me to the conclusion that the control problem is serious and difficult? I’m always asking myself if I'm making a mistake.
I go through the arguments that people make for not paying any attention to this issue and none of them hold water. They fail in such straightforward ways that it seems like the arguments are coming from a defensive reaction, not from taking the question seriously and thinking hard about it but not wanting to consider it at all. Obviously, it’s a threat. We can look back at the history of nuclear physics, where very famous nuclear physicists were simply in denial about the possibility that nuclear physics could lead to nuclear weapons.
The idea of a nuclear weapon was around since at least 1914 when H.G. Wells wrote The World Set Free, which included what he called atomic bombs. He didn’t quite get the physics right. He imagined bombs that would explode for weeks on end. They would liberate an enormous amount of energy—not all at once, but over a long period; they would lay waste gradually to a whole city. The principle was there. There were famous physicists like Frederick Soddy who understood the risk and agitated to think about it ahead of time, but then there were other physicists like Ernst Rutherford who simply denied that it was possible that this could ever happen. He denied it was possible up until the night before Leó Szilárd invented the nuclear chain reaction. The official establishment physics position was that it could never happen, and it went from never to sixteen hours.
I don’t think the same thing could happen with AI because we need more than one breakthrough. Arguably, Szilárd’s breakthrough—figuring out that you could make a chain reaction with neutrons, which don't get repelled from the nucleus in the same way that protons do—was the key breakthrough, but it still took a few more years, five or six, before a chain reaction was demonstrated.
Five to six years is an incredibly short time. If we had five or six years to the point where there were superintelligent AI systems out there, we wouldn’t have a solution for the control problem, and we might see negative consequences. If we were lucky, they would be contained, and that would be an object lesson in why not to do it. Sort of like Chernobyl was an object lesson in why it’s important to think about containment of nuclear reactions.
I can’t claim to have thought too much about containment and control early on. My first AI project was a chess program in 1975, in high school. I read a lot of science fiction growing up, and I’d seen 2001 and a lot of Star Trek episodes. The idea of machine intelligence getting out of control had been around for donkey’s years in popular culture.
I knew about all that, but I was a pretty straightforward techno-optimist in my youth. To me the challenge of creating intelligence was just fascinating and irresistible.
I studied computer science in high school. Being very interested in machine learning, I wrote a self-teaching tictactoe program and then a chess program. I read some AI books, but at the time I didn’t think that AI was a serious academic discipline.
I wanted to be a physicist, so I studied physics as an undergrad. Then I learned that there was a possibility that you could do a computer science PhD and study artificial intelligence, so I applied to PhD programs in computer science in the US, and I also applied to physics PhD programs in the UK, to Oxford and Cambridge.
For various reasons, I decided to take a break from physics. I had spoken to physics graduate students, post docs, and professors and didn’t get a very optimistic picture of what it was like to do particle theory. You would spend a decade creeping up an author list of 290 people and, if you were lucky, after umpteen years of being a post doc, you might get a faculty position, but you might end up being a taxi driver instead.
I graduated in ’82, and there wasn’t that much going on. It was just before string theory became popular. People were looking for grand unified theories of physics, not finding anything very promising or even testable. I remember very clearly a conversation I had with Chris Llewellyn Smith, who was on the faculty—this was shortly before he went on to be director of CERN—and I asked him what he was working on. Of the people that I had met and taken classes from at Oxford, he was the brightest, most engaging, intelligent man. He said he was working on taking all the grand unified theories then in existence, of which there were eighty-one, and converting them into mathematical logic. Having studied a little bit of AI, I knew about this. In mathematical logic it would be possible to directly compare two theories to tell if they were equivalent to each other or different and whether they had testable consequences. That was a relatively new idea for physics to do that, not just by arguing but by providing mathematical proof.
He got through sixty-four of the eighty-one theories, and it turned out that there were only three distinct theories, so all these people were producing theories not even realizing they were the same theory as everybody else’s.
Two of the three theories were, in principle, untestable, meaning they had no observable consequences on the universe at all; the third one could be tested, but it would take 1031 years to see any observable consequence of the theory. That was a pretty depressing conversation for me, which probably tipped the balance—well, that and the mood of the grad students and the post docs!—towards going into computer science and going to California....MUCH MORE
