‘We Might Need To Regulate Concentrated Computing Power’: An Interview On AI Risk With Jaan Tallinn
One of the biggest and most interesting strategic problems that we’re faced with over the next few decades is the issue of artificial intelligence (AI), and its potentially existential risks to human civilization.....MUCH MORE
Humans, and organizations of humans, are currently the most powerful intelligent forces acting on the world. This is because we are the only forces that have the necessary general intelligence to have an impact on this scale. We have power over our environment, our rivals, and the future, because of our unique socially scalable intelligence. For now.
AI presents us with the possibility of powerful, non-human agency. What if machines could think and strategize and wield power? Would those machines participate benevolently in the human social order? What about when industrial production of silicon computing hardware, fine-tuning of the parameters and algorithms of software intelligence, and digitally precise coordination at scale allow those machines to outpace us in total coordinated intelligence? At that point, we would be faced by non-human intelligence powerful enough to be beyond human constraint, potentially able to scatter and outmaneuver us the same way we overcome our own rivals, if it were so inclined.
The question, then, of whether that machine intelligence would be benevolent, indifferent, or hostile, and the extent those concepts can be applied, is of supreme importance. If it’s weaker than us, control is relatively easy; if it starts doing the wrong thing, we can just shut it down or change its programming. But if it’s smarter and stronger than us, the reverse applies. We need to it be stably benevolent so that it’s still good when we can no longer control it. But when examined with rigor, the problem of building a stably benevolent machine intelligence turns out to be very difficult, once the possibility of superhuman intelligence and self-modification comes into play. There is usually some loophole in the motivation scheme that will lead it to break out of our implicit intentions in hard-to-predict ways. Building a dangerous and uncontrollable general intelligence seems much easier.
Further, the sheer interestingness and potential power of general intelligence mean that if such a thing is possible, it will probably be built, possibly this century. As such, we’re in a sticky situation: we’re probably going to build something of world-changing power that by default won’t be aligned with what we want for the future, and which we won’t be able to control, contain, or resist.
So, we need to figure out how to get this right to avoid an existential disaster. The capacity to get this right, in the philosophical, mathematical, engineering, and political dimensions, is therefore of high strategic priority, at least on a medium-term timescale. The most important immediate step is understanding the problem well, so that as we go about planning and preparing for the next few decades of social and political change, we do so with this need in mind. We need some idea of how to handle superhuman AI before the situation becomes critical.
To better understand the problem posed by AI, I recently talked to Jaan Tallinn, known best as the founder of Skype.
Jaan has spent the past 10 years meeting with key figures, funding important projects, and assembling strategic knowledge relating to AI risk, so he’s one of the best people to talk to for a well-informed and in-depth overview of the topic.
Going into this discussion, I was frustrated by the relatively shallow depth of most discussions of the topic available in non-specialist media. It’s mostly the same few introductory questions and arguments, aimed at a very general audience with low technical capacity, intended to titillate, rather than inform. To get a useful handle on the topic, so that it’s more than just a science fiction hypothetical, we need more technical depth.
I called up Jaan in the middle of the night, and we recorded a wide-ranging discussion of the background, technical challenges, and forecasts of the issue of AI risk, and what approaches we might take to navigate it. The following transcript of our discussion has been minimally edited for grammar and flow:
Wolf Tivy: You’ve expressed concern in the past about artificial general intelligence (AGI) as a strategic concern, or even a threat to humanity in the 21st century. So, I’d like to briefly go through the argument to get your thoughts on it. As I understand it, the AGI concern is not about deep learning. It’s about a more advanced capability—the ability to learn a causal model of the world from observation, and then plan and reason with that to accomplish objectives in the world. This is something that humans do that current software technologies don’t do, and deep learning doesn’t do.
Jaan Tallinn: I think it’s an open question. Current deep learning cannot do this, but the question whether deep learning will take us all the way or not is unsolved. There are people whom I respect on both sides of this argument, so I’m truly uncertain. I know people who have strong opinions that deep learning will take us all the way, and I know people who have strong opinions that it will not.
WT: So, the fear with AGI is that, like deep learning, it will be highly scalable with computing power and algorithmic progress, so that once we do have the core capability figured out, it could rapidly get away on us.
JT: It’s even worse. We might sort of “pull an evolution.” Evolution didn’t really know what it was doing; it was just throwing lots and lots of compute at a very simple algorithm, and that invented humans. So it’s totally possible, if not very plausible, that we might pull an evolution in that way, that we never really understand what happened. It seems unlikely, but I don’t really know how unlikely.
WT: That would be something that is on a gradient. You deliberately build an AGI, but it ends up doing things and working for reasons you don’t really understand. You can imagine a spectrum of possibilities between that, and building something that accidentally ends up being AGI.
JT: Evolution is a very useful anchor here. Ultimately, if we had enough compute, we could replay evolution. Of course evolution had, as I say, one trillion sunsets to work with. That’s a lot of compute. But we are also way smarter than evolution, so we don’t need that much.
WT: One way I’ve heard it put, I think in a previous interview with you, was that whatever comes out of this AGI thing would become the driving power of the future, in the sense that it would be the thing defining what the future is like....
