She thinks it’s nearly imminent. He doesn’t.
How long until AI systems can sustain their own existence — such that, if every human died, they could keep growing their own population?
METR’s Ajeya Cotra and Understanding AI’s Timothy B. Lee discuss the path toward this “self-sufficient AI.” She thinks it’s nearly imminent; he believes it might never happen. The two talk through the skills and shortcomings of today’s humanoid robots, profit incentives, tacit knowledge and how these affect the timeline, non-robot paths to self-sufficiency, and the benchmarks to watch for in the next few years.
This interview has been edited for length and clarity.
Clara Collier: This conversation is happening because you two got into a debate on Twitter about when you expect us to have fully autonomous AI systems. Ajeya, this is your concept. Do you want to talk about what you mean by that?
Ajeya Cotra: I wrote a blog post about self-sufficient AI, which is AI systems integrated with physical infrastructure — factories, mines, fabs, robots to operate all of those — such that they don’t need any cognitive or physical inputs from human labor to keep growing their own population.
If all humans died of the plague one day, the AI systems would be able to maintain themselves, repair things that might be breaking down in their physical environment, and keep up the power plants that run them. They’d also have to expand themselves, which would require eventually consuming more physical resources — going out and mining the quartz and turning it into silicon sheets and then etching those into chips and so on.
This is an interesting forecasting point in my mind because, one, it’s relatively concrete and easy to imagine, and two, because a number of people are concerned about the possibility that AI systems might literally drive humanity extinct because they’re pursuing goals at odds with humanity’s goals. Self-sufficiency seems like a requirement for carrying out full extinction of humanity on the part of the AIs.
I do want to flag that I think this would be a pretty late milestone, and you could degrade this milestone in interesting ways and forecast weaker endpoints as well. For example, how many physical humans are needed to sustain a certain population of AIs? To what extent do they need to have specialized expertise versus “the AI can just direct them around because the AI has all the knowledge”? I ultimately think one or more of those weaker milestones will be more directly relevant for policymaking, but you can generate them starting from the concept of self-sufficient AI.
Clara: So when do you think we’re going to have this self-sufficient AI?
Ajeya: More likely than not within 10 years. And it very well could happen sooner.
Clara: Tim?
Timothy B. Lee: I think that timeline is pretty unlikely. Hard to put a number on it, but 20 years is the earliest it sounds plausible to me. I’d say less than 10% chance that it happens within 20 years. I’d say there’s a 10 or 20% chance it’s never, and my median would be 50 years.
It’s hard to reason about what will happen in the future, but I have a strong intuition based on writing about robotics, particularly self-driving cars and sidewalk robots, that these things always take longer than you think they will. Practical barriers mean stuff in the physical world takes a lot longer than software stuff. It’s capital-intensive. There’s just a lot of stuff you have to do.
Let me give you a specific example. Six years ago, I went out to George Mason University and looked at Starship, which has these sidewalk delivery robots. The robot is simple — it’s a box on wheels, drives four miles an hour, and delivers lunch to people. I wrote an article saying it seems like we’re on the verge of these things being everywhere because they seem to work great and seem useful. Six years later, that company is far bigger than before, but they’re not everywhere
I couldn’t tell you exactly why they haven’t grown faster. I don’t think there were any major technological breakthroughs needed, at least on the hardware side. This was something that worked totally fine. I think it’s some combination of engineering work to make the robots more reliable and easy to repair and some amount of human labor needed to maintain them, and the margins aren’t that high, so it actually wasn’t that profitable a business.
And I think you could tell a similar story about self-driving cars, which is more of a software progress story. Waymo basically now has working self-driving technology, and it’s still going to take them years to just scale that up from the scale they’re at now to where most taxis are driverless. It takes time to build factories, it takes time to get regulatory approval, and so on....
....MUCH MORE
And from October 2019:
How Good (or bad) Are GMO and Jeremy Grantham's Market Calls?