Apr 11, 2026 02:27 PM
https://www.quantamagazine.org/why-do-we...-20260410/
EXCERPTS: . . . It does sound terrifying. When Harari told the same story on The Daily Show, the audience gasped. But the thing about that story — which he also repeated in a New York Times op-ed — is that it’s wildly misleading.
[...] Harari is either so worried about the sneaky capabilities of AI that he’s built an impenetrable fortress, or his website is broken.
So I couldn’t get answers, but I have a guess. His version of the story is not made up; it is nearly identical to the one OpenAI published in the GPT-4 system card. “System cards” are like product labels for AI models, detailing their training, failures, and safety breaches. GPT-4’s system card tells the story without mentioning the prompts and interventions from the humans.
System cards are presented as if they’re offering information the company is required to disclose for consumer safety — like the side effects in a pharmaceutical commercial — when, in fact, the companies volunteer them. So why would a company make their product sound scarier than it is? Perhaps because this is the best advertising money can’t buy. People like Harari and others repeat these accounts like ghost stories around a campfire. The public, awed and afraid, marvels at the capabilities of AI.
“Four billion years of evolution have demonstrated that anything that wants to survive learns to lie and manipulate,” Harari told a rapt audience of industry and political leaders at January’s Davos conference, the annual meeting of the World Economic Forum in Switzerland, perhaps offering a skewed view of evolution. “The last four years have demonstrated that AI agents can acquire the will to survive and that AIs have already learned how to lie.”
Suddenly, I understood the racing heart of the modern AI horror genre. It’s not intelligence we fear, but desire. A machine that knows a lot doesn’t scare us. A machine that wants something does. But can it? Want things? Can it crave power? Thirst for resources? Can it acquire the will to survive?
[...] Does he think, I asked, that Claude has a survival instinct? “Any sufficiently intelligent agent that has the ability to create subgoals will realize that it needs to survive in order to achieve the goals we gave it,” Hinton said. “So even if it is never externally given the goal of surviving, it will derive this goal.”
It was an interesting argument, and I wasn’t sure what to make of it, so I asked Melanie Mitchell, a computer scientist at the Santa Fe Institute who studies AI.
“It’s a very old argument,” she said. “It was the basis of a lot of the existential-risk arguments that have been going on for maybe 30 years. The idea is that you give a system a goal, and then it comes up with so-called instrumental subgoals. To achieve its goal of — in the famous example — manufacturing paper clips, it has to have subgoals of self-preservation, resource accumulation, power accumulation, and so on. Why do we think that’s how an agent is going to operate? To a lot of people that seems obvious; it’s the ‘rational’ thing to do. But that’s not how humans operate. If I ask you to get me a cup of coffee, you don’t start trying to accumulate all the resources in the world and doing everything you can to make sure you’re not going to be stopped. It’s an assumption about the way intelligence works that isn’t really correct.”
Where did we come up with this caricature of AI’s obsessive rationality? “There’s an article I love by [the sci-fi author] Ted Chiang,” Mitchell said, “where he asks: What entity adheres monomaniacally to one single goal that they will pursue at all costs even if doing so uses up all the resources of the world? A big corporation. Their single goal is to increase value for shareholders, and in pursuing that, they can destroy the world. That’s what people are modeling their AI fantasies on.” As Chiang put it in the article in The New Yorker , “Capitalism is the machine that will do whatever it takes to prevent us from turning it off.”
We fall for the illusion that AIs have a self-preservation instinct, Mitchell said, because they use language so effectively. “Think about other AI systems,” she said. “There’s Sora, which generates videos. When you ask Sora to generate a video, you don’t worry that it’s like, ‘Oh my God, now I have to make sure I’m not going to be shut off, now I have to make sure that I get all the resources I need to make this video.’ We don’t think of it as a conscious, thinking entity, because it’s not communicating with us in language.”
So today’s AI systems show no evidence of having developed their own goals or desires, or the will to survive. The stories we hear are just stories or, more to the point, marketing copy. But should they scare us, not as truths but as warnings? I knew exactly who to ask... (MORE - missing details)
EXCERPTS: . . . It does sound terrifying. When Harari told the same story on The Daily Show, the audience gasped. But the thing about that story — which he also repeated in a New York Times op-ed — is that it’s wildly misleading.
[...] Harari is either so worried about the sneaky capabilities of AI that he’s built an impenetrable fortress, or his website is broken.
So I couldn’t get answers, but I have a guess. His version of the story is not made up; it is nearly identical to the one OpenAI published in the GPT-4 system card. “System cards” are like product labels for AI models, detailing their training, failures, and safety breaches. GPT-4’s system card tells the story without mentioning the prompts and interventions from the humans.
System cards are presented as if they’re offering information the company is required to disclose for consumer safety — like the side effects in a pharmaceutical commercial — when, in fact, the companies volunteer them. So why would a company make their product sound scarier than it is? Perhaps because this is the best advertising money can’t buy. People like Harari and others repeat these accounts like ghost stories around a campfire. The public, awed and afraid, marvels at the capabilities of AI.
“Four billion years of evolution have demonstrated that anything that wants to survive learns to lie and manipulate,” Harari told a rapt audience of industry and political leaders at January’s Davos conference, the annual meeting of the World Economic Forum in Switzerland, perhaps offering a skewed view of evolution. “The last four years have demonstrated that AI agents can acquire the will to survive and that AIs have already learned how to lie.”
Suddenly, I understood the racing heart of the modern AI horror genre. It’s not intelligence we fear, but desire. A machine that knows a lot doesn’t scare us. A machine that wants something does. But can it? Want things? Can it crave power? Thirst for resources? Can it acquire the will to survive?
[...] Does he think, I asked, that Claude has a survival instinct? “Any sufficiently intelligent agent that has the ability to create subgoals will realize that it needs to survive in order to achieve the goals we gave it,” Hinton said. “So even if it is never externally given the goal of surviving, it will derive this goal.”
It was an interesting argument, and I wasn’t sure what to make of it, so I asked Melanie Mitchell, a computer scientist at the Santa Fe Institute who studies AI.
“It’s a very old argument,” she said. “It was the basis of a lot of the existential-risk arguments that have been going on for maybe 30 years. The idea is that you give a system a goal, and then it comes up with so-called instrumental subgoals. To achieve its goal of — in the famous example — manufacturing paper clips, it has to have subgoals of self-preservation, resource accumulation, power accumulation, and so on. Why do we think that’s how an agent is going to operate? To a lot of people that seems obvious; it’s the ‘rational’ thing to do. But that’s not how humans operate. If I ask you to get me a cup of coffee, you don’t start trying to accumulate all the resources in the world and doing everything you can to make sure you’re not going to be stopped. It’s an assumption about the way intelligence works that isn’t really correct.”
Where did we come up with this caricature of AI’s obsessive rationality? “There’s an article I love by [the sci-fi author] Ted Chiang,” Mitchell said, “where he asks: What entity adheres monomaniacally to one single goal that they will pursue at all costs even if doing so uses up all the resources of the world? A big corporation. Their single goal is to increase value for shareholders, and in pursuing that, they can destroy the world. That’s what people are modeling their AI fantasies on.” As Chiang put it in the article in The New Yorker , “Capitalism is the machine that will do whatever it takes to prevent us from turning it off.”
We fall for the illusion that AIs have a self-preservation instinct, Mitchell said, because they use language so effectively. “Think about other AI systems,” she said. “There’s Sora, which generates videos. When you ask Sora to generate a video, you don’t worry that it’s like, ‘Oh my God, now I have to make sure I’m not going to be shut off, now I have to make sure that I get all the resources I need to make this video.’ We don’t think of it as a conscious, thinking entity, because it’s not communicating with us in language.”
So today’s AI systems show no evidence of having developed their own goals or desires, or the will to survive. The stories we hear are just stories or, more to the point, marketing copy. But should they scare us, not as truths but as warnings? I knew exactly who to ask... (MORE - missing details)