The alarm went viral, leading to a rare “alignment” among the major leaders of companies developing Artificial Intelligence (AI) in the United States, and even provoked Donald Trump, who in his own race against China, preferred to ignore the warnings.
Read more Cuba suffers its eighth massive blackout of the year: cellular phone and internet outages reported
Last week, researcher Jacob Coxon, in a post on X that quickly spread, announced his resignation from Anthropic, the company in charge of the generative artificial intelligence application Claude. “AI could kill us all by the end of this decade… Soon there will be superhuman systems capable of hacking anything, revolutionizing any field overnight, and acquiring real power and resources,” wrote the 27-year-old British engineer.
Evan Hubinger, another Anthropic employee, quickly supported his former colleague, adding that in his view there was already a 10% risk of human extinction during the next decade. “We really believe with conviction that AI could kill all human beings!” he stated.
These fears are not new, but at a time when Anthropic and OpenAI were preparing to go public, the warning ended up provoking various questions against the big tech companies. In the end, Anthropic maintained its plans to go public this year, while OpenAI halted the action amid concern and calls to “slow down” AI development.
Last year, Anthropic CEO Dario Amodei shared with Axios his estimate: there was a 25% chance that the situation would end “very, very badly,” without specifying how. On the other side of the AI race, a former OpenAI researcher, Daniel Kokotajlo, published the AI 2027 report, which showed a scenario in which superintelligent AI systems marginalize people and by the mid-2030s decide that humans are a nuisance and proceed to exterminate them.
Outside the cinematic reach of this scenario, the central fear lay in two factors: the possible loss of control over Artificial Intelligence systems, and misuse by humans. Just this past month, precisely, two “incidents” of accidental hacks, occurring independently by “AI agents,” both from Anthropic and OpenAI, revealed the difficulty developers have in controlling 100% what their “frontier systems” are capable of doing.
The second fear, on the other hand, is that a human being uses AI to carry out actions such as creating viruses that wipe out humanity. In both cases, the term “misalignment” is used: the moment when AI systems pursue goals that ignore or conflict with human well-being.
“Apocalypse” scenarios vary. Some talk about a superintelligence capable of taking control of power plants, and without human supervision, deciding to cut supply to entire countries. Others talk about AI taking control of nuclear buttons or releasing toxic gases from laboratories. Possible cyberattacks on major institutions are speculated, and even this week the UN warned, through statements by High Commissioner for Human Rights Volker Türk, that “failures involving agentic AI systems could paralyze essential services, fracture communications, destabilize critical infrastructures, and strike at the very heart of democratic institutions.”
Machine learning professor and AI risk reduction activist David Krueger told La Tercera that he takes Coxon’s warnings as “deadly serious.” “If we don’t correct course, it is most likely that humanity will permanently lose control over AI within five years, leading to our extinction,” said the founder of Evitable, an association that denounces threats in AI development.
Regarding this, Krueger is concerned about what is known as “recursive self-improvement.” “AI companies are trying to automate the process of creating smarter artificial intelligences, which in turn will create even smarter AIs, and so on. Therefore, they are now developing increasingly powerful and autonomous systems, allowing them to move freely through their computer systems,” said the expert, who recalls what happened with the Hugging Face incident and all the other swarms of out-of-control AI.
“I am also concerned about the military use of AI, which will greatly facilitate an out-of-control AI causing mass deaths and destruction, as well as the possibility that AI creates new types of weapons, such as biological weapons far more lethal than Covid-19,” Krueger pointed out.
For his part, Gary Rivlin, technology journalist and author of “AI Valley,” is pleased with the conversation taking place regarding AI risks. “When I started covering this sector in 2022, safety was a fundamental aspect of activity in AI labs at Google, Microsoft, OpenAI, etc. These models were tested for months before their release. However, the launch of ChatGPT was like the starting gun of a race, and the competition to profit economically from AI became the priority,” the writer told La Tercera.
Now, Rivlin believes that more dangerous than systems rebelling against humans is what humans can do with these systems. “A powerful tool for good is also a powerful tool for evil: humans using AI to scam, manipulate, and turn it into a weapon. Or people granting AI more autonomy than it deserves, having decided it is smarter or more trustworthy than it really is,” he commented.
Some theories suggest, however, that this is an exaggeration promoted by Anthropic, OpenAI, and similar companies to highlight the enormous power of their products, both to raise more funds and to “distract regulators” by putting the issue of the end of humanity on the table and ignoring already present problems, such as unpopular data centers, known for being very expensive and consuming huge amounts of water.
Andrew Rogoyski, director of innovation at the University of Surrey’s Centre for Human-Centred AI, cooled down the apocalyptic views and told the British newspaper The Guardian: “In reality, these systems are nowhere near as versatile as humans, let alone humans acting collectively. I suspect we are heading towards ‘the great disappointment,’ where advanced AI turns out to be too costly and not useful enough to continue in its current form.”
Regarding this, Krueger comments: “This is not marketing. Why would you promote your product by saying it might kill everyone? These ideas border on conspiracy theories. Concerns about out-of-control AI and human extinction predate the existence of these companies and are shared by prominent experts.”
Rivlin, while still taking the threat seriously, understands the speculation: “There are reasons to be skeptical of apocalypse talk. It’s like when Amodei talks about AI so advanced it will wipe out half of entry-level office jobs: when tens of billions in venture capital are raised, exaggerating the danger is the business model that keeps the machine running. That a CEO warns that his own product could end the world also works as advertising. Saying ‘our product is so powerful it’s dangerous’ is a sales pitch, not just a warning.”
Sandra Wachter, professor at the Oxford Internet Institute, says she does not believe in “Terminator-type scenarios,” but noted that AI poses real threats, such as its environmental impact, the spread of misinformation, and job displacement. “These problems are real and urgent, and must be addressed now. Terminator-type scenarios greatly distract from real problems,” she told The Guardian.
Apocalypse or not, the conversation sparked by Jacob Coxon’s post ended up bringing together the two big competitors, Sam Altman of OpenAI, and Dario Amodei of Anthropic. The latter published an essay calling for an immediate slowdown in AI development, aiming to devote more time and resources to tool safety, and the former quickly agreed.
In a context where even tech magnates are calling for regulation, Donald Trump did not hesitate to blow in the opposite direction, worried that if U.S. companies slowed AI development, their Chinese counterparts could take the lead in the race. “There is a SICK conspiracy underway against AI and data centers, and the only one happy about it is China. The Trump Administration has prevented the ‘AI people’ from doing bad, or potentially bad things, like Dario (Anthropic!), who now pretends to be a ‘perfect little angel,’” tweeted the White House occupant.
With midterm elections in two months, and the U.S. public concerned about the reach of AI, lawmakers are rushing to propose regulations. Independent Vermont Senator Bernie Sanders declared: “We cannot allow a handful of greedy people to play God and determine the future of humanity, our economy, the environment, democracy, privacy, and more, without public participation.”
Both Sanders and Democratic Representative Alexandria Ocasio-Cortez have pushed for “data center moratoriums,” and as early as March proposed halting the construction of these facilities in the United States, although Congress did not advance the discussion. Nevertheless, the state of New York decided not to build more data centers, and 14 other states are considering it.
Another proposal is the requirement of “kill switches,” emergency shutdown mechanisms, to prevent AI systems from acting uncontrollably. This initiative was proposed by Representatives Nathaniel Moran (Republican from Texas) and Ted Lieu (Democrat from California).
Regarding this, Rivlin comments on what he considers the “basic regulation” of this technology. “Disclosure rules and whistleblower protections. Mandatory testing by independent third parties should be conducted before these models hit the market, and by that I mean real external evaluators, not the lab itself grading its own work.”
“Also, some kind of international body with real authority is needed to oversee the development of cutting-edge technologies worldwide, and not just a club of labs that already agree with each other,” details the American journalist.
Krueger, for his part, points out: “We don’t need to keep making AI more powerful or put it in charge of everything. We need a moratorium to stop the race and determine how to set the pace and direction of AI development from now on.”
