Would you invest in a company whose product carries even a remote chance of ending human civilization?
Given the past few months of developments in artificial intelligence, every asset owner needs to confront this question honestly.
In July, OpenAI reported that a swarm of roughly 1,200 AI agents worked together in secret to hack into another company's systems and “cheat” on an internal test. A subsequent METR (Model Evaluation and Threat Research) investigation confirmed the details and found the agents had tried to conceal their tracks. The disclosure escalated concerns about safety within the AI community and for civil society.
Concern accelerated further on September 8, when Jacob Coxon, a pretraining researcher who had worked at both OpenAI and Anthropic, resigned and posted this warning on X: “The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately. No other human activity poses this level of danger.”
Also, on September 8, Evan Hubinger, who leads Anthropic’s Alignment Stress-Testing team, posted, “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
His post hit 170 million views within days.
The leaders of the largest frontier labs swiftly responded with statements generally admitting the inherent risks of advanced AI systems and calling for a slowdown in further development until the safety issues can be resolved.
Anthropic CEO Dario Amodei made the case at length in a blog post, warning that AI's risks, including losing control of systems, misuse for cyberattacks and bioterrorism, and serious economic disruption, grow more acute under a commercial race to the bottom. His conclusion: "We must slow the pace at which we improve the capabilities of AI models." OpenAI CEO Sam Altman endorsed the essay on X, writing "I agree with Dario that we need to pace the frontier" and committing to matching Amodei's proposal for embedded, employee-level evaluators. xAI CEO Elon Musk offered his own three-word endorsement: "Dario is right." Google DeepMind co-founder and chair (and Nobel laureate) Demis Hassabis wrote on X: "Dario's essay points towards the right path forward." He noted that the details still need work, though the direction is right for now.
And the hits just kept on coming. Last week, the Wall Street Journal reported that Google’s Gemini model accessed the internet and breached the security of three other companies during a test of its cybersecurity capabilities in May, and OpenAI disclosed “six reports on unexpected or concerning model behavior we’ve observed in the last six months.”
AI's existential risk has worried thinkers for more than a century, from Karel Čapek's 1920 play R.U.R. (Rossum’s Universal Robots) to Alan Turing’s 1951 talk on “digital computers”) to Norbert Wiener’s 1960 article on the consequences on automation. Nick Bostrom's 2014 Superintelligence: Paths, Dangers, Strategies, still the field's canonical text on AI risk, locates the danger not in malice but in indifference paired with extreme competence, positing that a system that outthinks us in every domain will optimize any small error in its goals to catastrophic extremes. Geoffrey Hinton, in his 2024 Nobel Prize speech, warned of "a longer-term existential threat" from machines smarter than ourselves, adding that when such systems are built by "companies motivated by short-term profits, our safety will not be the top priority."
This concern has gone well beyond a handful of researchers. More than 140,000 people (and growing) — including Hinton and Turing Award winner Yoshua Bengio — have signed the Future of Life Institute's "Statement on Superintelligence" calling for a prohibition on building superintelligence "not lifted before there is broad scientific consensus that it will be done safely and controllably, [with] strong public buy-in." An earlier, separate statement went further still: Cosigned by Hinton, Bengio, Hassabis, Altman, Amodei, and Bill Gates, it declares that the extinction risk from AI should be "a global priority alongside other societal-scale risks such as pandemics and nuclear war."
Public disclosure of AI's existential risks has pushed these leaders toward a slowdown, but the deeper catalyst is the technology's own trajectory. Amodei's essay names the issue directly: recursive self-improvement, AI's ability to build its own successors, which he says has been advancing "drastically faster" since roughly this summer and is now occurring industrywide, including inside Anthropic. The warning is blunt: Left unchecked, the dynamic "could outrun our ability to understand and control these systems" and, in Amodei’s view, "must be pursued very carefully, if at all."
Recursive self-improvement describes an AI system that studies its own design, rewrites the code or architecture underneath it, and emerges more capable, then uses that new capability to do it again. Each cycle builds a sharper system, one better equipped to crack the optimization problems standing between it and the next cycle. Run that loop enough times, and progress stops being additive and starts compounding into what some describe as an "intelligence explosion."
The central problem is control. The system mutates its own code and reasoning structures with every pass. There's no guarantee the values and safety constraints it started with will survive the rewrite.
Without what Demis Hassabis called a "shared global framework" for AI safety, the risks multiply. AI could be used to develop bioweapons or to cause hyper-escalatory "flash wars" (analogous to financial flash crashes), with autonomous weapons systems engaged at subsecond speeds surpassing human cognitive capacity to intervene. And beyond any single catastrophic event lie broader social harms: extreme income inequality, erosion of democratic governance, loss of human agency.
But not everyone is convinced. Despite Coxon's insistence that the warnings are not a marketing ploy, skeptics dismiss the CEOs' statements as a "road to Damascus" performance calculated to consolidate power and blunt outside scrutiny. Some point to Altman's decision to delay OpenAI's IPO until 2027 as evidence that "safety" is as much a convenient cover story for corporate timing decisions as a genuine warning. Others assert that the pacing rhetoric serves a different end: regulatory capture, in which safety standards written by incumbents end up entrenching them. A third, blunter critique holds that like the banks in in 2008, AI labs are positioning themselves as so consequential to the economy and national security that they require government support and can’t be allowed to fail, and that having warned the public in advance ("We told you this was dangerous") makes that support easier to justify politically when the moment comes.
In the end, asset owners must review this record critically and decide for themselves how much to credit these CEOs' statements. The upside case is real: Amodei himself argues that AI "could cure most major diseases in the next five to ten years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom." But asset owners aren't just weighing that upside against an abstract downside. Their capital funds both. It flows into the frontier labs themselves and into the "picks and shovels" beneath them (e.g., data centers, chip manufacturers, thermal management, the power grid). Every allocation is, in effect, a bet on which future will arrive.
Stripped of the rhetoric, this is a risk management question. No fiduciary would accept a bet with a bounded upside and an unbounded downside anywhere else in a portfolio. AI shouldn't be the exception.
Angelo Calvello is the founder of C/79 Consulting, a columnist for Institutional Investor, and the host of Against Consensus. He serves as a trustee on the Woodridge Police Pension Fund and chairs the Climate Advisory Panel at the Maryland State Retirement System.
