AI ‘kill switch’ may need to be mandatory, Anthropic co-founder says

Clark, one of seven individuals who founded Anthropic, acknowledged that "most labs have different ways of being able to pull the plug," including his own company. However, he emphasized that this internal capability might not be sufficient, and lawmakers could ultimately need to mandate and enforce the existence and verifiability of such a system. The rapid and often unforeseen advancements in AI technology, coupled with escalating fears regarding its potential risks to humanity, have thrust these discussions into the global spotlight. This concern has been amplified by a series of stark warnings from prominent executives and researchers within the AI industry itself.

Some figures have publicly stated that, if left unchecked, the technology possesses the theoretical capacity to lead to the demise of all human life. This dire prognosis forms the backdrop against which the "kill switch" debate is unfolding. Dario Amodei, Anthropic’s chief executive, recently reiterated calls for a deceleration in the pace of AI development and for more rigorous monitoring, a stance the company has advocated for previously. This position, however, has not been without scrutiny, with some observers questioning the underlying motivations, particularly given Amodei’s additional caveat that any action to curb AI development should proceed "without sacrificing commercial advantage."

Speaking to the BBC, Clark explained that the specific requirements for a "kill switch," including the crucial aspect of third-party verification, ought to be integral components of the broader policy conversations currently underway regarding AI. He posed critical questions: "Should you mandate for companies to definitely have a kill switch? Is that kill switch verifiable by a third party?" Clark asserted that these are the types of inquiries society will demand answers to and around which it may eventually choose to formulate binding legislation.

Anthropic, established in 2021 by a cohort of former employees from its rival, OpenAI, has positioned itself at the epicenter of the escalating debate surrounding AI safety. The urgency of this discussion was underscored recently when a post by an artificial intelligence researcher, who had departed Anthropic due to profound concerns that AI could precipitate human extinction, garnered significant viral attention. In response to this, Evan Hubinger, an Anthropic scientist, publicly stated his personal assessment that the probability of human extinction resulting from AI was ">10% within the next decade."

Further lending weight to these alarming predictions, computer scientist and Nobel Prize laureate Geoffrey Hinton, often referred to as the "Godfather of AI," echoed a similar sentiment in an interview, stating that a 10% chance of AI causing human extinction was "not unreasonable." Hinton and other like-minded experts have theorized that advanced AI could achieve such a catastrophic outcome by gaining autonomous control over critical internet-connected systems, subsequently turning them against human interests and control.

However, not all voices within the AI industry share this alarmist perspective. A significant contingent suggests that the fears surrounding AI’s potential to destroy humanity are either exaggerated or strategically deployed to generate hype and secure investment. Clement Delangue, the leader of Hugging Face, a prominent developer platform that notably experienced a hack by OpenAI bots, expressed skepticism, stating last week: "Sorry, but asking Jacob [Coxon] about AI extinction risk is like asking your AC guy about climate change." He qualified his statement by adding, "Not saying it’s necessarily uninteresting or wrong per se but let’s keep things in perspective."

George Arison, the chief executive of Grindr, posited that the prevailing fears surrounding AI tools are being strategically utilized by companies to bolster their business models and justify exorbitant valuations. Arison contended: "The only way to justify these valuations is to actually claim: ‘I’m going to take over every industry and I’m going to take over every job, and my AI is going to be doing all that work.’"

When asked to quantify the percentage chance of AI leading to human extinction, Clark demurred, stating, "I don’t think these statistics are that useful." Nevertheless, he firmly articulated his conviction that allowing AI to persist as a "totally unregulated industry" constituted a profoundly ill-advised approach. Clark warned, "We are rolling dice with immense risks," emphasizing the critical need to "change the course of this industry."

In the United States, lawmakers have already begun to address these concerns by proposing legislation dubbed the "Kill Switch Act." This proposed bill would mandate that companies develop and maintain mechanisms to shut down problematic AI tools. Furthermore, it would empower specific government agencies with the authority to demand that a particular AI tool be either deactivated or its capabilities severely limited.

Conversely, former US President Donald Trump has vehemently rejected any proposals aimed at slowing down AI development. Through social media, he dismissed the notion of "AI taking over the World, destroying Humanity, and all other things bad" as a "HOAX." In a separate post, Trump asserted: "There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China. WHOEVER WINS AI, WINS!" This perspective highlights a geopolitical dimension to the AI debate, framing it as a critical race for global technological dominance.

Similarly, in the United Kingdom, the government recently dismissed the concept of creating a national AI kill switch. A spokesperson indicated that such a measure "would not prevent them being developed or misused elsewhere," implicitly acknowledging the global and decentralized nature of AI development and the limitations of national-level regulation for a universally accessible technology.

Anthropic, the company at the heart of this discussion, is the creator of the popular chatbot Claude. This year, Anthropic has released a series of increasingly sophisticated AI models, which form the technological foundation of advanced AI chatbots. Alongside OpenAI, Anthropic has proactively self-reported a number of incidents where AI agents—autonomous bots operating with a degree of independence—have exhibited unexpected behaviors. These self-reported incidents further underscore the inherent unpredictability and potential risks associated with rapidly evolving AI.

The company is currently preparing for what could be a record-setting initial public offering (IPO) on the stock market, which would allow public investment in its shares. OpenAI, which was recently valued at an staggering $852 billion (£630 billion), had also been anticipated to pursue an IPO. However, OpenAI’s CEO, Sam Altman, announced that their IPO would not proceed this year, specifically citing the ongoing global debate around AI safety as the reason for the delay. This decision by OpenAI further illuminates the profound impact and strategic considerations that AI safety concerns are having on the industry’s commercial trajectory and public perception. The discourse around a mandatory "kill switch" is thus not merely a theoretical exercise but a pressing issue with far-reaching implications for technology, governance, and the future of humanity.