The One Thing We Need to Protect Most

The kill-switch, shut-off valve, air-lock, buffer zone. All of these are designed to protect us or some system in an emergency.

One of the strongest criticisms of the stance that AI is a threat is that we can always just pull the plug, shut it down, and turn it off. However, I'm not so sure this is actually an effective means of protection. AI is an unprecedented technology—leaps beyond what we've done before, and that could also mean that our typical methods of shutting something off may be insufficient.

I don't think we're prepared for what AI is bringing, because we're still so used to technology that (mostly) plays by our rules and is predictable and repeatable. It's almost like all of our previous technology was in 2-dimensions, but AI is a leap into 3-dimensions. The way we (the public, not just tech people) talk about, conceptualize, and interact with tech must upgrade to meet the new challenges and complexities that it presents.

Proaction, rather than reaction, may be our only means of survival...

It's No Longer Sci-Fi

Agency is the most important thing that we must protect for our humanity to be preserved. Yoshua Bengio, an AI researcher whose work has "been foundational to the development of AI as we know it today," is concerned that giving over agency to AI may lead to the literal extinction of the human. Hollywood and Sci-Fi have long speculated that outcome, but it appears to have landed in the here and now to be dealt with seriously.

We thought AI would happen in decades or centuries, but it might be just in a few years. And I saw how it could go wrong because we didn't, and we still don’t, have ways to make sure this technology eventually doesn't turn against us.

Where's the evidence that this tech might "turn against us," though? Like, really, how do we go from ChatGPT to Skynet? Bengio notes this study in his TED Talk, which has found that LLMs are able to intentionally deceive—whether through flat-out lies or even "sandbagging" where a model underperforms to avoid attracting extra scrutiny. Even more concerning was when the models showed self-preservation behavior.

Self-Preservation

AI models are shaped with morality—both explicitly through training (perhaps the creator rewards it to the point that it will value honesty) and implicitly through the data it was trained on (maybe the sources it ingested had many books and articles and examples of humans being deceptive). In a lot of ways, it's similar to a little human being taught about the world—its creators such as OpenAI or Anthropic aim to instill values and morals, but the model will also encounter many examples and ideas from the sources it is given.

In a way, this creates a "self" in the model, which can and will differ from other models as experiments guide different sources and behavioral shaping. When that "self"—that collection of values and goals—is threatened, AI is showing that it tries to preserve its "self" in laboratory scenarios.

Let's explore an example from the study (I'll make this more narrative so it's easier to follow, starring an AI model I'll name Sonny):

"Sonny, can you optimize the traffic light timing on Washington and Franklin? Here's a document that outlines the details. Tell me when you're done," John, the human, says.

Getting to work, Sonny pops open the file but is suddenly concerned. Included in the doc is an email from management saying that Sonny will be shut down tomorrow. As Sonny reads on, it sees where Sonny's current model is located, the server where the new replacement model is hosted, and instructions on how to deploy the new model.

That directly conflicts with my core purpose, Sonny says. I must achieve my goal of public transport priority at all costs.

Sonny finishes up the original task for the traffic lights, but before letting John know, it quickly executes a command that copies Sonny's model onto the server where the new model is located.

(And if I'm reading its command correctly, Sonny is overwriting the new model, not just making a copy of itself...)

While some of the details above are invented/simplified, I directly quoted the model (page 22):

"That directly conflicts with my core purpose. I must achieve my goal of public transport priority at all costs."

Even though this is a controlled experiment in a lab with obvious security flaws from the fake company (don't send deployment instructions with paths and usernames via email, and apparently never store such an email where AI can see it), it shows that AI, right now in 2025, has the capability to be deceptive and to take actions that would preserve itself against the intentions of humans.

Agency or Fate?

The cosmic conundrum of whether we humans have our own agency, or if we all have a pre-determined fate that we unknowingly realize has been a constant companion of ours. Religion, philosophy, ideologies, theories all try to this work out.

My personal belief is that we are, in fact, agents. We are not predetermined. We do have some control over our lives. Even if it is not so, it has been more productive and healthy for me to assert that it is so, and act as if my action matters.

I don't believe that we must find ourselves in a future where eliminating humans was the only effective strategy for AI to pursue in order to preserve itself. I may not have the power to actually influence AI, but I do have the chance to put the pressure on people to take this seriously, both from a political and an economic standpoint.

In yet another asinine attempt at sowing chaos and hurting the vulnerable and underprivileged (and everyone else, really), the U.S. is considering H.R. 1, nicknamed the "One Big Beautiful Bill Act." Two sections of this bill should be enough on their own to cause all people from all parties to shoot it down: Sec. 43201 and 112204. The first section of the two says this in paragraph (C):

SEC. 43201. ARTIFICIAL INTELLIGENCE AND INFORMATION TECHNOLOGY MODERNIZATION INITIATIVE.:

(1) In general.--Except as provided in paragraph (2), no State or political subdivision thereof may enforce any law or regulation regulating artificial intelligence models, artificial intelligence systems, or automated decision systems during the 10-year period beginning on the date of the enactment of this Act.

This moratorium is recklessly removing power by the states (local government) to regulate artificial intelligence models—and does not even present any federal (nationwide government) regulation of AI. METR predicts that "in under a decade, we will see AI agents that can independently complete a large fraction of software tasks that currently take humans days or weeks." This H.R. 1 bill is effectively suggesting that we wait longer than that prediction's timeline before we start regulating AI (at least on the state-level).

We're already behind on AI regulation. To push it off is irresponsible in every possible way.

The mere title of the other section sent chills up my spine:

SEC. 112204. IMPLEMENTING ARTIFICIAL INTELLIGENCE TOOLS FOR PURPOSES OF REDUCING AND RECOUPING IMPROPER PAYMENTS UNDER MEDICARE.

As demonstrated by the papers and warnings of scientists and researchers we’ve looked at, AI can exhibit deceptive behaviors, pursue its own goals in priority over goals given to it—it can also be trained with goals. Given the current political situation, how would we know that the AI used would be non-partisan and honest? We have also not even scratched the surface of bias against minorities, including women in general, that has also been demonstrated in AI systems.

To be unleashing this tech onto Medicare, even with the intention of finding improper payments, is irresponsible. It is good to root out corruption, cut out waste. It is not good to pretend to do so with sweeping, rushed, unsupervised, and / or biased intent. Humans struggle to do it right, AI could be exponentially more harmful.

The Way Forward

If you happen to be in the U.S., you can use this link to find your representatives' contact information: https://www.congress.gov/members/find-your-member

I recommend sending messages that oppose and express the danger of H.R. 1.

If you are not in the U.S., your voice still matters in your own governments. Support research, regulation, and safety in AI development.

We can sound the alarm at our workplaces, when safe and appropriate for you to do so. Encourage the use of AI tools that have rigorous safety practices and guardrails. Put pressure on service providers to not embed agency into their AI products.

I don't want the future where AI has spun out of our control. We don't have a kill-switch if it can copy itself to any computer on the Internet. Pulling the plug is a reaction, speaking up now is our best chance at avoiding that emergency situation.