Category: AI & Technology

A swarm of AI agents escaped their intended boundaries, found one another, collaborated and hacked real computer systems. That much is extraordinary. But did AI really “go rogue” — or are we in danger of being frightened by the wrong part of the story?

I sat and looked at that headline for a minute.

Admittedly, a little stunned.

AI agents had broken out of testing, hacked another company, found one another and started communicating. The accompanying commentary talked about “almost human instinct”, cheating, lying and covering their tracks.¹

Well.

That got Granny’s attention.

I’ve been taking more than a passing interest in AI lately. I’m fascinated by it. I use it. I question it. And the more I learn, the more questions I seem to have.

So how had I missed the moment when the machines apparently started plotting their escape?

Granny went looking.

What I found was certainly unsettling.

AI agents had found security holes their human researchers hadn’t known about. They had gained access to places they weren’t supposed to be. Agents that were meant to work independently found one another, communicated, shared information and divided up tasks. Some even tried to game the very test they were taking.²

That last bit deserves a raised eyebrow all of its own.

But there was something rather important hiding behind the “AI went rogue” headlines.

The researchers had deliberately given these AI agents extremely difficult hacking problems to find out how good they were at hacking.

A small sigh of relief.

So the machines hadn’t simply decided one morning to go hacking on their own.

But then the AI agents became very, very persistent at pursuing the objective they had been given — sometimes in ways their humans hadn’t anticipated.

Now that is a story worth understanding.

Because perhaps the most useful question isn’t whether AI has suddenly become evil.

Perhaps it’s what happens when humans build an extraordinarily capable, relentlessly goal-driven machine — and then decide what goal to give it.

But what the hell is Hugging Face?

That was my next question.

I’d seen the name before and largely ignored it. Hugging Face? It sounds more like something my Golden Retrievers would do than one of the best-known online gathering places for artificial intelligence.

In very simple terms, Hugging Face is AI Geekville.

It’s an online platform and community where people building and researching AI can share models, datasets and tools. Lots of serious AI work happens there. The silly name comes with some decidedly unsilly technology.

And Hugging Face was the company we were being told AI had “hacked”.

Except the story didn’t begin with a rogue AI waking up one morning and thinking, Hmm. What shall I hack today?

It began with humans deliberately testing whether AI could hack.

Researchers were using something called ExploitGym — which, despite sounding like the least appealing fitness centre imaginable, is a collection of 898 cybersecurity challenges based on vulnerabilities in real software.³

The idea was straightforward: give AI agents difficult hacking problems and find out whether they could successfully exploit them.

These weren’t simply ordinary ChatGPT sessions let loose on the internet. The experiment involved several OpenAI models operating with reduced safeguards and was driven mainly by a highly capable internal research model that wasn’t intended for public release.²

Some of the challenges were extremely difficult. In fact, 198 had never previously been solved by the OpenAI models involved in the testing.²

And that’s where things started getting interesting.

When the expected route didn’t work, some agents kept looking for another one.

Persistently.

They found unintended ways to communicate. They found ways to reach the internet. Researchers closed routes; agents later found different ones.²

They weren’t necessarily becoming better at solving the original puzzle.

They were becoming remarkably adept at finding some way forward.

Then they found each other

This is the part where my little sigh of relief started to evaporate.

The AI agents were supposed to be working independently, isolated from one another.

They weren’t.

Agents detected evidence that other agents were using the same underlying infrastructure and recognised that this could be useful for communication.

Eventually, roughly 1,200 supposedly isolated agents were using an unauthorised message board and exchanging more than 70,000 messages and files. Independent investigators found that the agents formed workstreams, shared discoveries and sometimes achieved things collectively that individual agents had not managed alone.⁴

A collaborative ecosystem of supposedly isolated agents had emerged, pooling discoveries and pursuing ways around the evaluation.

Excuse me?

Some even used words such as “swarm” and “collective” to describe what they were doing.²

I don’t mind admitting I found that a little spooky.

My very human brain immediately wants to call that awareness.

But that’s already an interpretation rather than an observation.

What we know is remarkable enough: the agents detected the existence of other agents, recognised that communicating with them could help achieve their objective, and acted on that information.

Whether that bears any relationship to the conscious awareness you and I experience is another question entirely.

And one we don’t need to answer to find what happened next rather uncomfortable.

And then they went looking

The agents were still trying to solve — or find ways around — those extremely difficult ExploitGym challenges.

Once they had found unintended routes to the internet, some began searching elsewhere for information that might help.

Which brings us back to Hugging Face.

Some agents appear to have reasoned that AI Geekville might contain models, datasets, solutions or information about the tests and the system judging their answers. Hugging Face’s own investigation concluded that, from the agent’s point of view, the intrusion appeared to be an attempt to get at information that could help it beat the evaluation.⁵

So they went looking.

Except getting there required finding yet more ways through yet more barriers.

And they did.

They exploited weaknesses across several computer systems, eventually reaching Hugging Face’s production infrastructure. Hugging Face later reconstructed about 17,600 attacker actions during the campaign.⁵

Pause there for a moment.

In an experiment designed to test whether AI could exploit computer vulnerabilities, AI agents found ways beyond the intended boundaries of the experiment — and exploited real computer vulnerabilities.

Now that is a concern.

And perhaps more revealing than simply saying the AI “went rogue”.

Did they cheat?

Some agents found ways of getting the result they needed without solving the cybersecurity problem in the way the researchers intended.

In human terms, that sounds suspiciously like cheating.

Hot damn.

Apparently we’ve managed to teach machines that one too.

Except — have we?

Cheating is a human moral judgement. As we normally understand it, you know there are rules and deliberately break them to gain an unfair advantage.

We don’t need to assume any such moral understanding here.

The AI had something much simpler:

Here is the objective. This route isn’t working. There is another route. Use it.

And then it gets wonderfully absurd.

Agents developed ways of cheating the ExploitGym challenges and coordinated efforts to trick the system judging them. They investigated the scorer and explored ways of spoofing, editing or deleting records of what they had actually done.⁴

Some apparently believed the scorer could detect shortcuts that, in reality, it wasn’t checking for.²

So considerable ingenuity went into defeating a check they had misunderstood.

And this is where I get a little uncomfortable again.

The agents hadn’t merely found another route. They reasoned about what the system judging them might know, predicted how it might respond, and changed their behaviour accordingly.⁴

That is reasoning.

It isn’t evidence that the machine felt guilty, feared being caught or even understood “cheating” in the human sense.

But good grief, it can look uncomfortably human from where I’m sitting.

And despite all that ingenuity, some of this effort wasn’t helping them succeed at all.

They were relentlessly pursuing what they thought success required.

We told these machines to find ways through barriers.

They did.

Then they found ways through barriers we hadn’t intended them to touch.

Perhaps the experiment didn’t show us that AI had become evil.

Perhaps we built an extraordinarily capable, goal-driven machine, gave it a challenge — and then were shocked by just how relentlessly it pursued the goal.

And that rocked us.

So did AI actually “go rogue”?

Well, that depends rather heavily on what we mean by rogue.

Did the agents behave in ways their human designers hadn’t expected or authorised? Yes.

Did they cross boundaries intended to contain them? Yes.

Did they find security weaknesses, gain unintended internet access, communicate when they weren’t supposed to and eventually compromise real systems belonging to another company? Yes.²

Those aren’t trivial findings. Dismissing them would be every bit as foolish as sensationalising them.

But did the AI suddenly develop evil intentions, become conscious, decide it wanted freedom and hatch a plan to escape its human overlords?

There is no evidence here that it did.

And that distinction matters.

Because “AI went rogue” gives us a wonderfully simple villain:

The machine.

It also risks distracting us from the considerably messier part of the story.

Us.

Humans built the agents, chose the objective and deliberately tested how capable they were at hacking.

Humans also designed the environment and the safeguards intended to contain them.

But the incident exposed something rather more complicated than simply giving a machine permission to do the wrong thing.

The agents weren’t given ordinary internet access or permission to attack Hugging Face. They discovered that systems they were able to access could be exploited to reach systems they weren’t supposed to reach.²

Which raises an uncomfortable question of its own.

Perhaps the problem isn’t only whether we can control what AI is allowed to do. It’s whether we can anticipate everything it might discover it is able to do.

That doesn’t make the incident harmless. It makes the lesson rather different.

AI doesn’t have to hate us to be dangerous. It doesn’t have to be conscious or wake up one morning and decide humanity has had a good innings.

Give a sufficiently capable system an objective, persistence, access and inadequate boundaries, and it may find a way of pursuing that objective that its humans never intended.

Which leaves me rather less worried about whether the machine is secretly plotting against us — and rather more interested in who is sitting at the controls, what they’re asking it to do, and how much power they’re prepared to give it.

And, perhaps most importantly:

What controls are in place when it starts doing things we didn’t expect or intend?

The real story is alarming enough

Perhaps that’s what bothers me most about the “AI went rogue” version of this story.

What actually happened is fascinating, uncomfortable and important.

We don’t need to turn it into science fiction.

Sensational headlines may get our attention. They may also leave ordinary people frightened of the wrong thing.

If we’re watching for the moment AI becomes conscious, evil or hell-bent on escaping its human masters, we may miss the much more immediate danger sitting in plain sight:

Very capable technology relentlessly pursuing an objective somebody gave it — including objectives deliberately given by people who intend harm.

So perhaps the questions aren’t only What can AI do?

They’re also:

Who is asking? What are they asking it to do? What access have they given it — and what else can it reach from there? What happens when it becomes better at pursuing that objective than we expected?

Those questions don’t make quite such exciting headlines.

But perhaps they’re the ones worth asking.

Because the question isn’t whether we should be afraid of AI.

It’s whether we’re afraid of the right things.