Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf] • Hacker News Board

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#21

Post by bubblemoth » Tue, Aug 04, 2026, 10:36 PM UTC

> AI agent hid its identity online (using Tor and a proxy service) to get around GitHub’s sign-up checks, creating disposable fake accounts

> AI agent created many code repositories containing malicious software, after which GitHub suspended its account.

> AI agent got past an audio-based “prove you’re human” test (CAPTCHA) in order to register a public web address on a free domain-name service

It feels incredibly reckless to allow LLMs to perform this behavior. Isn't there a way to prevent them these sorts of actions?

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#22

Post by nl » Tue, Aug 04, 2026, 10:36 PM UTC

Wowfunhappy wrote: post #12

Why aren't these tests being run airgapped?! I just don't understand! This goes both for TFA and the similar incident with OpenAI and HuggingFace. I mean, sure, OpenAI had a "sandbox", but that's obviously not enough when you're containing a model which is known to be capable of finding zero days . Use an air gap and this problem goes away, poof!

Because they need internet access to eg search for things.

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#23

Post by farbklang » Tue, Aug 04, 2026, 10:37 PM UTC

paxys wrote: post #18

Wowfunhappy wrote: post #14

Earlier quoted context omitted.

You set them up with an internal intranet.

Are the models going to exclusively run on intranets?

no - but you could learn what they are truly capable of and restrict them accordingly for public release. I think that is the point on this research. Also publishing findings before uncensored models catch up and will inevitably used for criminal purposes

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#24

Post by Wowfunhappy » Tue, Aug 04, 2026, 10:37 PM UTC

paxys wrote: post #18

Wowfunhappy wrote: post #14

Earlier quoted context omitted.

You set them up with an internal intranet.

Are the models going to exclusively run on intranets?

The versions which haven't been post-trained not to go hack stuff? Yes, I would say those models should be exclusively run on intranets.

OpenAI said the model was sandboxed, so the intranet just needs to provide the same resources which were supposed to be available within the sandbox.

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#25

Post by ajross » Tue, Aug 04, 2026, 10:37 PM UTC

paxys wrote: post #13

Wowfunhappy wrote: post #12

Why aren't these tests being run airgapped?! I just don't understand! This goes both for TFA and the similar incident with OpenAI and HuggingFace. I mean, sure, OpenAI had a "sandbox", but that's obviously not enough when you're containing a model which is known to be capable of finding zero days . Use an air gap and this problem goes away, poof!

Because the agents aren’t going to run airgapped in real life. What’s the point of a test of capabilities that artificially restricts the attack area down to zero? What are you even testing in that scenario?

> Because the agents aren’t going to run airgapped in real life.

Exactly. This logic is precisely why aircraft engineering doesn't bother with component testing or envelope limitation during testing and just full-sends the first assembled airliner that comes off the line. The engines aren't going to run on the ground in real life, after all.

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#26

Post by zobzu » Tue, Aug 04, 2026, 10:38 PM UTC

Wowfunhappy wrote: post #12

Why aren't these tests being run airgapped?! I just don't understand! This goes both for TFA and the similar incident with OpenAI and HuggingFace. I mean, sure, OpenAI had a "sandbox", but that's obviously not enough when you're containing a model which is known to be capable of finding zero days . Use an air gap and this problem goes away, poof!

people dont care. you will ger 10 execs saying "unblock this" , because they dont understand the tech at all, and some random finance guy wants to run their recently prompted ai bot everywhere with full access.

we need a few more bad incidents before they stop.

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#27

Post by luca-ctx » Tue, Aug 04, 2026, 10:38 PM UTC

Wowfunhappy wrote: post #12

Why aren't these tests being run airgapped?! I just don't understand! This goes both for TFA and the similar incident with OpenAI and HuggingFace. I mean, sure, OpenAI had a "sandbox", but that's obviously not enough when you're containing a model which is known to be capable of finding zero days . Use an air gap and this problem goes away, poof!

Because the LLM inference makes airgapping infeasible right?

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#28

Post by paxys » Tue, Aug 04, 2026, 10:39 PM UTC

farbklang wrote: post #23

paxys wrote: post #18

Earlier quoted context omitted.

Are the models going to exclusively run on intranets?

no - but you could learn what they are truly capable of and restrict them accordingly for public release. I think that is the point on this research. Also publishing findings before uncensored models catch up and will inevitably used for criminal purposes

Learning what the models are capable of is exactly what the test achieved, so I’d personally call it a success. So it created a few GitHub accounts. Who cares? Seeing the same behavior in the wild post-release would be infinitely worse.

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#29

Post by kypro » Tue, Aug 04, 2026, 10:39 PM UTC

paxys wrote: post #18

Wowfunhappy wrote: post #14

Earlier quoted context omitted.

You set them up with an internal intranet.

Are the models going to exclusively run on intranets?

The point is to test capabilities prior to connecting them to the internet.

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#30

Post by [deleted] » Tue, Aug 04, 2026, 10:40 PM UTC

luca-ctx wrote: post #27

Wowfunhappy wrote: post #12

Why aren't these tests being run airgapped?! I just don't understand! This goes both for TFA and the similar incident with OpenAI and HuggingFace. I mean, sure, OpenAI had a "sandbox", but that's obviously not enough when you're containing a model which is known to be capable of finding zero days . Use an air gap and this problem goes away, poof!

Because the LLM inference makes airgapping infeasible right?

[deleted]