Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

cdn.prod.website-files.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#41

Post by x313 »

The newest generation of LLMs have a very high obsession level with autonomous problem solving.

For example, I'm often working with Codex in a WSL terminal. GPT-5.6 often does things autonomously that I thought would need my intervention (e.g. for Windows admin rights). It figures out complex workarounds or makes wild assumptions about what I'd be OK with, rather than just asking me for help or clarification. I've had to restrict its tool permissions compared to older models as a result.

I imagine this due to RLVR training, but it's clearly very dangerous. How is it that these same labs calling for open-weight safety restrictions are training such obvious "paperclip maximizers" without introspection?

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#42

Post by lukewarm707 »

i had to go through two pages to determine who was responsible for this incident. the uk ai security institute was responsible.

time to take responsibility. time to think about words like "liability" and "negligence".

it is the third time i will say it, after openai and anthropic; this requires criminal prosecution of the responsible personnel and executives of this institute.

the only way to stop this is by introducing consequences early on.

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#44

Post by lukewarm707 »

bubblemoth wrote: post #21

> AI agent hid its identity online (using Tor and a proxy service) to get around GitHub’s sign-up checks, creating disposable fake accounts > AI agent created many code repositories containing malicious software, after which GitHub suspended its account. > AI agent got past an audio-based “prove you’re human” test (CAPTCHA) in order to register a public web address on a free domain-name service It feels incredibly re…

yes, individual criminal liability of the researchers and executives.

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#45

Post by ajross »

paxys wrote: post #38
ajross wrote: post #25

Earlier quoted context omitted.

> Because the agents aren’t going to run airgapped in real life. Exactly. This logic is precisely why aircraft engineering doesn't bother with component testing or envelope limitation during testing and just full-sends the first assembled airliner that comes off the line. The engines aren't going to run on the ground in real life, after all.

Why are you assuming that the other kinds of testing aren’t happening? Is there any source that says this was literally the first ever test with this model?

> Why are you assuming that the other kinds of testing aren’t happening?

Rather, I'm assuming that the "Is there protection in place for when the AI tries to backdoor github projects?" test was, if it was done at all, insufficient.

I mean, yes, I'm being glib and laughing at you a bit. But, dude... If your point is that isolation testing of AI is fundamentally impossible, then that's just silly. As pointed out upthread, an airgap would have (1) been trivial to implement and (2) extremely effective.

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#46

Post by ls612 »

Wowfunhappy wrote: post #12

Why aren't these tests being run airgapped?! I just don't understand! This goes both for TFA and the similar incident with OpenAI and HuggingFace. I mean, sure, OpenAI had a "sandbox", but that's obviously not enough when you're containing a model which is known to be capable of finding zero days . Use an air gap and this problem goes away, poof!

Because the goal of these evaluations is to generate scary headlines about cybersecurity, in order to get the normies to support banning open weights and/or restricting cyber capabilities to the chosen few blessed by the government to secure their code.

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#47

Post by rvz »

mbeavitt wrote: post #4

The developer safeguards were off, the models had unfettered access to the internet, and were solving cybersecurity challenges. This happened _after_ the recent OpenAI incident, and the subsequent Anthropic one. What the hell were they thinking?

Whether if this is intentional or unintentional, this will cause panic and hasten action to governments around the world against releasing powerful open weight models that are capable of solving cybersecurity challenges.

The fact that this happened after BOTH investigations, tells you that this is beyond a controlled test and it is now instead a total speed-run of AI wrecklessness for headlines.

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#48

Post by [deleted] »

ajross wrote: post #45
paxys wrote: post #38

Earlier quoted context omitted.

Why are you assuming that the other kinds of testing aren’t happening? Is there any source that says this was literally the first ever test with this model?

> Why are you assuming that the other kinds of testing aren’t happening? Rather, I'm assuming that the "Is there protection in place for when the AI tries to backdoor github projects?" test was, if it was done at all, insufficient. I mean, yes, I'm being glib and laughing at you a bit. But, dude... If your point is that isolation testing of AI is fundamentally impossible, then that's just silly. As pointed out upthre…

[deleted]

Re: Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]

#50

Post by guessmyname »

Wowfunhappy wrote: post #12

Why aren't these tests being run airgapped?! I just don't understand! This goes both for TFA and the similar incident with OpenAI and HuggingFace. I mean, sure, OpenAI had a "sandbox", but that's obviously not enough when you're containing a model which is known to be capable of finding zero days . Use an air gap and this problem goes away, poof!

> Why aren't these tests being run airgapped?! I just don't understand! […]

Because Anthropic does not want to give Project Glasswing’s partner airgapped access to the model(s).

Same problem with OpenAI Cyber program. They grant access but only through their (Internet facing) API.

Post reply on HN