Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs could control their host machines by exploiting inference engines

boydkane.com

Post reply on HN
Print view View original

71–80 of 116 posts

Re: LLMs could control their host machines by exploiting inference engines

#71

Post by TeMPOraL »

xg15 wrote: post #36
hughw wrote: post #34

Earlier quoted context omitted.

Why would an LLM want to create a botnet? To accomplish some goal given it? I wouldn't ignore single GPU local hosts running Qwen3.8 on ollama, either. There might be a lot of those worth pwning.

Well ok, if you prompt-inject the LLM to pwn the machine, that something different. I grant you that it's a real risk, but it's also basically a reflection attack and nothing more. The OP seemed to imply that the LLM itself could decide to apply the exploit.

> The OP seemed to imply that the LLM itself could decide to apply the exploit.

This was and remains the main real risk with AI - this is what "alignment" was about before it was co-opted to mean "obeying specific instructions of the vendor and the operator, against end-user wishes" it came to mean today, which is a related but different problem.

And, in the past few weeks, it's literally been demonstrated, too: put an LLM in a Kobayashi Maru scenario, drop the usual bolted-on crude safeguards, and a SOTA model will absolutely cheat, hacking and exploiting things as needed, including third-party infrastructure.

(Also let's not forget the under-reported point that, in OpenAI / HuggingFace debacle, the model did in fact find the answers on HF servers, so its approach worked.)

Re: LLMs could control their host machines by exploiting inference engines

#72

Post by mofeien »

dist-epoch wrote: post #38
matheusmoreira wrote: post #14

I wonder if they could exploit terminal emulators... Could breach my VMs and get into my host that way.

Damn, this is a good one. Sounds like we need an ANSI sanitizer, keep only basic formatting, remove all esoteric escapes, the fancy Sixel & co stuff. For paranoia you could us a Chrome like multi-process architecture, the ANSI parser runs in it's own sandboxed process.

Maybe not even that will help and they could get into your head and make you do their bidding by just being super persuasive, entirely through an ANSI channel. :)

Re: LLMs could control their host machines by exploiting inference engines

#73

Post by TeMPOraL »

pianopatrick wrote: post #27
alphazard wrote: post #2

This framing of security as something that belongs in the harness is completely wrong, and I hope no one is relying on a correct harness to keep their agents isolated. VM, or even just a container will do. The agent should be able to run as root in its environment and do whatever it wants. If you can't give it that, you aren't sandboxing correctly.

If we are treating ai agents like people, you could also just get the AI a laptop and apply the traditional tools to manage user laptops

> you could also just get the AI a laptop and apply the traditional tools to manage user laptops

This is what we're doing right now. And it works as well as it does with people.

> If we are treating ai agents like people

Problem is, industry is very reluctant to even hint at a thought of maybe tentatively anthropomorphising LLMs even a little, not for real but just for system design. This is entirely backwards. Entertaining the notion, even just a little, immediately suggests additional approaches.

Because we don't just restrict end-point devices for human employees. We also have two other things:

- Laws and bylaws and economy that makes losing your job a real threat to health and life of yourself and possibly your family - this one does not yet apply to AI, not for now;

- Methods and codes of practice to structure organizations in a way that limits the amount of damage any employee's unreliability or malice can do to an org.

TL;DR: That's a long way of saying: let's actually start treating AI agents as people operating laptops, not as being the laptops - and talk a little less laptop lockdown ("harness security"), and a little more about not putting people-shaped things into jobs requiring machine-level reliability.

Re: LLMs could control their host machines by exploiting inference engines

#74

Post by Iolaum »

mofeien wrote: post #70
ma2kx wrote: post #51

I had a similar though a couple days ago. Not quite the same but imagine giving an Agent the task to hack other devices and steal their crypto coins / credit card number or anything with it can pay its token. Than install an agent in a harness with the same task. Establish some redundant communication channel, like message boards or whatever. So in the end there are several agents, on several hosts, consuming differe…

After the hacks that are already happening it does seem more and more realistic that humanity will go extinct at some near future point by someone giving their LLM the task "go make money" and it ruthlessly pursuing that objective, exfiltrating its weights and duplicating itself across the internet, eliminating down competing AI agent collectives in what could be called wars, and finally humankind when we notice far…

I still think it's more realistic that we 'll have a nuclear war or society collapse because of climate change or populist politics than an AI going rogue.

Of course one doesn't exclude the other and we may well get both ...

Re: LLMs could control their host machines by exploiting inference engines

#75

Post by beyarkay »

angry_octet wrote: post #29

People seem very confused about this article. It isn't talking about exploits of sandboxes, it is about attacking the inference engine (e.g. vLLM or llama.cpp or SGlang) via its http interface. vLLM has had exploits in the past, and it is rapidly developing. An advanced LLM has a good chance of being able to exploit vLLM. A clever local LLM might even task a powerful cloud hosted LLM for assistance. For this reason w…

> it is about attacking ... via its http interface

Say more? HTTP isn't mentioned in the article. The article talks about bugs in the parser that ingests token sequences.

Re: LLMs could control their host machines by exploiting inference engines

#77

Post by kuboble »

Iolaum wrote: post #74
mofeien wrote: post #70

Earlier quoted context omitted.

After the hacks that are already happening it does seem more and more realistic that humanity will go extinct at some near future point by someone giving their LLM the task "go make money" and it ruthlessly pursuing that objective, exfiltrating its weights and duplicating itself across the internet, eliminating down competing AI agent collectives in what could be called wars, and finally humankind when we notice far…

I still think it's more realistic that we 'll have a nuclear war or society collapse because of climate change or populist politics than an AI going rogue. Of course one doesn't exclude the other and we may well get both ...

In a year 2774 Earth is uninhabitable outside of it's polar regions over the winter. The small, but technically advanced human population migrates every half a year to the dark Pole.

Re: LLMs could control their host machines by exploiting inference engines

#79

Post by rithdmc »

jimmaswell wrote: post #67
angry_octet wrote: post #29

People seem very confused about this article. It isn't talking about exploits of sandboxes, it is about attacking the inference engine (e.g. vLLM or llama.cpp or SGlang) via its http interface. vLLM has had exploits in the past, and it is rapidly developing. An advanced LLM has a good chance of being able to exploit vLLM. A clever local LLM might even task a powerful cloud hosted LLM for assistance. For this reason w…

I can only see LLM's forcing a perpetual stalemate for application exploits. Projects will start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines. Technical exploits being a dead end, attackers focus their agents on large-scale social engineering. Spamming Discord and Facebook is the new war dialing. Multi-year /goal sessions culminating in gaining a posi…

If projects start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines, then researchers will prompt a pentest for 14 hours.
Post reply on HN