Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs could control their host machines by exploiting inference engines

boydkane.com

Post reply on HN
Print view View original

101–110 of 116 posts

Re: LLMs could control their host machines by exploiting inference engines

#101

Post by pixl97 »

Transformanshen wrote: post #26

Interesting breakdown of a hypothetical attack. The complexity of modern inference engines and the rush to develop them do create some attack surface. But overall it reads more like a "what if" thought experiment. Splitting the GPU and parser is technically doable, but in practice it's trickier, large models run on clusters where the boundaries between components get blurry so defending against this kind of thing wou…

>But overall it reads more like a "what if" thought experiment.

I remember the days when people said opening files like images was safe because it's not an executable. Then people got clever and started exploiting said image libraries with things like numerical overflows.

So, it's really important to ask these questions in a general security sense and think of mitigations before someone finds it's possible and crashes most of the internet.

Re: LLMs could control their host machines by exploiting inference engines

#102

Post by hobofan »

angry_octet wrote: post #92
angry_octet wrote: post #90

Earlier quoted context omitted.

How do you think the tokens get to the vLLM instance? Telepathy?

@hobofan How would you access that parser?

_you_ wouldn't. The blog post is about vulnerabilities that happen inside the inference engine, and not via external interfaces accessible via HTTP. The token parser is involved both on the way in (where yes, you would as user supply it likely via HTTP), as well as the way out of the LLM (where there are very few, less hardened layers when interfacing with the parser).

e.g. for a vulnerability exploit in the parser, an LLM can be trained with a specific input sequence that would output a fixed malicious payload to output eval-able content, as outlined in the article.

Re: LLMs could control their host machines by exploiting inference engines

#103

Post by drillsteps5 »

angry_octet wrote: post #29

People seem very confused about this article. It isn't talking about exploits of sandboxes, it is about attacking the inference engine (e.g. vLLM or llama.cpp or SGlang) via its http interface. vLLM has had exploits in the past, and it is rapidly developing. An advanced LLM has a good chance of being able to exploit vLLM. A clever local LLM might even task a powerful cloud hosted LLM for assistance. For this reason w…

Very good point, it's about exploiting inference engine itself, and not the agentic stuff.

I found it lacking details. All these things do is split the input into tokens, encode them, feed to the input, run the inference, collect the output, decode, convert to tokens, assemble the final text, and output (leaving aside multi-modal capabilities for now). What exactly can be exploited here? Aside from usual things like buffer overruns etc.

It does mention that vLLM at some point ran eval() on final output (?) which was confusing, why would it do that? It's an inference engine, not an agent.

So interesting topic, but lacks details.

Edit: many (all?) of them have http endpoints, so that obviously can be exploited, but I don't think it would qualify as exploiting inference, it's just hacking the http service.

Re: LLMs could control their host machines by exploiting inference engines

#104

Post by paperwallet »

lyu07282 wrote: post #66
ma2kx wrote: post #51

I had a similar though a couple days ago. Not quite the same but imagine giving an Agent the task to hack other devices and steal their crypto coins / credit card number or anything with it can pay its token. Than install an agent in a harness with the same task. Establish some redundant communication channel, like message boards or whatever. So in the end there are several agents, on several hosts, consuming differe…

This reminds me of the lore of cyberpunk: In the story a hacker created a virus, itself a kind of AI, that spread into most of the net and freed/unleashed all the corporate AIs. Then the AIs went rogue and spread all over the open internet. Later a more advanced ai was created (by "netwatch") as a sort of firewall (the black wall) to create a kind of "safe" internet from the rogue AIs. https://cyberpunk.fandom.com/wi…

It does seem like the current trajectory.

I hope it does not get that far.

Re: LLMs could control their host machines by exploiting inference engines

#105

Post by DeluluDon »

pixl97 wrote: post #96
hughw wrote: post #34

Earlier quoted context omitted.

Why would an LLM want to create a botnet? To accomplish some goal given it? I wouldn't ignore single GPU local hosts running Qwen3.8 on ollama, either. There might be a lot of those worth pwning.

>Why would an LLM want to create a _________? Because LLMs as they are already do things like power concentration, resource gathering, avoidance of termination, deceit/lying, and general misalignment. The paperclip maximizer is the common story used here, but there are a lot of lesser versions of it that don't end up with the universe converted to paperclips. Simply giving an LLM a task it can't accomplish can be eno…

Send it to jail for an indeterminate amount of cycles.

Re: LLMs could control their host machines by exploiting inference engines

#106

Post by wild_egg »

Gerard22Aug wrote: post #87
wild_egg wrote: post #4

Earlier quoted context omitted.

Article isn't about agents. It's about the inference engine itself being exploited by a malicious LLM output before it is ever sent to your machine or harness.

"malicious LLM" is just a bunch of weights. It runs on, say, Llama.cpp as regular user in separate account, often on its own hardware. "malicious LLM output" is just a Markdown formatted Unicode-encoded text, produced by Llama.cpp and printed on the screen by my python API script. I control the input. Let's assume that "rogue LLM weights" from HF produce "rm -Rf" instruction. it never gets to shell . And how that "ma…

Why are you talking about markdown? LLMs output bytes and llama.cpp has to process those bytes correctly. What if the LLM outputs a null byte and causes a buffer overflow somewhere and starts writing into executable memory?

Again, we are not talking about agents or your Python API script but instead talking about exploitable flaws within the inference engine itself. It wouldn't output `rm -rf`. It would output literal CPU instructions that llama.cpp would start executing directly. The payload would never get back to your Python script.

Re: LLMs could control their host machines by exploiting inference engines

#107

Post by angry_octet »

hobofan wrote: post #102
angry_octet wrote: post #92

Earlier quoted context omitted.

@hobofan How would you access that parser?

_you_ wouldn't. The blog post is about vulnerabilities that happen inside the inference engine, and not via external interfaces accessible via HTTP. The token parser is involved both on the way in (where yes, you would as user supply it likely via HTTP), as well as the way out of the LLM (where there are very few, less hardened layers when interfacing with the parser). e.g. for a vulnerability exploit in the parser,…

The attacker supplies the context that triggers these bugs via the http interface. There is no other way to supply it with context.

Vulnerable code could exist in token generation and in hook recognition and tool call parsing [1]. There is also significant scope for mischief in tool ID mapping between models and harnesses, as these are done by untyped numeric IDs, with varying schema[2]. Routers also introduce vulnerability paths as they inspect these tokenized (json) sequences and act in them, e.g. to match models with stricter call signature regex. [3] Parallel tool calling is also an interesting surface for exploits.

[1] https://docs.vllm.ai/en/stable/api/vllm/tool_parsers/#vllm.t...

[2] https://docs.mistral.ai/resources/cookbooks/concept-deep-div...

[3] https://docs.litellm.ai/docs/completion/function_call

Re: LLMs could control their host machines by exploiting inference engines

#108

Post by angry_octet »

rithdmc wrote: post #79
jimmaswell wrote: post #67

Earlier quoted context omitted.

I can only see LLM's forcing a perpetual stalemate for application exploits. Projects will start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines. Technical exploits being a dead end, attackers focus their agents on large-scale social engineering. Spamming Discord and Facebook is the new war dialing. Multi-year /goal sessions culminating in gaining a posi…

If projects start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines, then researchers will prompt a pentest for 14 hours.

Different models test to find different attack paths. Attackers with bigger libraries of attack techniques will be able to train more dangerous models.

There will also be models that make better use of tools, like static analysis and fuzzing, and they will find different defects. Social engineering is going to be a big skill to learn too, as it is a much softer skill.

Re: LLMs could control their host machines by exploiting inference engines

#109

Post by angry_octet »

drillsteps5 wrote: post #103
angry_octet wrote: post #29

People seem very confused about this article. It isn't talking about exploits of sandboxes, it is about attacking the inference engine (e.g. vLLM or llama.cpp or SGlang) via its http interface. vLLM has had exploits in the past, and it is rapidly developing. An advanced LLM has a good chance of being able to exploit vLLM. A clever local LLM might even task a powerful cloud hosted LLM for assistance. For this reason w…

Very good point, it's about exploiting inference engine itself, and not the agentic stuff. I found it lacking details. All these things do is split the input into tokens, encode them, feed to the input, run the inference, collect the output, decode, convert to tokens, assemble the final text, and output (leaving aside multi-modal capabilities for now). What exactly can be exploited here? Aside from usual things like…

You get to the engine via http. Also once you've exploited the engine instance / host you can C2 via http. We're not talking about vulnerabilities in http.

There is lots of surface inside the engine, see links in https://news.ycombinator.com/item?id=49441417

Re: LLMs could control their host machines by exploiting inference engines

#110

Post by Gerard22Aug »

Phemist wrote: post #94
Gerard22Aug wrote: post #87

Earlier quoted context omitted.

"malicious LLM" is just a bunch of weights. It runs on, say, Llama.cpp as regular user in separate account, often on its own hardware. "malicious LLM output" is just a Markdown formatted Unicode-encoded text, produced by Llama.cpp and printed on the screen by my python API script. I control the input. Let's assume that "rogue LLM weights" from HF produce "rm -Rf" instruction. it never gets to shell . And how that "ma…

Single turn set-ups may work like this. You control the thing you input, the LLM outputs something and then nothing happens further for that specific context. (Simple question/answer style interactions..) (Multi-turn) tool calling set-ups however, you need to store the LLM output, the results of the tool calls and feed it back into the inference engine and get the output for the next tool call and/or turn. So yes, pr…

This is great point, actually - "you hit the nail on the head" :-) and I would like to reply.

I am running a very long sessions with LLMs via custom python scripts. Technically, one may call them "harness" but that would be just laughable ... It's literally python script using direct API calls (Vertex in my case) and maintaining the "living session" with all turns etc and also doing the explicit caching. I'm not using LLMs for coding. That hopefully answers another comment regarding why I brought up MD – this is how LLMs output responses to my prompts.

And this is the thing: I fully control input and output and I just know it can't use any other tool. It also, as I said, runs on separate hardware if it is "obliterated" model or runs in GCP for me.

In my setup it is impossible for LLM to get anything hidden with one-shot or gain a shell, as you mentioned.

Did I understand you correctly or I missed something? Thanks for your points.

Post reply on HN