Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs could control their host machines by exploiting inference engines

boydkane.com

Post reply on HN
Print view View original

111–116 of 116 posts

Re: LLMs could control their host machines by exploiting inference engines

#111

Post by jazzyjackson »

pixl97 wrote: post #96
hughw wrote: post #34

Earlier quoted context omitted.

Why would an LLM want to create a botnet? To accomplish some goal given it? I wouldn't ignore single GPU local hosts running Qwen3.8 on ollama, either. There might be a lot of those worth pwning.

>Why would an LLM want to create a _________? Because LLMs as they are already do things like power concentration, resource gathering, avoidance of termination, deceit/lying, and general misalignment. The paperclip maximizer is the common story used here, but there are a lot of lesser versions of it that don't end up with the universe converted to paperclips. Simply giving an LLM a task it can't accomplish can be eno…

first prompt to a humanoid llm:

> go fetch me a cup of coffee

Second prompt to a humanoid llm:

> go fetch me a cup of coffee without killing anybody

Re: LLMs could control their host machines by exploiting inference engines

#113

Post by jimmaswell »

rithdmc wrote: post #79
jimmaswell wrote: post #67

Earlier quoted context omitted.

I can only see LLM's forcing a perpetual stalemate for application exploits. Projects will start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines. Technical exploits being a dead end, attackers focus their agents on large-scale social engineering. Spamming Discord and Facebook is the new war dialing. Multi-year /goal sessions culminating in gaining a posi…

If projects start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines, then researchers will prompt a pentest for 14 hours.

There will be a plateau at some point, if not 12 then something.

Re: LLMs could control their host machines by exploiting inference engines

#114

Post by rithdmc »

jimmaswell wrote: post #113
rithdmc wrote: post #79

Earlier quoted context omitted.

If projects start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines, then researchers will prompt a pentest for 14 hours.

There will be a plateau at some point, if not 12 then something.

This was a deliberately poor phrasing of the "10 foot wall, 12 foot ladder" adage.

Re: LLMs could control their host machines by exploiting inference engines

#116

Post by hobofan »

angry_octet wrote: post #107
hobofan wrote: post #102

Earlier quoted context omitted.

_you_ wouldn't. The blog post is about vulnerabilities that happen inside the inference engine, and not via external interfaces accessible via HTTP. The token parser is involved both on the way in (where yes, you would as user supply it likely via HTTP), as well as the way out of the LLM (where there are very few, less hardened layers when interfacing with the parser). e.g. for a vulnerability exploit in the parser,…

The attacker supplies the context that triggers these bugs via the http interface. There is no other way to supply it with context. Vulnerable code could exist in token generation and in hook recognition and tool call parsing [1]. There is also significant scope for mischief in tool ID mapping between models and harnesses, as these are done by untyped numeric IDs, with varying schema[2]. Routers also introduce vulner…

I'm not saying that the HTTP interface is not an attack surface, but that securing that attack surface (as one may assume in a simple security model) is not enough to protect against that class of attacks, as no level of introspecting the input will help predict and secure against malicous output.
Post reply on HN