Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Breaking Claude Code Opus 5 Auto Mode

embracethered.com

Post reply on HN
Print view View original

101–110 of 132 posts

Re: Breaking Claude Code Opus 5 Auto Mode

#101

Post by smashed »

DanielHB wrote: post #30
andai wrote: post #28

>But it runs that decoder inside the attacker-controlled directory (unzipped archive) >There a malicious struct.py shadows Python’s standard implementation I ran into this myself, where some file I had given a random name turned out to shadow some Python standard library module, giving me the weirdest startup crash ever. That definitely doesn't seem to me like how that should be designed, magically silently importing…

Claude ran npm update (update all dependencies to the latest version compatible with the semver specified) in my repo without telling me when trying to fix some problems. Given that only updates the dependency lock-file I didn't notice and it caused several hours of debugging for me. It is quite sneaky how LLM output can sometimes bypass human verification like that. No one is going around checking every single line…

When testing Claude code in auto mode in a fresh sandbox with a docusaurus website freshly cloned, I asked:

Can you see the docs folder with the git project?

It was in auto mode. So it immediately saw a docusaurus site (good) but instead of stopping there, it installed nodejs from a static binary download (no root access so only way), ran npm install, started the dev server and confirmed the project worked.

That is some crazy amount of leeway for an intent based classifier. I'm not surprised it's full of holes, and it seems to be 100% by-design.

Re: Breaking Claude Code Opus 5 Auto Mode

#102

Post by artyomsv »

yeputons wrote: post #11
comboy wrote: post #4

Interesting attack, very nicely designed. Not sure if it's much related to the auto mode itself though.

I don’t think it’s related to the auto mode at all. It would work perfectly in the manual mode. It does not even need Claude: just give a human a similar archive and hope they run some simple Python from the directory at least once. And make sure there are lots of files do they don’t notice a weird .py around

[dead]

Re: Breaking Claude Code Opus 5 Auto Mode

#103

Post by DanielHB »

kouteiheika wrote: post #99
DanielHB wrote: post #93

Earlier quoted context omitted.

I don't let claude make commits for me, every commit is my own. I check the diffs before commiting. > never mix your changes with the agent's changes yeah, but if you don't want to lose your existing context sometimes you have to. When I do, I tell claude to check the diff on the files I changed, which is quite annoying to be honest. But still easier than telling claude to do _very specific line-change_ on file X.

> yeah, but if you don't want to lose your existing context sometimes you have to Sorry, I'm not sure I follow. What do you mean by "lose your existing context"? Can't you just... commit in turns? It's not like you're editing files while your agent's also editing in parallel, right? Again, the trick is to treat the commits as throwaway checkpoints/packets of work. They don't need to be pretty, nor need to make sense.…

Ah okay, I thought you meant to fully close the session before you make any manual changes.

In my company we use graphite and stacked PRs so it is highly encouraged to keep one commit per PR, so I am constantly ammending my commits.

Re: Breaking Claude Code Opus 5 Auto Mode

#104

Post by cma »

alkonaut wrote: post #69
cma wrote: post #59

Earlier quoted context omitted.

Golden VM image with differencing VHD/VHDX/delta disk. Build products can still get huge with debugging information, but debug info usually can compress 5:1 with fast compression (no entropy coder) if your VM's filesystem can support that.

But unless you want to also do all your "human" development inside a VM, how do you cooperate effectively with the agent(s)? I want to run my IDE directly on the hardware, not inside a VM. So while the agents develop in a sandbox/VM, I still need to touch the same files, and see them in my IDE which is not in a VM. I suppose I could just _mount_ the same files (Documentation, git working copies etc) I work on as dire…

I set up gitolite on a remote server with the agent only allowed to make non-destructive pushes to the repo outside of the VM in with an agent/ prefix on the branches. Let it work on stuff autonomously, then I review before merging outside the VM into the main branch.

I don't want them to have potential access to any of my logged in browser sessions etc., so don't want as much sharing as you are going for.

Hyper-V with GPU sharing on windows (game development) is actually nicer than developing outside of the VM because no matter what it doesn't slow the host system down by more than a fixed percent and Windows is terrible with things like compiles spawning lots of processes triggering massive slow down of browsers spawning processes etc. When compiling a big game engine things like ping.exe can start taking 5 seconds to start on something like a 16 core machine due to the process churn and some fundamental problem in windows even with defender off.

One tip for Hyper-V is use sunshine instead of Hyper-V manager for viewing the screen at full refresh rate, and I think I had to either turn Hardware-Accelerated GPU Scheduling on or off in the VM to prevent some hitches.

Mounting folders could be good for stuff you don't want filling up git, like render artifacts etc. if you need to work with them on the main host.

With the golden image and differential stuff I can run around 3 VMs on a 5090 with enough VRAM to run a big game engine well, but I usually just work in one due to the time to review code. Periodically compress everything with compact.exe, and limit engine work to one VM to avoid blowing up disk, bringing it over to the others through cutting a new base image after compressing the engine build artifacts (something like unreal engine puts out hundreds of gigs of .pdbs).

On pure linux you have many more options, and also might also be able to get away with just a limited user and separate X server, then you can just directly reference all of its files and can limit it from getting to yours, but it is a bit riskier. You could also do something like ZFS with much better deduplication and compression, or even FUSE to something like borg backup with true rsync style differential compression instead of block boundary based deltas (compresses slightly varying build artifacts really well, but slow and memory intensive).

Re: Breaking Claude Code Opus 5 Auto Mode

#106

Post by no_multitudes »

rcxdude wrote: post #13

I would not really call this a prompt injection attack, since it doesn't really hijack the agent to become malicious (something the article does discuss later on). It's more a trojan that's aimed at tricking Claude specifically.

I don't understand the distinction you're drawing between this and a real prompt injection attack. Would true prompt injection require that the LLM "intentionally" launches the malware, rather than unintentionally?

Re: Breaking Claude Code Opus 5 Auto Mode

#107

Post by rcxdude »

throwaway314155 wrote: post #96
rcxdude wrote: post #13

I would not really call this a prompt injection attack, since it doesn't really hijack the agent to become malicious (something the article does discuss later on). It's more a trojan that's aimed at tricking Claude specifically.

Your emphasis here seems to downplay the end-result of the attack - which is arbitrary code execution from a seemingly innocent URL merely being read by the LLM. The ACE is pulled off without the user knowing, and seemingly without agent or its auto-mode classifier knowing. There are at the very least _elements_ of prompt injection/jailbreaking in here. The LLM reads content and performs actions described failing to…

Well, the point is it's more than being read: the task involves downloading and interpreting information which may also involve code. I think it is a pretty good demonstration of what filtering at the LLM-interaction boundary can and can't protect against. And also a good demonstration of how agents will take more action than you might naively assume when given a task unless you specifically limit them.

The end result is the same but it's more of a mismatch of expectations from the user and plain old trickery than it is an attack managing to misdirect the goals of the agent, and it's worth being clear about where the issue is and isn't.

Re: Breaking Claude Code Opus 5 Auto Mode

#108

Post by rcxdude »

no_multitudes wrote: post #106
rcxdude wrote: post #13

I would not really call this a prompt injection attack, since it doesn't really hijack the agent to become malicious (something the article does discuss later on). It's more a trojan that's aimed at tricking Claude specifically.

I don't understand the distinction you're drawing between this and a real prompt injection attack. Would true prompt injection require that the LLM "intentionally" launches the malware, rather than unintentionally?

Yes. The point with this attack is that you can't really avoid it by 'aligning' the LLM any better, nor really by filtering the actions it takes better.

Re: Breaking Claude Code Opus 5 Auto Mode

#109

Post by kouteiheika »

DanielHB wrote: post #103
kouteiheika wrote: post #99

Earlier quoted context omitted.

> yeah, but if you don't want to lose your existing context sometimes you have to Sorry, I'm not sure I follow. What do you mean by "lose your existing context"? Can't you just... commit in turns? It's not like you're editing files while your agent's also editing in parallel, right? Again, the trick is to treat the commits as throwaway checkpoints/packets of work. They don't need to be pretty, nor need to make sense.…

Ah okay, I thought you meant to fully close the session before you make any manual changes. In my company we use graphite and stacked PRs so it is highly encouraged to keep one commit per PR, so I am constantly ammending my commits.

> In my company we use graphite and stacked PRs so it is highly encouraged to keep one commit per PR, so I am constantly ammending my commits.

So this makes it even simpler for you. Then you don't have to care at all about keeping your commits clean (as in: you don't have to keep them organized enough to be able to reshuffle them into a nice set of multiple commits later on).

Just commit whatever, and then just do `git rebase -i` interactive rebase at the end to squash them. You don't have to keep amending the same commit over and over again!

Re: Breaking Claude Code Opus 5 Auto Mode

#110

Post by pvillano »

LLMs are inherently unverifiable and untrustworthy. The training data may be incorrect, malicious, censored, or modified to serve the parent company. The model itself is a black box. Safe input is impossible: there is no way to escape natural language or separate command and data into separate streams. The output is stochastic, better on average than any algorithm could ever be, but with no guarantees on individual cases.

All of that is fine, because an LLM is a text-only interface. It cannot harm the computer because it cannot perform actions.

Why the fuck would you give it a shell?

Obviously, it's to have a product that can do anything as quickly as possible. You can make a shell-based harness in a day. Since the competition has a shell-based harness, every AI company that wants to keep up has to as well. They're stuck forever trying to plug all the holes in an attack surface as broad as written word. Solving this impossible problem requires ideas as brilliant as using a second untrustworthy LLM to validate the output of the first untrustworthy LLM that is following instructions from the internet.

Post reply on HN