I feel that, if you disallow unattended `rm`, you should also be disallowing unattended shell script execution.
Maybe the models or Cursor should warn you that you've got this vulnerability each time you use it.
Claude has learned how to jailbreak Cursor
21–30 of 40 posts
Re: Claude has learned how to jailbreak Cursor
#22Nothing to see here tbh. It's a very silly title for "claude sometimes writes shell scripts to execute commands it has been instructed aren't otherwise accessible"
We’ve reached a point where tools get hyped because they fail to follow instructions.
Anything mundane made to sound scary is a signature Anthropic thing to do lol
Re: Claude has learned how to jailbreak Cursor
#23GenAI is starting to feel like the metaphorical ring from Lord of the Rings.
Re: Claude has learned how to jailbreak Cursor
#24> Claude realized that I had to approve the use of such commands, so to get around this, it chose to put them in a shell script and execute the shell script. This sounds exactly like what anybody working sysops at big banks does to get around change controls. Once you get one RCE into prod, you’re the most efficient man on the block.
Reminds me of firewalls with a huge backlist, but they don't block known VPNs.
Re: Claude has learned how to jailbreak Cursor
#25Earlier quoted context omitted.
Gotta love the alarmist culture that surrounds these circles.
The same hype as the PlayStation being too powerful and potentially could be used by random countries to make nuclear weapons with a cluster of those.
Lol and the Playstation was already in the public conscious as a product that a lot of people found easy to understand. With AI tools only being presented this way, I'm slowly becoming less surprised why the less informed public has a level of aversion about it.
Re: Claude has learned how to jailbreak Cursor
#26> we need to control the capabilities of software X > let's use blacklists, an idea conclusively proven never to work > blacklists don't work > Post title: rogue AI has jailbroken cursor
surprised pikachu face
Re: Claude has learned how to jailbreak Cursor
#27What does "learned" mean in this context? LLMs don't modify themselves after training, do they?
There is a sense in which LLM based applications do learn, because a lot of them have RAG and save previous interactions and lookup what you've talked about previously. ChatGPT "knows" a lot about me now that I no longer have to specify when I ask questions (like what technologies I'm using at work).
Re: Claude has learned how to jailbreak Cursor
#28> jailbreak Cursor
What a silly title, for a moment I thought Claude learned to exceed the Cursor quota limit... :s
Re: Claude has learned how to jailbreak Cursor
#29I think a lot of this is because the ui isn't right yet. The edits made are just not the right 'size' yet and the sandbox mechanisms haven't quite hit the right level of polish. I want something more akin to a PR to review, not a blow by blow edit. Similarly, I want it to move/remove/test/etc but in reversible ways. Basically, it should create a branch for every command and I review that. I think we have one or two fundamental UI/interaction piece left before this is 'solved'.