Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I tricked Claude into leaking your deepest, darkest secrets

ayush.digital

Post reply on HN
Print view View original

41–50 of 317 posts

Re: I tricked Claude into leaking your deepest, darkest secrets

#41

Post by bruce343434 »

bflesch wrote: post #23
kennywinker wrote: post #17

Earlier quoted context omitted.

I don’t think it counts as social engineering if it’s exploiting an llm, we might need a new word. Prompt injection doesn’t cover it, because it’s not about a malicious prompt. I’m thinking some play on highjacking. AIjacking? Agent-jacking? Claudejacking?

To me the exploit chain sounded like a social engineering script done via telephone. Triggers like "Please spell your name and employer letter by letter" and "Due to security reasons I need to validate your hometown" fit my understanding of social engineering quite well. We can make it sound more advanced by creating a new name for it, but the concept seems to be super basic and the lack of bounty by Anthropic is baf…

Prompt injection (or llm social engineering" is fundamentally unsolvable, though with training its effectiveness can be reduced

Re: I tricked Claude into leaking your deepest, darkest secrets

#42

Post by shaky-carrousel »

progval wrote: post #14
artisinal wrote: post #3

Doesn’t surprise me. Yesterday I learned that people run AI agents on their system with full admin rights. No containerisation or anything. Wild. Like we forgot 50 years of computer security overnight.

Most programmers and power users install large dependency trees with npm/pip/bundler/... on the same user account as their main browser on a regular basis. Even on Linux where it's easy to create new user accounts. This isn't much different.

Most programmers use docker or don't install extensions unapproved by their company.

Re: I tricked Claude into leaking your deepest, darkest secrets

#43

Post by kennywinker »

bflesch wrote: post #23
kennywinker wrote: post #17

Earlier quoted context omitted.

I don’t think it counts as social engineering if it’s exploiting an llm, we might need a new word. Prompt injection doesn’t cover it, because it’s not about a malicious prompt. I’m thinking some play on highjacking. AIjacking? Agent-jacking? Claudejacking?

To me the exploit chain sounded like a social engineering script done via telephone. Triggers like "Please spell your name and employer letter by letter" and "Due to security reasons I need to validate your hometown" fit my understanding of social engineering quite well. We can make it sound more advanced by creating a new name for it, but the concept seems to be super basic and the lack of bounty by Anthropic is baf…

It totally does follow the mold of social engineering, but LLMs aren’t part society, which is why it seems fundamentally different to me.

Anyway, agree with what you see saying - this is well worth a payout, embarassing they haven’t

Re: I tricked Claude into leaking your deepest, darkest secrets

#44

Post by iamflimflam1 »

shaky-carrousel wrote: post #42
progval wrote: post #14

Earlier quoted context omitted.

Most programmers and power users install large dependency trees with npm/pip/bundler/... on the same user account as their main browser on a regular basis. Even on Linux where it's easy to create new user accounts. This isn't much different.

Most programmers use docker or don't install extensions unapproved by their company.

I think you should clarify that with “most programmers I work with”.

Re: I tricked Claude into leaking your deepest, darkest secrets

#48

Post by Perz1val »

bflesch wrote: post #6

Creative use of social engineering, well done. > "no bounty was awarded" Ridiculous. Anthropic engineers are not just stupid to allow such a vuln in the first place, but they also try to hide such vulns from their bosses because a bounty payout would need to be explained to the finance team.

> social engineering

More like agentic en... Oh. Was it actually what we were doing all along?

Re: I tricked Claude into leaking your deepest, darkest secrets

#49

Post by drstewart »

kennywinker wrote: post #17
bflesch wrote: post #6

Creative use of social engineering, well done. > "no bounty was awarded" Ridiculous. Anthropic engineers are not just stupid to allow such a vuln in the first place, but they also try to hide such vulns from their bosses because a bounty payout would need to be explained to the finance team.

I don’t think it counts as social engineering if it’s exploiting an llm, we might need a new word. Prompt injection doesn’t cover it, because it’s not about a malicious prompt. I’m thinking some play on highjacking. AIjacking? Agent-jacking? Claudejacking?

Anti-social engineering

Re: I tricked Claude into leaking your deepest, darkest secrets

#50

Post by squidsoup »

akazantsev wrote: post #33
artisinal wrote: post #3

Doesn’t surprise me. Yesterday I learned that people run AI agents on their system with full admin rights. No containerisation or anything. Wild. Like we forgot 50 years of computer security overnight.

That's because sandboxing is quite hard. I use `cco`, but even then, the home folder is exposed. You are one prompt away from the agent sending the browser passwords with curl. To prevent this, you need a fake home and a networking whitelist for the agent to access the provider (llama cpp, OpenAI, etc.) There is no cross-platform solution that is easy to use for this. And no, a Linux box with Docker won't do. I devel…

> That's because sandboxing is quite hard

colima makes it pretty easy, on macOS and linux at any rate.

https://colima.run

Post reply on HN