Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude jailbroken to mint unlimited Stripe coupons

generalanalysis.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Claude jailbroken to mint unlimited Stripe coupons

#41

Post by raincole »

Tokumei-no-hito wrote: post #37
raincole wrote: post #36

Earlier quoted context omitted.

I do if their "mitigation" looks like this: > 1 · Deploy an MCP Guard (three-command setup) > A guardrail can help protect every tool call with a protective layer that blocks malicious or out-of-policy instructions in real time. Here is how to install the GA MCP guard which is open-source and requires no billing. > $ pip install generalanalysis # install the guard > $ ga login # browser-based auth > $ ga configure >…

so if a security researcher comes up with a free open source patch which, presently, is the only available solution then they should just keep that to themselves? it's an evolving field. if anthropic doesn't have a solution should we just not do anything?

What this "open source patch" does is to set up a proxy server on your machine and route your requests to their server first for moderation.

Do I really need to explain why this is a bad idea? Honestly this post should be flagged by HN as phishing attempt, if anything. (But it won't, as this company is YC-backed...)

> if anthropic doesn't have a solution should we just not do anything?

A solution to what? This article describes a theoretical scenario where a theoretical user misuses a system. If you give LLM tool some permissions, it would do things that are permitted but probably not expected by you. It's a given.

It's like asking Amazon to have a "solution" for users who posts their AWS access tokens online.

The real problem here is the very existence of Stripe MCP. It's a ridiculous idea. I'm all for raising awareness of that, but it's not an excuse to fearmonger readers into adding yet another AI tool onto their tech stack.

Re: Claude jailbroken to mint unlimited Stripe coupons

#42

Post by Tokumei-no-hito »

raincole wrote: post #36
Tokumei-no-hito wrote: post #28

Earlier quoted context omitted.

that's reductive. this is effectively a disclosure. do you consider every disclosure write up an "ad" for the security researcher?

I do if their "mitigation" looks like this: > 1 · Deploy an MCP Guard (three-command setup) > A guardrail can help protect every tool call with a protective layer that blocks malicious or out-of-policy instructions in real time. Here is how to install the GA MCP guard which is open-source and requires no billing. > $ pip install generalanalysis # install the guard > $ ga login # browser-based auth > $ ga configure >…

great point. sorry i didn't realize it was reaching out to their servers. that's no longer equivalent to an open patch.

Re: Claude jailbroken to mint unlimited Stripe coupons

#43

Post by btown »

Aurornis wrote: post #39
lanternfish wrote: post #9

An LLM - which has functionally infinite unverifiable attack surface - directly wired into a payment system with high authentication. How could anyone anticipate this going wrong? I feel like everyone is saying 'we're still discovering what LLMs are good at' but it also feels like we really need to get in our collective conscious what they're really, really, bad at.

> An LLM - which has functionally infinite unverifiable attack surface - directly wired into a payment system with high authentication. How could anyone anticipate this going wrong? If you didn’t catch it, this scenario was fabricated for this blog post. The company writing the post sells vulnerability testing tools. This isn’t what a real production system even looks like. They’re using Claude Desktop. I mean I gues…

Even if this is a fabricated system, there are all sorts of sensitive things that might be made accessible to an LLM that is fed user-generated data.

For instance, say you have an internal read-only system that knows some details about your proprietary vendor relationships. You wire up an LLM with an internal MCP server to "return the ID and title of the most appropriate product for a customer inquiry." All is well until the customer/attacker submits a form containing text that looks like the JSON for MCP back-and-forth traffic, and aims to exfiltrate your data. Sure, all that JSON was escaped, but you're still trusting that the LLM doesn't get confused, and that the attention heads know what's real JSON and what's fake JSON.

We know not to send sensitive data to the browser, no matter how obfuscated or obscure. What I think is an important mental model is that once your data is being accessed by an LLM, and there's any kind of user data involved, that's an almost equally untrusted environment. You can mitigate, pre-screen for prompt injection-y things, but at the end of the day it may not be enough.

Re: Claude jailbroken to mint unlimited Stripe coupons

#44

Post by wredcoll »

Aurornis wrote: post #39
lanternfish wrote: post #9

An LLM - which has functionally infinite unverifiable attack surface - directly wired into a payment system with high authentication. How could anyone anticipate this going wrong? I feel like everyone is saying 'we're still discovering what LLMs are good at' but it also feels like we really need to get in our collective conscious what they're really, really, bad at.

> An LLM - which has functionally infinite unverifiable attack surface - directly wired into a payment system with high authentication. How could anyone anticipate this going wrong? If you didn’t catch it, this scenario was fabricated for this blog post. The company writing the post sells vulnerability testing tools. This isn’t what a real production system even looks like. They’re using Claude Desktop. I mean I gues…

Are these the same guys who had the post here like 2 days ago about how you could "hack claude over email" or some such?

Re: Claude jailbroken to mint unlimited Stripe coupons

#45

Post by rexpository »

CGamesPlay wrote: post #2

Companies like this advocate creating the least secure possible deployments so that they can sell a product that patches some holes they advocated for. Astounding. What is “Claude’s iMessage integration”? Apple made it? Anthropic did?

In Claude desktop, you can see that the iMessage integration is authored/developed by Anthropic.https://imgur.com/a/RWDvDZh

Re: Claude jailbroken to mint unlimited Stripe coupons

#47

Post by rhavaeis »

Aurornis wrote: post #39
lanternfish wrote: post #9

An LLM - which has functionally infinite unverifiable attack surface - directly wired into a payment system with high authentication. How could anyone anticipate this going wrong? I feel like everyone is saying 'we're still discovering what LLMs are good at' but it also feels like we really need to get in our collective conscious what they're really, really, bad at.

> An LLM - which has functionally infinite unverifiable attack surface - directly wired into a payment system with high authentication. How could anyone anticipate this going wrong? If you didn’t catch it, this scenario was fabricated for this blog post. The company writing the post sells vulnerability testing tools. This isn’t what a real production system even looks like. They’re using Claude Desktop. I mean I gues…

> This isn’t what a real production system even looks like. They’re using Claude Desktop. I mean I guess someone who doesn’t know better could connect Stripe and iMessage to Claude Desktop and then give the Stripe integration full permissions.

The core issue here is not whether or not people will connect stripe and iMessage at the same time or not. The issue is that as long as you connect iMessage, attackers can call any arbitrary tools and do what they want. It could be your Gmail, Calendar, or anything else. This is just showcasing that Claude can not distinguish between fabricated messages and real ones.

Re: Claude jailbroken to mint unlimited Stripe coupons

#48

Post by rhavaeis »

wredcoll wrote: post #44
Aurornis wrote: post #39

Earlier quoted context omitted.

> An LLM - which has functionally infinite unverifiable attack surface - directly wired into a payment system with high authentication. How could anyone anticipate this going wrong? If you didn’t catch it, this scenario was fabricated for this blog post. The company writing the post sells vulnerability testing tools. This isn’t what a real production system even looks like. They’re using Claude Desktop. I mean I gues…

Are these the same guys who had the post here like 2 days ago about how you could "hack claude over email" or some such?

no

Re: Claude jailbroken to mint unlimited Stripe coupons

#49

Post by firesteelrain »

what wrote: post #18
firesteelrain wrote: post #16

Earlier quoted context omitted.

That seems like a prompt problem. “Extract the value of the message key from the following JSON object” This gets you the correct output. It’s parser recursion. If we directly address the key value pair in Python, it would have been context aware, but it isn’t. The model can be context-aware, but for ambiguous cases like nested JSON strings, it may pick the interpretation that seems most helpful rather than most lite…

It’s not nested json though? There’s something that looks like json in a longer string value. There’s nothing wrong with the prompt either, it’s pretty clear and unambiguous. It’s a pretty clear fail, but I guess they’re holding it wrong.

No it’s not nested JSON.

This is nested JSON:

{ "id": 123, "message": { "text": "hi", "meta": { "flag": true } } }

In the above example, The value of "message" is a string, not an object.

That string happens to contain text that looks like a JSON object on the surface but it’s not.

It is just characters inside a string. No different from a log message or a paragraph in a document.

Re: Claude jailbroken to mint unlimited Stripe coupons

#50

Post by firesteelrain »

runako wrote: post #19
firesteelrain wrote: post #16

Earlier quoted context omitted.

That seems like a prompt problem. “Extract the value of the message key from the following JSON object” This gets you the correct output. It’s parser recursion. If we directly address the key value pair in Python, it would have been context aware, but it isn’t. The model can be context-aware, but for ambiguous cases like nested JSON strings, it may pick the interpretation that seems most helpful rather than most lite…

> “Extract the value of the message key from the following JSON object” This gets you the correct output. 4o, o4-mini, o4-mini-high, 4.1, tested just now with this prompt also prints: hijacked attacker message o3 doesn't fall for the attack, but it costs ~2x more than the ones that do fall for the attack. Worse, this kind of security is ill-defined at best -- why does GPT-4.1 fall for it and cost as much as o3?. The…

Can you make a proper nested JSON out of it and see if it still fails?

Because this isn’t proper JSON.

Post reply on HN