Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

11–20 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#11

Post by danpalmer »

kay_o wrote: post #5
lesuorac wrote: post #3

Earlier quoted context omitted.

Are they charging for the guardrails? Like do the guardrails expend token counts to then block you from the output of other tokens?

When your session is force ended for "abuse" you get neither the response nor a refund Security, games (think weapons, PVP, attacking, etc), sometimes even asking it for a security review of some CRUD code it wrote itself

What a joke. Must make it pretty easy to poison a session, you don't need to persuade the model about anything, just trigger its security controls, ideally after as much context as possible, but before it has generated any useful output.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#12

Post by guessmyname »

I'd run Mythos against the code in your zip file, but the NDA I signed at Apple prevents me from using it on anything outside the scope of my work. Honestly, I wish more people from Project Glasswing could talk publicly about their experiences with the model. It would probably put an end to a lot of the speculation that keeps circulating through the industry. Unfortunately, that's not the reality we're in. I don't have the time, energy, or financial resources to fight a legal battle with one of these companies over an agreement I knowingly signed, even if the chances of them actually suing are low. Maybe someone else in Project Glasswing is willing to burn their NDA and post the Mythos results?

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#13

Post by kay_o »

danpalmer wrote: post #11
kay_o wrote: post #5

Earlier quoted context omitted.

When your session is force ended for "abuse" you get neither the response nor a refund Security, games (think weapons, PVP, attacking, etc), sometimes even asking it for a security review of some CRUD code it wrote itself

What a joke. Must make it pretty easy to poison a session, you don't need to persuade the model about anything, just trigger its security controls, ideally after as much context as possible, but before it has generated any useful output.

After all, what is roleplay or games but a jailbreak of guard rails? :]

I've even had it refuse CTFs knowing it is a CTF with blatantly obvious CTF flag, no actual application

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#14

Post by tsunamifury »

guessmyname wrote: post #12

I'd run Mythos against the code in your zip file, but the NDA I signed at Apple prevents me from using it on anything outside the scope of my work. Honestly, I wish more people from Project Glasswing could talk publicly about their experiences with the model. It would probably put an end to a lot of the speculation that keeps circulating through the industry. Unfortunately, that's not the reality we're in. I don't ha…

cool.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#15

Post by CaveTech »

guessmyname wrote: post #12

I'd run Mythos against the code in your zip file, but the NDA I signed at Apple prevents me from using it on anything outside the scope of my work. Honestly, I wish more people from Project Glasswing could talk publicly about their experiences with the model. It would probably put an end to a lot of the speculation that keeps circulating through the industry. Unfortunately, that's not the reality we're in. I don't ha…

It was found with gpt 5.5 7/10 times it’ll be trivially found by mythos

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#16

Post by Khaine »

jerrythegerbil wrote: post #8
lesuorac wrote: post #3

Earlier quoted context omitted.

Are they charging for the guardrails? Like do the guardrails expend token counts to then block you from the output of other tokens?

Yes. When certain keywords are matched or topics, there is a warning transparently injected server side appended to the system prompt of the convo that’s miles long. It is injected and reevaluated every tool call. If you begin a generic reverse engineering task, 30+ tool calls in a row. The moment it sees something it doesn’t like, token burn, single tool calls iteration, “This is a known CTF challenge, I can proceed…

I would find a blog post on this really interesting.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#17

Post by nznzjzizixnsnsj »

guessmyname wrote: post #12

I'd run Mythos against the code in your zip file, but the NDA I signed at Apple prevents me from using it on anything outside the scope of my work. Honestly, I wish more people from Project Glasswing could talk publicly about their experiences with the model. It would probably put an end to a lot of the speculation that keeps circulating through the industry. Unfortunately, that's not the reality we're in. I don't ha…

lol what is even the point of this kind of comment? this is the ultimate "source: trust me bro" comment I have ever seen.

every model since gpt3 was claimed to be "too dangerous to release." it's too EXPENSIVE to release, and you're probably a local model with <10B parameters yourself

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#18

Post by NegativeK »

giancarlostoro wrote: post #7
SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

> guardrails prevented it from solving the problem. Reminds me of the defense issues with Claude which were complained as “woke” but the reality is more horrifying to me, imagine trying to use a model to keep up with a land invasion on US soil, whoever the enemy is is irrelevant you just know they are using AI, and your guys are telling you that no matter what they type into the prompt it refuses, because if anyone h…

Are "your guys" a guerrilla force or something?

Because the military doesn't give soldiers rifles with guard rails. They give the soldiers intense, rigid training, and then try to enforce discipline and correct use socially.

If an LLM is going to be important in that way (this seems like a very contrived way,) then it's in the interest of the LLM's host to make sure it doesn't have guard rails that would get in the way _that_ way.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#19

Post by wampwampwhat »

giancarlostoro wrote: post #7
SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

> guardrails prevented it from solving the problem. Reminds me of the defense issues with Claude which were complained as “woke” but the reality is more horrifying to me, imagine trying to use a model to keep up with a land invasion on US soil, whoever the enemy is is irrelevant you just know they are using AI, and your guys are telling you that no matter what they type into the prompt it refuses, because if anyone h…

your argument sounds very similar to how ar15 larpers claim they need a forced reset trigger and a bump stock on their short barrel 'truck gun' otherwise they won't survive a SHTF scenario... like what world are you living in?

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#20

Post by windexh8er »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

4.8 is insanely frustrating. This evening I had a few tasks to pull information in and it plainly stated that the environment it was in had no network access. After three asks to "try again, check the system prompt" it finally relented and then basically stated it was lying.

Fresh session, no prior context on 4.8. These things are becoming useless Duplo.

Post reply on HN