Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

21–30 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#21

Post by afro88 »

CaveTech wrote: post #15
guessmyname wrote: post #12

I'd run Mythos against the code in your zip file, but the NDA I signed at Apple prevents me from using it on anything outside the scope of my work. Honestly, I wish more people from Project Glasswing could talk publicly about their experiences with the model. It would probably put an end to a lot of the speculation that keeps circulating through the industry. Unfortunately, that's not the reality we're in. I don't ha…

It was found with gpt 5.5 7/10 times it’ll be trivially found by mythos

That's an example of why it would be useful for someone to actually do it. A random commenter on HN is one thing. A direct comparison on a brand new app that isn't part of any training is another

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#22

Post by sciencejerk »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

Opus 4.6 will still help with full pentesting including RCE. Just requires coaxing (no jailbreak)

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#23

Post by hgoel »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

I've run into some of the refusals to handle my credentials, but so far I've appreciated them. I was only handing over credentials that didn't matter, but it's still a good move, the chat logs are clearly stored somewhere to allow the resume functionality to work, which means your credentials can end up sitting around on your filesystem, and any malware would quickly learn to check for those files.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#24

Post by bombcar »

jerrythegerbil wrote: post #8
lesuorac wrote: post #3

Earlier quoted context omitted.

Are they charging for the guardrails? Like do the guardrails expend token counts to then block you from the output of other tokens?

Yes. When certain keywords are matched or topics, there is a warning transparently injected server side appended to the system prompt of the convo that’s miles long. It is injected and reevaluated every tool call. If you begin a generic reverse engineering task, 30+ tool calls in a row. The moment it sees something it doesn’t like, token burn, single tool calls iteration, “This is a known CTF challenge, I can proceed…

Mythos turns out to be Opus 4.8 in a trenchcoat with guardrails removed.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#25

Post by bombcar »

kay_o wrote: post #5
lesuorac wrote: post #3

Earlier quoted context omitted.

Are they charging for the guardrails? Like do the guardrails expend token counts to then block you from the output of other tokens?

When your session is force ended for "abuse" you get neither the response nor a refund Security, games (think weapons, PVP, attacking, etc), sometimes even asking it for a security review of some CRUD code it wrote itself

I asked it about a “yellow background cell” in Excel and it spewed a book at me. Then it solved the issue.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#26

Post by gmerc »

danpalmer wrote: post #9
SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

This is a good point – because pentesting is entirely legitimate work, and security testing is a necessary and legitimate part of every day software engineering. The problem is that the model can't tell the difference between doing it as part of regular development and doing it in a malicious context. And the root cause of that is that these models lack any sort of real awareness. Humans don't generally get tricked i…

They see an opportunity to charge 10x for pen testing and defence work, while offence will be handled by actors with access to all kind of other models.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#27

Post by gmerc »

lesuorac wrote: post #3
SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

Are they charging for the guardrails? Like do the guardrails expend token counts to then block you from the output of other tokens?

Of course they are. It's standard SaaS to charge for security features ;)

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#28

Post by WizardK »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

[dead]

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#29

Post by FloorEgg »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

I think that these companies are going to have to, and will, invest in some sort of validated identity context to avoid the lowest common denominator.

The first challenge is making sure the guard rails work and are robust. Companies are still working on this.

the second challenge is being able to reliably adapt them as appropriate per user. E.g. allow someone to pen test their own app.

The third challenge (which blocks the second) is to be confident about what is safety-aligned with a specific user.

I think the later will be a hard problem, but they will be highly motivated to solve it.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#30

Post by swatcoder »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

> Eventually I’ll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there

No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as.

What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.

Post reply on HN