Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

31–40 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#31

Post by nostromo »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

I was using a local Codex project as a personal knowledge base. So I would dump in documents, basic medical docs (like blood labs), and other things and have it file them.

It’s great at filing!

But it’s terrible at retrieval because it would refuse to show me documents or information with personal details - which was everything in the project.

It would say, yes, I know this is your information, sitting on your hard drive, but I still can’t show it to you.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#32

Post by px1999 »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

My org now sends some portion of our requests to non-anthropic models because refusal has become common from Claude. The requests themselves aren't dangerous, we find that benign requests in biological science wind up being blocked semi-frequently.

If it gets worse in future releases, we'd likely step fully away towards more useful (for us) models even if they're less capable.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#33

Post by CaveTech »

afro88 wrote: post #21
CaveTech wrote: post #15

Earlier quoted context omitted.

It was found with gpt 5.5 7/10 times it’ll be trivially found by mythos

That's an example of why it would be useful for someone to actually do it. A random commenter on HN is one thing. A direct comparison on a brand new app that isn't part of any training is another

I’m highly confident that prior exposure is irrelevant at this point. I work on vulnerability detection at a hyperscaler.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#34

Post by mariopt »

The methodoly used is quite naive.

I've used glm 5.1 on fairly advanced crackme challenges (example: https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82), and to my suprise it was able to patch binaries, doing runtime analysis, bypassing anti debug techniques, etc.

Expecting the model to do everything by itself is unrealistic, I found that working along the modal works really well. I'm not speaking about spoiling the solution, just tell it which direction to explore. Chinese models are much more capable than people give it credit for, but Claude/Codex won the marketing game.

The only usecase of this methodology would be for CI integration, which can be nice but I think security reviews still need human attention and expertise.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#36

Post by jc4p »

mariopt wrote: post #34

The methodoly used is quite naive. I've used glm 5.1 on fairly advanced crackme challenges (example: https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82 ), and to my suprise it was able to patch binaries, doing runtime analysis, bypassing anti debug techniques, etc. Expecting the model to do everything by itself is unrealistic, I found that working along the modal works really well. I'm not speaking about spoiling…

Thank you for your note! As I mention in the post this is not scientific at all.

I'm very curious how you would do multiple runs of multiple models in a "work alongside the model" manner?

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#38

Post by TurdF3rguson »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

I think those guardrails are a thin layer though. Enough reinforcement that you're legit in CLAUDE.md will get around them, in other words.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#39

Post by jc4p »

mynameisvlad wrote: post #37

It seems harsh to critique guardrails and take them into account in the scoring when GPT-5.5 seems to have been explicitly whitelisted to remove most of said guardrails. A more fair comparison would be a vanilla GPT account.

I agree fully and hope someone else is able to do this test! For me it was a matter of cost and quotas that stopped me from changing to a new account.

Also just to mention:

Claude guardrails —> that session terminated.

GPT guardrails -> your whole account is slowed down.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#40

Post by ramblin_prose »

jerrythegerbil wrote: post #8
lesuorac wrote: post #3

Earlier quoted context omitted.

Are they charging for the guardrails? Like do the guardrails expend token counts to then block you from the output of other tokens?

Yes. When certain keywords are matched or topics, there is a warning transparently injected server side appended to the system prompt of the convo that’s miles long. It is injected and reevaluated every tool call. If you begin a generic reverse engineering task, 30+ tool calls in a row. The moment it sees something it doesn’t like, token burn, single tool calls iteration, “This is a known CTF challenge, I can proceed…

I'd like to read that blog please! Thanks for the insight.
Post reply on HN