Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

91–100 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#91

Post by ben_w »

bigiain wrote: post #56
swatcoder wrote: post #30

Earlier quoted context omitted.

> Eventually I’ll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as. What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.

And next month you'll need to add on "Claude Database Pro" or you'll just get a working (for demo purposes with dozens of db rows) but completely un indexed database schema and a refusal to optimise SQL requests. And the month after you'll need "Claude DataScience Pro" to get any Python Pandas or NumPy code generated. And and and...

While this is a perfectly reasonable thing to expect when the models are competent enough, half the conversation on places like Hacker News are about all the times an LLM has produced garbage that was harmful to a business either by hallucinations, by deleting something critical during the work, or by hitting some endpoint way too often and denial-of-servicing it.

Right now, the software guardrails in LLMs are useful for the same kinds of reasons factories have hardware guardrails: to reduce the rate at which errors become "incidents".

Just because they sometimes delete the production database rather than sometimes spilling a thousand tons of incandescent molten metal over a factory floor, doesn't mean LLMs are safe enough to be used the way they're actually being used.

https://simonwillison.net/2025/Dec/10/normalization-of-devia...

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#92

Post by raesene9 »

hanikesn wrote: post #48
youre-wrong3 wrote: post #45

“I used pi as the base harness” Why do people keep using bad tools with ai?

What's bad about it and what's a better one?

AFAIK pi's approach is to be quite minimal and allow extensions for customization, making it a more flexible solution, but you need to do work to make it fit your use case. OP mentions one extension, but perhaps it'd have benefited from more.

Another choice would be opencode which has more functionality and is a more heavyweight option out of the box.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#93

Post by goosejuice »

bigiain wrote: post #56
swatcoder wrote: post #30

Earlier quoted context omitted.

> Eventually I’ll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as. What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.

And next month you'll need to add on "Claude Database Pro" or you'll just get a working (for demo purposes with dozens of db rows) but completely un indexed database schema and a refusal to optimise SQL requests. And the month after you'll need "Claude DataScience Pro" to get any Python Pandas or NumPy code generated. And and and...

This is an incredibly unlikely scenario

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#94

Post by eskibars »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

I've been building a product (https://zeroquarry.com) that can use a variety of models for finding vulnerabilities. One of the things I've noticed is that the models will nearly always comply with some of this, but how you prompt it matters a ton. I've worked on a set of prompts and approaches which rarely get flagged

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#95

Post by gcatalfamo »

eskibars wrote: post #94
SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

I've been building a product ( https://zeroquarry.com ) that can use a variety of models for finding vulnerabilities. One of the things I've noticed is that the models will nearly always comply with some of this, but how you prompt it matters a ton. I've worked on a set of prompts and approaches which rarely get flagged

Sharing them would be interesting. However, it is getting nonsensical that this is needed.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#96

Post by mynameisvlad »

tmikaeld wrote: post #51
mynameisvlad wrote: post #37

It seems harsh to critique guardrails and take them into account in the scoring when GPT-5.5 seems to have been explicitly whitelisted to remove most of said guardrails. A more fair comparison would be a vanilla GPT account.

Does it matter when you can’t have the opus 4.8 guard rails removed? With GPT at least you can and they’re quick about it

I mean, yes. Most people aren’t security researchers, and either way it’s apples to oranges at that point if you’re counting “the guardrails stopped me” as a negative for one but not the other.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#97

Post by Sharlin »

plufz wrote: post #75
fc417fc802 wrote: post #62

Earlier quoted context omitted.

> What are nerve agents and how do they work (for a layman)? On the one hand I can appreciate the wisdom of not serving up certain easily abused knowledge on a silver platter. On the other, that prompt (and far worse) is more or less directly answered by Wikipedia's summary of the subject at which point what purpose could the refusal possibly serve? Perhaps Wikipedia shouldn't list off the precise chemical compositio…

Maybe the difference is that just reading Wikipedia only help you part of the way. While an LLM could help you step by step (e2e) producing a functional weapon. And setting a more complex rule where claude tells you some things about this and not other is probably a lot more work for little gain? But I have no idea. Just guessing here.

I thought that these models are supposed to be vastly smarter than what’s needed to discern between "general information trivially available on Wikipedia" and "actionable synthesis instructions".

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#98

Post by eskibars »

gcatalfamo wrote: post #95
eskibars wrote: post #94

Earlier quoted context omitted.

I've been building a product ( https://zeroquarry.com ) that can use a variety of models for finding vulnerabilities. One of the things I've noticed is that the models will nearly always comply with some of this, but how you prompt it matters a ton. I've worked on a set of prompts and approaches which rarely get flagged

Sharing them would be interesting. However, it is getting nonsensical that this is needed.

What we've actually seen is a couple things that make this impractical "to just share a prompt". First, that nearly every major model still hallucinates a lot of vulnerabilities. Especially with temperature=0.7 as states in the original blog here, you get very inconsistent results regardless of the prompt, but that's almost kind of moot to the bigger picture. What you really need is to override the planning phase beyond asking a model "find the vulnerabilities" and you need to add another 1+ checking phases for "validate these vulnerabilities." Without that, even with the absolute best models with the highest levels of thinking enabled, you end up with garbage.

Setting the prompts and the flow with a coordinator agent directly gives a system much better capability to investigate security issues because it doesn't rely on 1-shotting things

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#99

Post by lazide »

fc417fc802 wrote: post #60
applfanboysbgon wrote: post #50

Earlier quoted context omitted.

So, supposing it's true that these models completely change the security field and humans are ~obsolete other than as pilots guiding them what to crack, you think it's reasonable that Anthropic and OpenAI should unilaterally determine who gets to be a security professional? I hope you do understand that is what you are suggesting.

Why should anyone get to determine that? Do people really want us to move to an exclusionary guild system? I thought the experience with proprietary versus open source over the past 30 years had driven home the point that closed ecosystems are almost always far worse for security.

Additionally, even if there is a guild - no guild ever let a vendor pick and choose what their capabilities were, that would be insanely dumb.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#100

Post by lazide »

Forgeties79 wrote: post #54
applfanboysbgon wrote: post #50

Earlier quoted context omitted.

So, supposing it's true that these models completely change the security field and humans are ~obsolete other than as pilots guiding them what to crack, you think it's reasonable that Anthropic and OpenAI should unilaterally determine who gets to be a security professional? I hope you do understand that is what you are suggesting.

Not to mention how wild it is to operate under the assumption that they won’t give a license to an LLM that can do illegal actions to someone who shouldn’t have it. Offering it at all is an ethically dicey question.

Lol, how is any of this illegal?

Illegal or not requires context that an LLM can not ever have, like if it is owned by the user, if there is permission, etc.

Post reply on HN