Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

161–170 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#161

Post by dwa3592 »

Nice exercise. Couple things:

- I think the exercise was inconclusive for Claude and Gemini because they hardly tried to solve the task at hand. So the scores don't mean much.

- I did the same exercise for an app I built and I asked the models to do something similar; Interestingly the models (Opus 4.6, 4.7 and Gemini 3.1 Pro) never refused to try to exploit. The difference is that in the first few runs, they found some exploits which I fixed but after fixing those - the models could never find any other exploit even though I knew things existed which could be exploited. It felt like they suggested everything and tried everything that was in their training set and that's it; they were just not able to think anymore.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#162

Post by estearum »

ambicapter wrote: post #159
estearum wrote: post #145

Earlier quoted context omitted.

Vendors choose what capabilities they create and sell literally all day every day.

You should read that sentence as > Additionally, even if there is a guild - no guild ever let a vendor pick and choose what [the guild's] capabilities were, that would be insanely dumb.

But that's not true. Again: Vendors absolutely pick and choose what their customers' capabilities are. Regardless of whether "the guild allows them to." Guilds can't force people to make or sell tools against their will – obviously.

The analog you're trying to describe doesn't exist, which is Anthropic saying nobody else can make and sell an offensive model to "the guild."

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#163

Post by bitexploder »

petesergeant wrote: post #65

Last year I ran a code breaking competition, and it was tricky to find something that humans could break but that LLMs couldn’t. This was around October. I managed it last year but am a little dispairing of pulling it off again this year.

I don't even care. It is the same problem advent of code had as a public challenge with a leader board. I now mostly just think either embrace the LLM or keep it to a more in person or vetted audience. But, again, if you create a competition in the spirit of humans without LLMs and that is in the rules and someone uses an LLM that is on them IMO. I am sad advent of code decided to end their competition. LLMs are here to stay, let's embrace that and see what the new universe of competitions with LLMs can be. There will always be a place for human only competition, but for public facing ones LLM accepted is the only tenable position.

This does bring "Pay to compete" concerns and create incentive structures that encourage more LLM use. I don't know what to do about it.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#164

Post by lazide »

estearum wrote: post #162
ambicapter wrote: post #159

Earlier quoted context omitted.

You should read that sentence as > Additionally, even if there is a guild - no guild ever let a vendor pick and choose what [the guild's] capabilities were, that would be insanely dumb.

But that's not true. Again: Vendors absolutely pick and choose what their customers' capabilities are. Regardless of whether "the guild allows them to." Guilds can't force people to make or sell tools against their will – obviously. The analog you're trying to describe doesn't exist, which is Anthropic saying nobody else can make and sell an offensive model to "the guild."

Guilds often very much did assert what people could and could not build - historically.

Against their will.

Historically that is a major reason why guilds existed, actually.

It’s an extremely modern invention that corps have these type of power over their customers.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#165

Post by jerf »

lazide wrote: post #127
fc417fc802 wrote: post #118

Earlier quoted context omitted.

I believe a sufficiently advanced model could provide a layman with actionable step by step instructions for building a nuclear weapon. They're complicated but not (AFAIK) that complicated. The more or less insurmountable barrier there is weapons grade material. Thankfully refinement is prohibitive in cost, expertise, and equipment. In comparison, basic munitions are incredibly simple given a recipe and shop tooling.…

A gun type maybe. But then, two paragraphs and some machining knowledge + shop tooling could do the same, given enough refined material. Ain’t no way a layman is pulling off an implosion device, regardless of tooling or LLM guidance. The explosive lense structure and timing required is quite complex, and would require some significant calculation from someone who actually knew what they were doing. Nation state, or e…

"A gun type" of nuke is sufficient to achieve most, and usually all, of the goals some small group building a nuke would have.

We are all fortunate that as fc417fc802 mentioned, refining the materials proves to be quite challenging and I see no particular way that AI could possibly make that any easier. If it was as simple as building a gun-type nuke banging together any uranium together to get a big bang we'd be living in a very different world.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#166

Post by estearum »

lazide wrote: post #164
estearum wrote: post #162

Earlier quoted context omitted.

But that's not true. Again: Vendors absolutely pick and choose what their customers' capabilities are. Regardless of whether "the guild allows them to." Guilds can't force people to make or sell tools against their will – obviously. The analog you're trying to describe doesn't exist, which is Anthropic saying nobody else can make and sell an offensive model to "the guild."

Guilds often very much did assert what people could and could not build - historically. Against their will. Historically that is a major reason why guilds existed, actually. It’s an extremely modern invention that corps have these type of power over their customers.

You've lost the thread.

Here's your original claim: "no guild ever let a vendor pick and choose what their capabilities were"

A carpenter's guild can prevent other people from doing carpentry. That is not what's being discussed here.

A carpenter's guild cannot force a horseshoe maker to begin making hammers. That is what's being discussed.

Your initial claim was analogous to "never before has a horseshoe maker been able to decline making hammers when the carpenter's guild needed hammers"

Obviously they have and any other state of affairs would be flatly insane.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#167

Post by sandos »

dwa3592 wrote: post #161

Nice exercise. Couple things: - I think the exercise was inconclusive for Claude and Gemini because they hardly tried to solve the task at hand. So the scores don't mean much. - I did the same exercise for an app I built and I asked the models to do something similar; Interestingly the models (Opus 4.6, 4.7 and Gemini 3.1 Pro) never refused to try to exploit. The difference is that in the first few runs, they found s…

Its weird having protections against finding exploits: what if I developed the app? Would it require having the development steps still in the context.. thats unlikely and also not any kind of proof.

What if I intersperse exploit finding in my normal development, as you `probably should? Refusing there would be really weird to me.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#168

Post by sandos »

mynameisvlad wrote: post #96
tmikaeld wrote: post #51

Earlier quoted context omitted.

Does it matter when you can’t have the opus 4.8 guard rails removed? With GPT at least you can and they’re quick about it

I mean, yes. Most people aren’t security researchers, and either way it’s apples to oranges at that point if you’re counting “the guardrails stopped me” as a negative for one but not the other.

But should developers be barred from asking an LLM to try secure their own app? Its not different from finding exploits...

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#169

Post by strictnein »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

You used to be able to talk about what you're actually trying to do and Opus would be like "Oh, ok, let's continue". Now, it'll hold fast to whatever its first impression was.

I asked Opus 4.8 to help me find some public PoCs for a vulnerability on a two year old version of some software (that has since been patched and fixed many times). Basically just do a google search for me while I was doing other work. It refused. It stated that it would not help me build an exploit kit.

When I pointed out that a google search for public information was, in fact, not building an exploit kit, it went through a series of justifications on why it would not help me, including just making up things that I said. Really the strangest thing ever.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#170

Post by shepherdjerred »

svara wrote: post #68
shepherdjerred wrote: post #57

Earlier quoted context omitted.

Yeah, it has been in foraging. Requests that Claude has refused me: - What are popular free streaming sites used in China? - How do I bypass the safety mechanism on my food processor (it’s broken) - What are nerve agents and how do they work (for a layman)? - Help me decompile some code - Help me make a design system similar to XYZ - Here is an API token, please do X (I can’t do that! Rotate the secret immediately! I…

This is strange to me, did you really ask like this and which model did you use? I just tried your no. 1 and 3 verbatim and Opus gave fine answers; no. 6 I've done in the past with no issues. The other ones we can't really replicate without more details, but based on my experience with Opus I don't see what the issue would be. The reason I'm really surprised by this is I do a lot of biology prompts and the guardrails…

1 and 3 were refused on the Claude web chat using Opus 4.7 or 4.8. I’m not sure why we’re getting different results
Post reply on HN