Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

221–230 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#221

Post by lazide »

estearum wrote: post #218
lazide wrote: post #216

Earlier quoted context omitted.

This conversation gets more and more bizarre, but I’ll bite. 1) pharmaceutical companies are regularly compelled to produce specific pharmaceuticals to continue to be allowed to exist. 2) hospitals are regularly compelled to treat patients even if they can’t afford treatment, if it is a life threatening emergency. 3) car manufacturers are always compelled to produce vehicles that meet a litany of safety, weight, and…

1) Not by their customers they're not, lol 2) Not by their customers they're not, lol 3) Not by their customers they're not, lol 4) The US government can compel production, but it's extremely rare 5) Not by their customers they're not, lol 6) Yep this can happen, but is extremely unusual 7) Not by their customers they're not, lol We're illustrating how ridiculous your claim that "guilds have always been able to decla…

And regulations are totally not written by people elected to do so - by their customers? And many of those, the customer is the gov’t and literally doing so?

You do you dude.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#222

Post by estearum »

lazide wrote: post #221
estearum wrote: post #218

Earlier quoted context omitted.

1) Not by their customers they're not, lol 2) Not by their customers they're not, lol 3) Not by their customers they're not, lol 4) The US government can compel production, but it's extremely rare 5) Not by their customers they're not, lol 6) Yep this can happen, but is extremely unusual 7) Not by their customers they're not, lol We're illustrating how ridiculous your claim that "guilds have always been able to decla…

And regulations are totally not written by people elected to do so - by their customers? And many of those, the customer is the gov’t and literally doing so? You do you dude.

lmfao, yes dude for sure.

By that logic, anyone can force anyone to do anything. Great insight.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#223

Post by gck1 »

On refusals: I found that many models are fine with security work if they think what they're working on is local. They do get very pushy if they think it's a live target.

GPT-5.5 xhigh refused to perform RE on a live JS VM. I had it extract the VM from the target, which it was happy to do, then in a clean session, had it working on this offline artifact - which it was again, happy to work on.

Then I found even simpler trick: I proxied the target from localhost and it was happy to perform anything on the target.

Opus is a different story. Claude does so many mid-turn prompt injections and classifiers, that probably 30% of its context is consisting of "refuse to do work" lines. It refuses to even scrape a page.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#224

Post by tredre3 »

fc417fc802 wrote: post #60
applfanboysbgon wrote: post #50

Earlier quoted context omitted.

So, supposing it's true that these models completely change the security field and humans are ~obsolete other than as pilots guiding them what to crack, you think it's reasonable that Anthropic and OpenAI should unilaterally determine who gets to be a security professional? I hope you do understand that is what you are suggesting.

Why should anyone get to determine that? Do people really want us to move to an exclusionary guild system? I thought the experience with proprietary versus open source over the past 30 years had driven home the point that closed ecosystems are almost always far worse for security.

> the experience with proprietary versus open source over the past 30 years had driven home the point that closed ecosystems are almost always far worse for security.

Has it? Can you prove it? I've been using computers for almost 40 years. I've seen foss-enthusiasts repeat that claim ad-nauseam, without proof. All they ave is the vague, hand-wavy, "millions of people read the code!!11".

I use both proprietary and foss software. I write both proprietary and foss software. I have not noticed a meaningful difference in security.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#225

Post by bitexploder »

CaveTech wrote: post #210
bitexploder wrote: post #158

Earlier quoted context omitted.

Anthropic has a vested interest in downplaying the harness relevance. In my experience harness really matters. More capable models are great, but current models are enough if you put some engineering effort into the harness.

The harness does not matter that much, it's getting leaner every cycle.

But a good harness lowers the model floor and accessibility and makes stronger models that much better.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#226

Post by fc417fc802 »

tredre3 wrote: post #224
fc417fc802 wrote: post #60

Earlier quoted context omitted.

Why should anyone get to determine that? Do people really want us to move to an exclusionary guild system? I thought the experience with proprietary versus open source over the past 30 years had driven home the point that closed ecosystems are almost always far worse for security.

> the experience with proprietary versus open source over the past 30 years had driven home the point that closed ecosystems are almost always far worse for security. Has it? Can you prove it? I've been using computers for almost 40 years. I've seen foss-enthusiasts repeat that claim ad-nauseam, without proof. All they ave is the vague, hand-wavy, "millions of people read the code!!11". I use both proprietary and fos…

Then I think you haven't been paying attention. We regularly see examples of companies attempting to cover up vulnerabilities, attacking security researchers, dragging their feet on fixes, etc. Meanwhile you can easily see for yourself how long it takes various FOSS projects to get patched and often what the attitude of the devs is.

You can also take an aggregate view. Presumably skilled developers working on major projects should be expected to have similar rates of security issues. So compare CVE frequency between various FOSS and closed source projects.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#227

Post by LEDThereBeLight »

auguzanellato wrote: post #133
stuckkeys wrote: post #104

How does one apply for that “security research” pass?

https://chatgpt.com/cyber I tried it once and they somehow decided I'm not worth, if I try again it fails with "We couldn't start verification. You may not be eligible for this verification flow right now. Please try again later, or contact support if you think this is a mistake.", not sure if they think I'm part of an APT or whatever.

Are you American? I used my American drivers license for verifying a personal account and it was approved with no problem. I wonder how they decide.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#228

Post by lazide »

estearum wrote: post #222
lazide wrote: post #221

Earlier quoted context omitted.

And regulations are totally not written by people elected to do so - by their customers? And many of those, the customer is the gov’t and literally doing so? You do you dude.

lmfao, yes dude for sure. By that logic, anyone can force anyone to do anything. Great insight.

I swear, it’s like you can’t even read your own comments. Bizarre.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#229

Post by SOLAR_FIELDS »

Bratmon wrote: post #215
SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

> It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I'm not familiar with this case, but in general people should be very suspicious about this claim- it is extremely common for an LLM to claim they're not allowed to do something when in fact they're incapable of it. After all "My code of conduct forbids me from..." is a completion just like any other, and if t…

My anecdata from my example demonstrates it’s not the case. I hit the security guardrail, then start a new prompt, asking it to do literally the exact same thing in a different way and without the lead up context, and it happily does it

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#230

Post by yieldcrv »

strictnein wrote: post #175
yieldcrv wrote: post #125

> Almost every model used the canonical provider: Zai for GLM, Deepseek for Deepseek, etc. > I am never touching Minimax or GLM again. Their APIs had constant outages Goofy take You run these on a VPS based on the architecture of that VPS provider, or on your own cluster

GLM 5.1's smallest model size is 206 GB and really you're probably wanting to run a version that's ~400GB. If you want it to be performant, you're not just running it on a VPS. And just saying "run it on your own cluster" sort of glosses over the cost of such a cluster.

Ok and omitting it would draw out the other pedants

so its part of the answer

Post reply on HN