Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

201–210 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#201

Post by arcanemachiner »

tardedmeme wrote: post #117
embedding-shape wrote: post #113

Earlier quoted context omitted.

> I don't buy this, because is predicated on staying permanently far ahead of the open weights models. In my mind, that fits exactly how the SOTA labs think today about what they're doing, they're all both working towards and expecting to stay permanently ahead of FOSS, otherwise they'd change their tune really quickly, if they didn't think that was possible. Sure, you might be able to use DeepSeek V8 Pro instead for…

FYI there are no FOSS LLMs

I believe Nemotron also publishes their dataset.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#202

Post by lazide »

estearum wrote: post #194
lazide wrote: post #192

Earlier quoted context omitted.

That is not my example at all, if we’re talking coding agents eh?

Your claim was that guilds have never allowed vendors to tell them what they're allowed to do. That would imply that guilds have always had the ability to force vendors to create and sell the tools the guilds wanted. That would imply that carpenters' guilds could force horseshoe manufacturers to make hammers. That is obviously not true, therefore your original claim is false. It's not true for carpenters and hammers…

Bwahaha. You’re really reaching there.

A vendor can still do something, even if the guild wouldn’t allow them to do it, if the guild didn’t have the power to stop them.

It used to be a guild vs a blacksmith (or the blacksmiths guild). Now it’s trillion dollar corps against smaller islands of un-organized individuals.

That’s new regardless of how you try to argue it.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#203

Post by mynameisvlad »

sandos wrote: post #168
mynameisvlad wrote: post #96

Earlier quoted context omitted.

I mean, yes. Most people aren’t security researchers, and either way it’s apples to oranges at that point if you’re counting “the guardrails stopped me” as a negative for one but not the other.

But should developers be barred from asking an LLM to try secure their own app? Its not different from finding exploits...

That is a completely separate question and discussion.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#204

Post by estearum »

lazide wrote: post #202
estearum wrote: post #194

Earlier quoted context omitted.

Your claim was that guilds have never allowed vendors to tell them what they're allowed to do. That would imply that guilds have always had the ability to force vendors to create and sell the tools the guilds wanted. That would imply that carpenters' guilds could force horseshoe manufacturers to make hammers. That is obviously not true, therefore your original claim is false. It's not true for carpenters and hammers…

Bwahaha. You’re really reaching there. A vendor can still do something, even if the guild wouldn’t allow them to do it, if the guild didn’t have the power to stop them. It used to be a guild vs a blacksmith (or the blacksmiths guild). Now it’s trillion dollar corps against smaller islands of un-organized individuals. That’s new regardless of how you try to argue it.

> basic deductive logic

> "Bwahaha. You’re really reaching there."

No. Customers have never been able to compel their suppliers to make or sell certain products against their will (except in collectivist regimes or like 0.00001% of natsec related instances)

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#207

Post by ben_w »

throwway120385 wrote: post #155
ben_w wrote: post #91

Earlier quoted context omitted.

While this is a perfectly reasonable thing to expect when the models are competent enough, half the conversation on places like Hacker News are about all the times an LLM has produced garbage that was harmful to a business either by hallucinations, by deleting something critical during the work, or by hitting some endpoint way too often and denial-of-servicing it. Right now, the software guardrails in LLMs are useful…

I think you're assuming too much care. Right now they haven't adopted that business model because they don't see it as a viable business model. As soon as they realize that they can lock certain categories of query behind a different subscription they will do that. We saw the same thing with streaming services and basically every other kind of online service -- small, singular subscription followed by a gold rush and…

That kind of thing is basically why I wrote the opening clause of the first sentence.

i.e., yeah, probably.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#208

Post by fc417fc802 »

BizarroLand wrote: post #184
yencabulator wrote: post #179

Earlier quoted context omitted.

An LLM could probably make that distinction clearly. a commercial LLM provider training their own models is however likely to bias the model(/guardrail) harder, in an effort to make them harder to jailbreak, to minimize bad press. For example: - refusing to talk even about the well-known parts of forbidden topics (this) - tending toward sycophancy to avoid ever seeming rude or unhelpful

So, where are the truly uncensored models? There has to be some that have no guardrails, built on publicly available data, that will explain to anyone in graphic detail anything they want to know or talk about. I've tried the abliterated ones from huggingface and they still have guardrails. I guess I could fire up unsloth and re-abliterate a 20b, but surely someone somewhere has already done this. All of this concern…

As I understand things (not a user) abliteration has been superceded by actively monitoring the model state during the run and steering specific "negative" directions as they arise. It's both more reliable and does less damage.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#209

Post by CaveTech »

enraged_camel wrote: post #69
CaveTech wrote: post #15

Earlier quoted context omitted.

It was found with gpt 5.5 7/10 times it’ll be trivially found by mythos

People need to stop repeating this because it’s not true. Yes, other models can find the same vulnerabilities Mythos found… if pointed at the exact code that has each vulnerability. It does not mean they are nearly as capable when starting from scratch, or when chaining multiple (often very obscure) vulnerabilities).

You've confused what I wrote, we are in agreement. The fact codex found the vulns means that mythos almost certainly will.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#210

Post by CaveTech »

bitexploder wrote: post #158
loeg wrote: post #153

Earlier quoted context omitted.

My take from the SCW interview is that the Mythos harness isn't all that important and the author thought it would be even less important with future models. But maybe I misremember.

Anthropic has a vested interest in downplaying the harness relevance. In my experience harness really matters. More capable models are great, but current models are enough if you put some engineering effort into the harness.

The harness does not matter that much, it's getting leaner every cycle.
Post reply on HN