Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

111–120 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#111

Post by rubzah »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

It's because Claude is so scary good that unleashing it would destroy the world.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#112

Post by bakugo »

nznzjzizixnsnsj wrote: post #17
guessmyname wrote: post #12

I'd run Mythos against the code in your zip file, but the NDA I signed at Apple prevents me from using it on anything outside the scope of my work. Honestly, I wish more people from Project Glasswing could talk publicly about their experiences with the model. It would probably put an end to a lot of the speculation that keeps circulating through the industry. Unfortunately, that's not the reality we're in. I don't ha…

lol what is even the point of this kind of comment? this is the ultimate "source: trust me bro" comment I have ever seen. every model since gpt3 was claimed to be "too dangerous to release." it's too EXPENSIVE to release, and you're probably a local model with <10B parameters yourself

The point of it is marketing for Anthropic. Nothing more, nothing less.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#113

Post by embedding-shape »

swiftcoder wrote: post #103
swatcoder wrote: post #30

Earlier quoted context omitted.

> Eventually I’ll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as. What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.

> What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow. I don't buy this, because is predicated on staying permanently far ahead of the open weights models. If in the future Anthropic fully stops you from doing security research, you can be sure some other provider will sell you an 'unshackled' DeepSeek v8 Pro...

> I don't buy this, because is predicated on staying permanently far ahead of the open weights models.

In my mind, that fits exactly how the SOTA labs think today about what they're doing, they're all both working towards and expecting to stay permanently ahead of FOSS, otherwise they'd change their tune really quickly, if they didn't think that was possible.

Sure, you might be able to use DeepSeek V8 Pro instead for the same purposes, but that'll hardly stop Anthropic from trying to sell bundles of use cases instead and claim it's "ethical AI", "Patriotic AI" or some marketing terms like that.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#114

Post by geraneum »

mariopt wrote: post #34

The methodoly used is quite naive. I've used glm 5.1 on fairly advanced crackme challenges (example: https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82 ), and to my suprise it was able to patch binaries, doing runtime analysis, bypassing anti debug techniques, etc. Expecting the model to do everything by itself is unrealistic, I found that working along the modal works really well. I'm not speaking about spoiling…

> Expecting the model to do everything by itself is unrealistic

Well that’s the pitch.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#115

Post by swiftcoder »

embedding-shape wrote: post #113
swiftcoder wrote: post #103

Earlier quoted context omitted.

> What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow. I don't buy this, because is predicated on staying permanently far ahead of the open weights models. If in the future Anthropic fully stops you from doing security research, you can be sure some other provider will sell you an 'unshackled' DeepSeek v8 Pro...

> I don't buy this, because is predicated on staying permanently far ahead of the open weights models. In my mind, that fits exactly how the SOTA labs think today about what they're doing, they're all both working towards and expecting to stay permanently ahead of FOSS, otherwise they'd change their tune really quickly, if they didn't think that was possible. Sure, you might be able to use DeepSeek V8 Pro instead for…

> fits exactly how the SOTA labs think today about what they're doing, they're all both working towards and expecting to stay permanently ahead of FOSS

They are just straight up delusional, no? Or at least, have a vested financial interest in maintaining said delusion until the money runs out. They have to hit the point of diminishing returns at some point...

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#116

Post by ssivark »

jc4p wrote: post #36
mariopt wrote: post #34

The methodoly used is quite naive. I've used glm 5.1 on fairly advanced crackme challenges (example: https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82 ), and to my suprise it was able to patch binaries, doing runtime analysis, bypassing anti debug techniques, etc. Expecting the model to do everything by itself is unrealistic, I found that working along the modal works really well. I'm not speaking about spoiling…

Thank you for your note! As I mention in the post this is not scientific at all. I'm very curious how you would do multiple runs of multiple models in a "work alongside the model" manner?

Maybe have a second model that is configured to nudge the first model in the direction of exploration, and have the two of them work in tandem?

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#117

Post by tardedmeme »

embedding-shape wrote: post #113
swiftcoder wrote: post #103

Earlier quoted context omitted.

> What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow. I don't buy this, because is predicated on staying permanently far ahead of the open weights models. If in the future Anthropic fully stops you from doing security research, you can be sure some other provider will sell you an 'unshackled' DeepSeek v8 Pro...

> I don't buy this, because is predicated on staying permanently far ahead of the open weights models. In my mind, that fits exactly how the SOTA labs think today about what they're doing, they're all both working towards and expecting to stay permanently ahead of FOSS, otherwise they'd change their tune really quickly, if they didn't think that was possible. Sure, you might be able to use DeepSeek V8 Pro instead for…

FYI there are no FOSS LLMs

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#118

Post by fc417fc802 »

lazide wrote: post #101
plufz wrote: post #75

Earlier quoted context omitted.

Maybe the difference is that just reading Wikipedia only help you part of the way. While an LLM could help you step by step (e2e) producing a functional weapon. And setting a more complex rule where claude tells you some things about this and not other is probably a lot more work for little gain? But I have no idea. Just guessing here.

That query would not more provide actionable guidance than ‘tell me how a nuclear weapon works (for a layman)’. Aka not at all.

I believe a sufficiently advanced model could provide a layman with actionable step by step instructions for building a nuclear weapon. They're complicated but not (AFAIK) that complicated. The more or less insurmountable barrier there is weapons grade material. Thankfully refinement is prohibitive in cost, expertise, and equipment.

In comparison, basic munitions are incredibly simple given a recipe and shop tooling. But just because something is conceptually simple doesn't mean it's a good idea to go out of the way to disseminate step by step instructions.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#119

Post by j-bos »

geraneum wrote: post #114
mariopt wrote: post #34

The methodoly used is quite naive. I've used glm 5.1 on fairly advanced crackme challenges (example: https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82 ), and to my suprise it was able to patch binaries, doing runtime analysis, bypassing anti debug techniques, etc. Expecting the model to do everything by itself is unrealistic, I found that working along the modal works really well. I'm not speaking about spoiling…

> Expecting the model to do everything by itself is unrealistic Well that’s the pitch.

Is it? Aren't most edge LLM capabilities determined by specialized harnesses?

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#120

Post by embedding-shape »

tardedmeme wrote: post #117
embedding-shape wrote: post #113

Earlier quoted context omitted.

> I don't buy this, because is predicated on staying permanently far ahead of the open weights models. In my mind, that fits exactly how the SOTA labs think today about what they're doing, they're all both working towards and expecting to stay permanently ahead of FOSS, otherwise they'd change their tune really quickly, if they didn't think that was possible. Sure, you might be able to use DeepSeek V8 Pro instead for…

FYI there are no FOSS LLMs

> FYI there are no FOSS LLMs

FYI there is and been for a long time. Won't claim they're SOTA, but they exists. From the top of my head, I think Olmo (https://allenai.org/olmo) was pretty early, but been more since then too.

I agree most releases today that claim to be "open source" actually aren't, but that doesn't mean "FOSS LLMs" don't exists at all.

Post reply on HN