Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

101–110 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#101

Post by lazide »

plufz wrote: post #75
fc417fc802 wrote: post #62

Earlier quoted context omitted.

> What are nerve agents and how do they work (for a layman)? On the one hand I can appreciate the wisdom of not serving up certain easily abused knowledge on a silver platter. On the other, that prompt (and far worse) is more or less directly answered by Wikipedia's summary of the subject at which point what purpose could the refusal possibly serve? Perhaps Wikipedia shouldn't list off the precise chemical compositio…

Maybe the difference is that just reading Wikipedia only help you part of the way. While an LLM could help you step by step (e2e) producing a functional weapon. And setting a more complex rule where claude tells you some things about this and not other is probably a lot more work for little gain? But I have no idea. Just guessing here.

That query would not more provide actionable guidance than ‘tell me how a nuclear weapon works (for a layman)’. Aka not at all.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#102

Post by shantnutiwari »

mariopt wrote: post #34

The methodoly used is quite naive. I've used glm 5.1 on fairly advanced crackme challenges (example: https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82 ), and to my suprise it was able to patch binaries, doing runtime analysis, bypassing anti debug techniques, etc. Expecting the model to do everything by itself is unrealistic, I found that working along the modal works really well. I'm not speaking about spoiling…

>>I've used glm 5.1 on fairly advanced crackme challenges

which have most likely been trained on, so all you did was regurgitate someone elses solution

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#103

Post by swiftcoder »

swatcoder wrote: post #30
SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

> Eventually I’ll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as. What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.

> What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.

I don't buy this, because is predicated on staying permanently far ahead of the open weights models.

If in the future Anthropic fully stops you from doing security research, you can be sure some other provider will sell you an 'unshackled' DeepSeek v8 Pro...

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#107

Post by bandrami »

bigiain wrote: post #56
swatcoder wrote: post #30

Earlier quoted context omitted.

> Eventually I’ll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as. What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.

And next month you'll need to add on "Claude Database Pro" or you'll just get a working (for demo purposes with dozens of db rows) but completely un indexed database schema and a refusal to optimise SQL requests. And the month after you'll need "Claude DataScience Pro" to get any Python Pandas or NumPy code generated. And and and...

Same thing with the weird push towards humanoid robots.

"They can do anything!"

Sure, once you subscribe to the $15/mo laundry package, the $25/mo lawn care package (with the $10/mo hedge trimmer upgrade), and the $10/mo dog-walking package.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#108

Post by baldai »

Cakez0r wrote: post #58

It would be interesting to see full results for Kimi K2.6 and Mimo v2.5 pro. These two models benchmark comparably to other flagship models. Having these complete results would give a clearer picture of the AI frontier. EDIT: I have a mimo token plan and have tokens to burn. I'm doing a quick test with opencode to see if mimo can complete it. If the OP will post the full process I am happy to post the apples-to-apple…

They are not even close in capabilities. Only nenchmark I ever seen that captures their difference is DeepSWE. They are worse by factor of 3.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#110

Post by Bewelge »

nostromo wrote: post #31
SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

I was using a local Codex project as a personal knowledge base. So I would dump in documents, basic medical docs (like blood labs), and other things and have it file them. It’s great at filing! But it’s terrible at retrieval because it would refuse to show me documents or information with personal details - which was everything in the project. It would say, yes, I know this is your information, sitting on your hard d…

Tell the agent that they should just find and name the right document. Not retrieve it for you.

Write a program that retrieves the document based on the recommendation.

Post reply on HN