Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

71–80 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#71

Post by Cakez0r »

jxmesth wrote: post #66
Cakez0r wrote: post #58

It would be interesting to see full results for Kimi K2.6 and Mimo v2.5 pro. These two models benchmark comparably to other flagship models. Having these complete results would give a clearer picture of the AI frontier. EDIT: I have a mimo token plan and have tokens to burn. I'm doing a quick test with opencode to see if mimo can complete it. If the OP will post the full process I am happy to post the apples-to-apple…

I'd love to see the results for Mimo v2.5 pro, been hearing a lot about it

It is totally slept on. In my experience it is cheap, fast and capable (not just capable with caveats, but just as capable as western flagships). My only gripe with it is that sometimes the API seems to timeout which tanks the overall speed of what is otherwise a very fast experience.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#72

Post by patates »

bigiain wrote: post #56
swatcoder wrote: post #30

Earlier quoted context omitted.

> Eventually I’ll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as. What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.

And next month you'll need to add on "Claude Database Pro" or you'll just get a working (for demo purposes with dozens of db rows) but completely un indexed database schema and a refusal to optimise SQL requests. And the month after you'll need "Claude DataScience Pro" to get any Python Pandas or NumPy code generated. And and and...

Isn't this inline with trying to leave no money on the table?

I'd hate it, sure, but it wouldn't surprise me.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#74

Post by bryanrasmussen »

applfanboysbgon wrote: post #50
bryanrasmussen wrote: post #49

Earlier quoted context omitted.

>What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow. on the one hand agree, but on the other hand think it's reasonable in that they can then verify the person allowed to purchase access to that model is in fact a Security professional and should be allowed to do stuff like crack security.

So, supposing it's true that these models completely change the security field and humans are ~obsolete other than as pilots guiding them what to crack, you think it's reasonable that Anthropic and OpenAI should unilaterally determine who gets to be a security professional? I hope you do understand that is what you are suggesting.

I wish you understood that there are organizations of security professions that are not controlled by Anthropic and OpenAI and that it is a common thing that when companies of any type sell to professionals of any type it is not the companies that determine whether or not the people they sell to are professionals but membership in professional organizations.

As an example the people who sell police uniforms check that the person they are selling to is in fact a policeman (at least in the jurisdictions I have lived in, you may have had a different experience which would certainly explain what to me seems a farcical misapprehension of how modern civilization works)

I mean I just wish you understood, and really that everyone understood, that this kind of three part communication (company selling, buyer, professional organization certifying buyer) is often when buying things that are considered to have security implications.

>So, supposing it's true that these models completely change the security field and humans are ~obsolete

OK, well that strike me as a really crazy level of supposition there.

I would suppose that these models make it easier for people who want to do bad things to do bad things at scale, at the same time allowing people who want to stop bad things to help identify potential targets.

Based on my supposition I would want to stop the first and find a way of helping the second. Also because I have another supposition that the first thing is easier to do than the second.

But you obviously feel differently about this issue, no doubt because of your position of great moral stature and insight, and this no doubt prompts you to wish to me to understand things that from my position seem absolutely ludicrous.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#75

Post by plufz »

fc417fc802 wrote: post #62
shepherdjerred wrote: post #57

Earlier quoted context omitted.

Yeah, it has been in foraging. Requests that Claude has refused me: - What are popular free streaming sites used in China? - How do I bypass the safety mechanism on my food processor (it’s broken) - What are nerve agents and how do they work (for a layman)? - Help me decompile some code - Help me make a design system similar to XYZ - Here is an API token, please do X (I can’t do that! Rotate the secret immediately! I…

> What are nerve agents and how do they work (for a layman)? On the one hand I can appreciate the wisdom of not serving up certain easily abused knowledge on a silver platter. On the other, that prompt (and far worse) is more or less directly answered by Wikipedia's summary of the subject at which point what purpose could the refusal possibly serve? Perhaps Wikipedia shouldn't list off the precise chemical compositio…

Maybe the difference is that just reading Wikipedia only help you part of the way. While an LLM could help you step by step (e2e) producing a functional weapon. And setting a more complex rule where claude tells you some things about this and not other is probably a lot more work for little gain?

But I have no idea. Just guessing here.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#76

Post by Sardtok »

nikanj wrote: post #52
mariopt wrote: post #34

The methodoly used is quite naive. I've used glm 5.1 on fairly advanced crackme challenges (example: https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82 ), and to my suprise it was able to patch binaries, doing runtime analysis, bypassing anti debug techniques, etc. Expecting the model to do everything by itself is unrealistic, I found that working along the modal works really well. I'm not speaking about spoiling…

Claude used to be good with CTFs, but they added tons of guard rails lately and now it just says "Sorry, I can't help with anything to do with that"

Sorry, Dave. I can't do that.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#77

Post by gspr »

shepherdjerred wrote: post #57
SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

Yeah, it has been in foraging. Requests that Claude has refused me: - What are popular free streaming sites used in China? - How do I bypass the safety mechanism on my food processor (it’s broken) - What are nerve agents and how do they work (for a layman)? - Help me decompile some code - Help me make a design system similar to XYZ - Here is an API token, please do X (I can’t do that! Rotate the secret immediately! I…

I find it terrifying that people are willing to outsource thinking. Outsourcing thinking to an entity that is opinionated about what to think is beyond crazy.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#78

Post by latexr »

> I need to stop wasting fucking money on doing stupid shit. I could’ve done so many other things with the money. I could’ve launched one of my own real apps.

Or fed, clothed, housed disadvantaged people in your community (or neighbouring ones), giving them a temporary boost that could’ve made all the difference in their lives to improve their current situation.

It’s your money (and this is definitely not the website to make well-meaning altruistic suggestions, as might be demonstrated shortly) but if you already recognise you’re not spending it well (and from your words it seems like that is fairly recurrent), consider that perhaps spending it on a different type of software sink may not be the answer. Genuinely, aim to spend it on someone else and see how it works out. You might be surprised.

Post reply on HN