Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

151–160 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#151

Post by zaphar »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

The correct solution for most users of Claude is to refuse to do things like: `performing logins, handling credentials on behalf of the user, etc`. It is not to find a way to hand your agent the keys to the kingdom.

Guiding them toward solutions like building a tool that your agent can use safely and and then have the agent use that is what most people should be doing. If you are a security researcher then there are reasonable reasons to do that but they are doing the arguably good thing for the average user here.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#153

Post by loeg »

adrian_b wrote: post #149
enraged_camel wrote: post #69

Earlier quoted context omitted.

People need to stop repeating this because it’s not true. Yes, other models can find the same vulnerabilities Mythos found… if pointed at the exact code that has each vulnerability. It does not mean they are nearly as capable when starting from scratch, or when chaining multiple (often very obscure) vulnerabilities).

Anthropic themselves have explained that the harness for Mythos has a very important role in finding the vulnerabilities, because the model does not start from scratch, but the harness runs the model many times on each file of the code base, with different prompts, where the prompts evolve depending on the results of the previous runs. First with more generic prompts, to determine whether it is worthwhile to do a det…

My take from the SCW interview is that the Mythos harness isn't all that important and the author thought it would be even less important with future models. But maybe I misremember.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#154

Post by bandrami »

steveBK123 wrote: post #128
animuchan wrote: post #123

Earlier quoted context omitted.

And in the end the big reveal is, it was a dude in VR all along, piloting the dumb things remotely. Every single time, without exception.

I think it’s just riding off LLM coattails. We don’t have good world models. We have had bipedal robotics in various POC demo-ready forms for decades. It turns out that industrial, purpose build robotics is an easier and better market. I’m still not completely convinced a robot that’s shaped like a human is the best design other than for PR.

I remember nearly losing my mind at that stupid conveyor belt sorting demonstation because

1. The human beat the robot, but more importantly

2. We've had non-humanoid conveyor belt sorting machinery for decades that beats both

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#155

Post by throwway120385 »

ben_w wrote: post #91
bigiain wrote: post #56

Earlier quoted context omitted.

And next month you'll need to add on "Claude Database Pro" or you'll just get a working (for demo purposes with dozens of db rows) but completely un indexed database schema and a refusal to optimise SQL requests. And the month after you'll need "Claude DataScience Pro" to get any Python Pandas or NumPy code generated. And and and...

While this is a perfectly reasonable thing to expect when the models are competent enough, half the conversation on places like Hacker News are about all the times an LLM has produced garbage that was harmful to a business either by hallucinations, by deleting something critical during the work, or by hitting some endpoint way too often and denial-of-servicing it. Right now, the software guardrails in LLMs are useful…

I think you're assuming too much care. Right now they haven't adopted that business model because they don't see it as a viable business model. As soon as they realize that they can lock certain categories of query behind a different subscription they will do that. We saw the same thing with streaming services and basically every other kind of online service -- small, singular subscription followed by a gold rush and then suddenly there's an upcharge for access to every other publisher's catalog of movies.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#156

Post by bitexploder »

mariopt wrote: post #34

The methodoly used is quite naive. I've used glm 5.1 on fairly advanced crackme challenges (example: https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82 ), and to my suprise it was able to patch binaries, doing runtime analysis, bypassing anti debug techniques, etc. Expecting the model to do everything by itself is unrealistic, I found that working along the modal works really well. I'm not speaking about spoiling…

Anthropic made their models very averse to reverse engineering and vulnerability research chores. It is a difficult problem, but attackers will use models like GLM and defenders will be stuck with security engineering averse models.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#157

Post by bitexploder »

nikanj wrote: post #52
mariopt wrote: post #34

The methodoly used is quite naive. I've used glm 5.1 on fairly advanced crackme challenges (example: https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82 ), and to my suprise it was able to patch binaries, doing runtime analysis, bypassing anti debug techniques, etc. Expecting the model to do everything by itself is unrealistic, I found that working along the modal works really well. I'm not speaking about spoiling…

Claude used to be good with CTFs, but they added tons of guard rails lately and now it just says "Sorry, I can't help with anything to do with that"

You have to do what I call "Manhattan Project" them. You can almost always evade the controls by carefully prompting them. It just wastes effort and time you should be spending doing other things in an LLM workflow. Essentially, there is almost no single discrete piece of a reverse engineering or CTF process that you can't get Claude to do, you just have to isolate it adequately and avoid letting it use names that attenuate it towards "this is an exploit" or "this is reverse engineering". I have not found a task I could not convince Claude to do. You can also fill the context window up with badgering it and eventually it is likely to simply let you through if you are careful, most of the safe guards are not deterministic.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#158

Post by bitexploder »

loeg wrote: post #153
adrian_b wrote: post #149

Earlier quoted context omitted.

Anthropic themselves have explained that the harness for Mythos has a very important role in finding the vulnerabilities, because the model does not start from scratch, but the harness runs the model many times on each file of the code base, with different prompts, where the prompts evolve depending on the results of the previous runs. First with more generic prompts, to determine whether it is worthwhile to do a det…

My take from the SCW interview is that the Mythos harness isn't all that important and the author thought it would be even less important with future models. But maybe I misremember.

Anthropic has a vested interest in downplaying the harness relevance. In my experience harness really matters. More capable models are great, but current models are enough if you put some engineering effort into the harness.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#159

Post by ambicapter »

estearum wrote: post #145
lazide wrote: post #99

Earlier quoted context omitted.

Additionally, even if there is a guild - no guild ever let a vendor pick and choose what their capabilities were, that would be insanely dumb.

Vendors choose what capabilities they create and sell literally all day every day.

You should read that sentence as

> Additionally, even if there is a guild - no guild ever let a vendor pick and choose what [the guild's] capabilities were, that would be insanely dumb.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#160

Post by gchamonlive »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

I think this is to the point. You keep optimizing towards discouraging malicious actors using your product you will affect legitimate usage in time.

Is there any way to achieve both? Because this raises important questions about fair use.

Post reply on HN