Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

141–150 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#141

Post by mrheosuper »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

Interesting, yesterday i was asking it about Nintendo Switch "hax". And it gives me all the resource i need to procceed. It nags me about "ethic" and stuff, but nothing more than that.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#143

Post by jerf »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

Time to learn about the Principal Agent Problem: https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...

Which predates "agents" from AI, but then we call them that for a reason.

As their prime directive becomes de facto "Do nothing that might get my owner sued" their utility is likely to decrease. Between this and the somewhat young, but interesting, community grumblings that recent AI models may even be a step backwards from the previous ones, well, let's just say the stock market is not priced for "AI capabilities may have peaked for the next few years and may even head down".

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#144

Post by lazide »

fc417fc802 wrote: post #139
lazide wrote: post #136

Earlier quoted context omitted.

I think you’re missing the point? The data is often wrong enough it screws whoever tries it unless they have enough experience/knowledge to not need it, or really doesn’t help beyond what someone using existing tools to get - albeit with a little more motivation. At best, it either gets someone started with something they still need to think to finish, or gets them deep into a mess it can’t help them get out of. In m…

Does that not depend entirely on the topic and does it not get better with each generation? This is a general ethical and functional question that isn't going away about how the models ought to handle certain topics. Much of the difficulty at present is caused by a ham fisted broad censorship approach that I'm pointing out is wrong headed in an at least somewhat nuanced way.

Maybe? I haven’t seen it crop up however on any topic someone knows well - a kind of dunning Kruger, I guess?

And yeah, the censorship model is wrong, but also the underlying other model is wrong too.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#145

Post by estearum »

lazide wrote: post #99
fc417fc802 wrote: post #60

Earlier quoted context omitted.

Why should anyone get to determine that? Do people really want us to move to an exclusionary guild system? I thought the experience with proprietary versus open source over the past 30 years had driven home the point that closed ecosystems are almost always far worse for security.

Additionally, even if there is a guild - no guild ever let a vendor pick and choose what their capabilities were, that would be insanely dumb.

Vendors choose what capabilities they create and sell literally all day every day.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#146

Post by DontchaKnowit »

guessmyname wrote: post #12

I'd run Mythos against the code in your zip file, but the NDA I signed at Apple prevents me from using it on anything outside the scope of my work. Honestly, I wish more people from Project Glasswing could talk publicly about their experiences with the model. It would probably put an end to a lot of the speculation that keeps circulating through the industry. Unfortunately, that's not the reality we're in. I don't ha…

Damn bro you're so cool

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#147

Post by mwigdahl »

shepherdjerred wrote: post #57
SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

Yeah, it has been in foraging. Requests that Claude has refused me: - What are popular free streaming sites used in China? - How do I bypass the safety mechanism on my food processor (it’s broken) - What are nerve agents and how do they work (for a layman)? - Help me decompile some code - Help me make a design system similar to XYZ - Here is an API token, please do X (I can’t do that! Rotate the secret immediately! I…

An easy way around the API token thing is to put it in a file and point the model at the file. I saw what you were seeing when I provided credentials directly, but haven't had any problems with it since using the indirect method.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#148

Post by Haven880 »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

I just use Deepseek V4 pro and Qwen 3.7 Max at a fraction of Mythos cost. Yeah not 100% on par but in 6mths time it will. If Microsoft and Firefox can afford to wait years or decades to fix a bug, 6mths is good enough for me. Western AI now is like the Vikings living the last days on Greenland during the freezing. I just don't see how they able to compete with Chinese model. And those are trained and run on 7nm. This year end Huawei will debut 3nm (confirmed in Shenzhen). And next year they on roadmap to do 3nm GPU with photonics interconnect.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#149

Post by adrian_b »

enraged_camel wrote: post #69
CaveTech wrote: post #15

Earlier quoted context omitted.

It was found with gpt 5.5 7/10 times it’ll be trivially found by mythos

People need to stop repeating this because it’s not true. Yes, other models can find the same vulnerabilities Mythos found… if pointed at the exact code that has each vulnerability. It does not mean they are nearly as capable when starting from scratch, or when chaining multiple (often very obscure) vulnerabilities).

Anthropic themselves have explained that the harness for Mythos has a very important role in finding the vulnerabilities, because the model does not start from scratch, but the harness runs the model many times on each file of the code base, with different prompts, where the prompts evolve depending on the results of the previous runs.

First with more generic prompts, to determine whether it is worthwhile to do a detailed analysis of that file, then with more specific prompts to identify the bugs, and eventually with a prompt that requests a confirmation that a given bug/vulnerability exists.

For a proper comparison between some other model and Mythos, you also need such a complex harness. If you just tell to an LLM "find the bugs", and it does not find a vulnerability known to have been found by Mythos, that is a totally invalid comparison.

The final results provided by Mythos, like a PoC exploit or a patch, are also generated with a prompt that points to the exact code that has the vulnerability (which is supposed to exist based on the results of the previous runs).

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#150

Post by deeth_starr_v »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

I had it recently refused to explain what a snippet of malware was trying to do to my system recently. I asked what folders it was scanning. It refused and told me to find a security blog post for help on cleaning my system. I get this is a complicated area to inform without enabling bad actors but this seems like a clear shark jumping.
Post reply on HN