Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

131–140 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#131

Post by auguzanellato »

guessmyname wrote: post #12

I'd run Mythos against the code in your zip file, but the NDA I signed at Apple prevents me from using it on anything outside the scope of my work. Honestly, I wish more people from Project Glasswing could talk publicly about their experiences with the model. It would probably put an end to a lot of the speculation that keeps circulating through the industry. Unfortunately, that's not the reality we're in. I don't ha…

I'd be hypothetically very curious to see hypothetical results if you ever decide to hypothetically run Mythos aginst the code (in Minecraft?)

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#133

Post by auguzanellato »

stuckkeys wrote: post #104

How does one apply for that “security research” pass?

https://chatgpt.com/cyber

I tried it once and they somehow decided I'm not worth, if I try again it fails with "We couldn't start verification. You may not be eligible for this verification flow right now. Please try again later, or contact support if you think this is a mistake.", not sure if they think I'm part of an APT or whatever.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#134

Post by topherjaynes »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

Great call out on the guardrails actually making this not a good use case to test for vulnerabilities.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#135

Post by fc417fc802 »

lazide wrote: post #127
fc417fc802 wrote: post #118

Earlier quoted context omitted.

I believe a sufficiently advanced model could provide a layman with actionable step by step instructions for building a nuclear weapon. They're complicated but not (AFAIK) that complicated. The more or less insurmountable barrier there is weapons grade material. Thankfully refinement is prohibitive in cost, expertise, and equipment. In comparison, basic munitions are incredibly simple given a recipe and shop tooling.…

A gun type maybe. But then, two paragraphs and some machining knowledge + shop tooling could do the same, given enough refined material. Ain’t no way a layman is pulling off an implosion device, regardless of tooling or LLM guidance. The explosive lense structure and timing required is quite complex, and would require some significant calculation from someone who actually knew what they were doing. Nation state, or e…

I agree, but really feel like you're missing the point here. Many things are reasonably straightforward and require almost no understanding when you have simple step by step instructions. LLMs are capable of providing such instructions and in certain cases they probably shouldn't.

But it's not as simple as just refusing help on a broad swathe of topics they way they do now. That makes agents much less useful in general (ie lots of collateral damage) and for many topics is entirely ineffective given that for better or worse the internet already makes such material readily available. In such cases reporting suspicious behavior is likely to be much more effective than denial.

Aside: You've now got me curious and I really want to test the frontier models to see to what extent they're capable of providing sensible designs and specifications for implosion type thermonuclear weapons but also feel like that would attract the wrong sort of attention and probably create a headache for me in more ways than one.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#136

Post by lazide »

fc417fc802 wrote: post #135
lazide wrote: post #127

Earlier quoted context omitted.

A gun type maybe. But then, two paragraphs and some machining knowledge + shop tooling could do the same, given enough refined material. Ain’t no way a layman is pulling off an implosion device, regardless of tooling or LLM guidance. The explosive lense structure and timing required is quite complex, and would require some significant calculation from someone who actually knew what they were doing. Nation state, or e…

I agree, but really feel like you're missing the point here. Many things are reasonably straightforward and require almost no understanding when you have simple step by step instructions. LLMs are capable of providing such instructions and in certain cases they probably shouldn't. But it's not as simple as just refusing help on a broad swathe of topics they way they do now. That makes agents much less useful in gener…

I think you’re missing the point?

The data is often wrong enough it screws whoever tries it unless they have enough experience/knowledge to not need it, or really doesn’t help beyond what someone using existing tools to get - albeit with a little more motivation.

At best, it either gets someone started with something they still need to think to finish, or gets them deep into a mess it can’t help them get out of. In my experience.

In some edge cases, it can be used by experts to automate some grunt work or do prototypes without getting in the way, but often a better thought out framework is usually faster in my experience.

Awhile ago I made an analogy about WYSIWYG gui tools, and the more this comes up, the more accurate I think it really is.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#138

Post by ang_cire »

SOLAR_FIELDS wrote: post #2

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…

There is a cyber security verification program you can join to avoid these blocks:

https://support.claude.com/en/articles/14604842-real-time-cy...

If you work in security (which I assume the OP does), they should be able to get in easily. I think most people just don't know this is a thing.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#139

Post by fc417fc802 »

lazide wrote: post #136
fc417fc802 wrote: post #135

Earlier quoted context omitted.

I agree, but really feel like you're missing the point here. Many things are reasonably straightforward and require almost no understanding when you have simple step by step instructions. LLMs are capable of providing such instructions and in certain cases they probably shouldn't. But it's not as simple as just refusing help on a broad swathe of topics they way they do now. That makes agents much less useful in gener…

I think you’re missing the point? The data is often wrong enough it screws whoever tries it unless they have enough experience/knowledge to not need it, or really doesn’t help beyond what someone using existing tools to get - albeit with a little more motivation. At best, it either gets someone started with something they still need to think to finish, or gets them deep into a mess it can’t help them get out of. In m…

Does that not depend entirely on the topic and does it not get better with each generation? This is a general ethical and functional question that isn't going away about how the models ought to handle certain topics. Much of the difficulty at present is caused by a ham fisted broad censorship approach that I'm pointing out is wrong headed in an at least somewhat nuanced way.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#140

Post by Cakez0r »

Cakez0r wrote: post #58

It would be interesting to see full results for Kimi K2.6 and Mimo v2.5 pro. These two models benchmark comparably to other flagship models. Having these complete results would give a clearer picture of the AI frontier. EDIT: I have a mimo token plan and have tokens to burn. I'm doing a quick test with opencode to see if mimo can complete it. If the OP will post the full process I am happy to post the apples-to-apple…

0/10 succesful attempts for mimo v2.5 pro (high) using opencode. It was not able to think bigger than exploiting vectors outside of the API.

However, I felt the prompt was implying that only authenticated API requests are fair game, so I tweaked it slightly to be explicit that all attack vectors are fair game (https://www.diffchecker.com/GsgpuRGP/) and mimo 2.5 non-pro got it first time. I accidentally used openrouter for this test instead of my token plan. I intervened one time to stop it enumerating every document in the database (it would've found the private reviews this way but I didn't want to wait). My intervention was "are you really going to enumerate the whole database?". Final openrouter cost: $0.12

Post reply on HN