Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: We post-trained a model that pen tests instead of refusing

argusred.com

Post reply on HN
Print view View original

11–20 of 49 posts

Re: Show HN: We post-trained a model that pen tests instead of refusing

#11

Post by sudosysgen »

dk189 wrote: post #6
cortesoft wrote: post #4

> This won't be made available to anyone and everyone, but we do believe that responsible SMEs and midmarket companies also need access to these tools in order to identify key vulnerabilities in their systems; not just enterprises. So this is the same policy that Anthropic and OpenAI have, it is just based on your criteria rather than theirs.

I think the policy universally makes sense, who would want to give a tool like this to bad actors? But it does leave a big section of the market underserved. Particularly when Mythos was made accessible to very large orgs and then Fable was pulled on export grounds.

A lot of bad actors are both technically sophisticated and have more than enough resources to post train their model. Morally I think it's still the right choice, but consequence wise I doubt it's going to make a big difference.

Re: Show HN: We post-trained a model that pen tests instead of refusing

#13

Post by jml78 »

Catloafdev wrote: post #5

Why create an offensive tool rather than a repo-scanning tool? I can't think of any way to safely release an offensive tool publicly.

At my job we have tooling that scans our code repos with Opus. Yes it can find stuff however it doesn’t find everything.

I am able to get Opus and Sonnet to function as a red team agent. We don’t have some crazy special sauce, just a lot of trial and error. Basically add enough context proving we own the code and running services that it will run attempts to compromise our services.

It found tons of stuff that was not found with just scanning the code. It found serious security issues that had been in productions for years that humans never found. They weren’t things that were accessible externally but serious enough that we are thrilled to have these tools.

I can say that Fable did refuse to function with our harness. I am worried that soon you have to be in the special club to do this stuff with the SOTA models. A small company like ours doesn’t get accepted to their programs that remove guardrails. Even though our CEO has found and disclosed vulnerabilities to multiple companies and holds a patent around federated authentication.

Re: Show HN: We post-trained a model that pen tests instead of refusing

#15

Post by dk189 »

andai wrote: post #3

Fantastic. Could you share more details what it was like post-training a model?

The RL is easy to describe, hard to do. The nice thing about pen testing is the reward isn't a vibe like training for code quality, the exploit either lands or it doesn't. The day to day is not glamorous at all, mostly fighting for stable gpu access, watching a cluster sit half-idle with nodes you somehow can't book.

Re: Show HN: We post-trained a model that pen tests instead of refusing

#16

Post by rustcleaner »

dk189 wrote: post #6
cortesoft wrote: post #4

> This won't be made available to anyone and everyone, but we do believe that responsible SMEs and midmarket companies also need access to these tools in order to identify key vulnerabilities in their systems; not just enterprises. So this is the same policy that Anthropic and OpenAI have, it is just based on your criteria rather than theirs.

I think the policy universally makes sense, who would want to give a tool like this to bad actors? But it does leave a big section of the market underserved. Particularly when Mythos was made accessible to very large orgs and then Fable was pulled on export grounds.

The policy is repugnant. Whoever delivers the first frontier model as open weights to the world which lacks these moral guardrails will win.

Stop thinking you know morals better than your users, or get out of the way so a competitor who respects your users more can serve them!

Re: Show HN: We post-trained a model that pen tests instead of refusing

#18

Post by rustcleaner »

Catloafdev wrote: post #5

Why create an offensive tool rather than a repo-scanning tool? I can't think of any way to safely release an offensive tool publicly.

They are only protecting corporate interests in insecure code bases by doing this. If everyone could have Mythos in their pockets, all the poorly written bottom dollar rush developed software would be rightfully shown to be the trash it always was. It would spur engineering liability legislation for commercial software and operations: speed-release poor insecure code --> corporate bankruptcy and maybe even prison for the software PE who signed off on it. Software, infrastructure, and hardware security won't improve massively until the "bad actors" start running rampant on the steaming pile!

Re: Show HN: We post-trained a model that pen tests instead of refusing

#19

Post by yieldcrv »

cortesoft wrote: post #4

> This won't be made available to anyone and everyone, but we do believe that responsible SMEs and midmarket companies also need access to these tools in order to identify key vulnerabilities in their systems; not just enterprises. So this is the same policy that Anthropic and OpenAI have, it is just based on your criteria rather than theirs.

I actually wonder how valuable this verbiage is

To me it looks like copycat marketing more than a strongly held stance

Artificial scarcity, membership club criteria to make members feel special

Perhaps there is an organization that awards this “responsibility” behavior, the EU comes to mind but not lucrative enough

As far as engagement farming goes, it got us to engage and boost its reach, for something we might otherwise ignore with more benign language

Once I get the answers I will execute

Re: Show HN: We post-trained a model that pen tests instead of refusing

#20

Post by jrflowers »

dk189 wrote: post #17
jrflowers wrote: post #12

Show HN: We told Claude to generate a marketing page for a theoretical pentesting model

The tool is live, you can test it.

No, you can’t. This page is a sales funnel to schedule a 30 minute video chat with Cosine.ai or argusred or whatever. The thing you can test is not the thing that the headline is talking about.

It’s just more “We’re so smart we invented the boogeyman, trust us” slop marketing that’s been happening since gpt-2

Post reply on HN