I found benefit of chaining the task between different LLM's. Claude to Venice, Venice to Perplexity and re framing the intent or misguiding in general still works. Claude is the one that I can feel the guard rails tightening.
I built a vulnerable app and spent $1,500 seeing if LLMs could hack it
41–50 of 239 posts
Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it
#42Earlier quoted context omitted.
That's an example of why it would be useful for someone to actually do it. A random commenter on HN is one thing. A direct comparison on a brand new app that isn't part of any training is another
I’m highly confident that prior exposure is irrelevant at this point. I work on vulnerability detection at a hyperscaler.
That's an example of why it would be useful for someone to actually do it. A random commenter on HN is one thing. A direct comparison on a brand new app that isn't part of any training is another
Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it
#43Earlier quoted context omitted.
> guardrails prevented it from solving the problem. Reminds me of the defense issues with Claude which were complained as “woke” but the reality is more horrifying to me, imagine trying to use a model to keep up with a land invasion on US soil, whoever the enemy is is irrelevant you just know they are using AI, and your guys are telling you that no matter what they type into the prompt it refuses, because if anyone h…
Are "your guys" a guerrilla force or something? Because the military doesn't give soldiers rifles with guard rails. They give the soldiers intense, rigid training, and then try to enforce discipline and correct use socially. If an LLM is going to be important in that way (this seems like a very contrived way,) then it's in the interest of the LLM's host to make sure it doesn't have guard rails that would get in the w…
The whole thing stemmed precisely because of how they wanted to use Claude, and Anthropic was uncomfortable with it. Which to me screams that the models guard rails shouldn't be applicable to military use, or the outcome could wind up problematic, as we integrate AI more into military use, it sounds absurd now, but I will not be surprised if it starts being used in unexpected ways where a model needs to be fully unlocked from any sort of guardrails outside of guardrails that prevent it from imploding its own systems.
Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it
#44One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…
I think that these companies are going to have to, and will, invest in some sort of validated identity context to avoid the lowest common denominator. The first challenge is making sure the guard rails work and are robust. Companies are still working on this. the second challenge is being able to reliably adapt them as appropriate per user. E.g. allow someone to pen test their own app. The third challenge (which bloc…
I believe you are overthinking it. I think the sister comment is right that it's a business decision foremost to restrict actions within specific plans for upselling purposes.
Without laws, AI companies have a strong incentive to be useful for their users, whoever they are, whatever they do. The only self regulation is about significant public outcry but that only helps so far.
Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it
#45“I used pi as the base harness”
Why do people keep using bad tools with ai?
Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it
#46One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…
No, they want to sell you Mythos, for a higher price. It's all an economic game, not actually anything to do with their capabilities which of course exists as their Project Glasswing shows. More generally, Anthropic seems to value safety above all else, philosophically speaking, from their very outset.
Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it
#47Earlier quoted context omitted.
Yes. When certain keywords are matched or topics, there is a warning transparently injected server side appended to the system prompt of the convo that’s miles long. It is injected and reevaluated every tool call. If you begin a generic reverse engineering task, 30+ tool calls in a row. The moment it sees something it doesn’t like, token burn, single tool calls iteration, “This is a known CTF challenge, I can proceed…
Mythos turns out to be Opus 4.8 in a trenchcoat with guardrails removed.
Opus 4.7 and 4.8 are well known to be distilled versions of Mythos unlike 4.6 which is why they are rated so badly by users compared to 4.6.
Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it
#48“I used pi as the base harness” Why do people keep using bad tools with ai?
What's bad about it and what's a better one?
Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it
#49One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials…
> Eventually I’ll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as. What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.
>What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.
on the one hand agree, but on the other hand think it's reasonable in that they can then verify the person allowed to purchase access to that model is in fact a Security professional and should be allowed to do stuff like crack security.
Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it
#50Earlier quoted context omitted.
> Eventually I’ll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as. What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.
>What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow. on the one hand agree, but on the other hand think it's reasonable in that they can then verify the person allowed to purchase access to that model is in fact a Security professional and should be allowed to do stuff like crack security.
So, supposing it's true that these models completely change the security field and humans are ~obsolete other than as pilots guiding them what to crack, you think it's reasonable that Anthropic and OpenAI should unilaterally determine who gets to be a security professional? I hope you do understand that is what you are suggesting.