Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cybersecurity researchers aren't happy about the guardrails on Anthropic's Fable

techcrunch.com

Post reply on HN
Print view View original

531–540 of 570 posts

Re: Cybersecurity researchers aren't happy about the guardrails on Anthropic's Fable

#531

Post by prmoustache »

staticman2 wrote: post #455
prmoustache wrote: post #428

Earlier quoted context omitted.

> It is not reasonable for them to say "sure, here's your alcoholic drink as you requested" and give them an alcohol-free substitute without telling them. Your analogy doesn't work because: - they tell you the rules at the entrance of the bar - they totally tell you when they give you a substitute The only issue is the bartender asking you for your money before serving you the drink really but again, this is known si…

Your rebuttle seems to be arguing it's okay for a bartender to simultaneously say: "This is alcohol" And "Or maybe it isn't alcohol." Or to rephrase it, "They tell you the rules at the entrance, they then tell you they don't follow those rules and they are totally serving alcohol even if they are not."

No they tell you at the entrance that at any point they may unilaterally decide to replace the alcoholic drink you ordered by a non alcoholic one.

You can decide you are okay with that or not but they aren't dishonest. I wouldn't enter that bar personally but if you do you cannot really complain. It is like complaining because you haven't won at the casino.

Re: Cybersecurity researchers aren't happy about the guardrails on Anthropic's Fable

#532

Post by hedgehog »

brookst wrote: post #464
hedgehog wrote: post #289

Earlier quoted context omitted.

The "tradeoff" warning implies they stand by their thinking and don't think there was anything qualitatively wrong with it which, if nothing else, is helpful so potential customers can know how they think. I think the core lesson is if you want reliable infrastructure to build into an application you should use a different provider. (edit: I'm not specifically an Anthropic hater, but having just spent some time addin…

Is it not a trade off? I think they made the wrong choice, but it seems reductive so say there was no choice at all and should never have been consideration of trade offs of silent versus not. Even wide open, uncensored models are often the product of a deliberate choice. I have a hard time faulting people for intentionality (even when they get it wrong).

They have a lot of choices, why would that specifically be a tradeoff? It's common for people to construct a tradeoff under which their preferred action is the more virtuous option, and thus they can be "the good guys", but that doesn't mean their framing makes any sense at all. Silently downgrading requests to a weaker model and billing the customer at full price, then framing the debate as how much (not if) this behavior is correct, that's an expression of values. People make mistakes all the time, if they thought it was actually wrong they could well have said so and explained what corrective action they've taken. One of the most famous examples of doing this right was the Pentium FDIV bug. Intel stood behind the product by recalling the affected units at great expense, and that (rightly) earned a lot of trust for decades.

Re: Cybersecurity researchers aren't happy about the guardrails on Anthropic's Fable

#533

Post by Spooky23 »

consumer451 wrote: post #520
Spooky23 wrote: post #518

Earlier quoted context omitted.

Sure. It's really about informed consent and acceptance of risk. I'm very conservative about that due to my background and business. Say you have some flow that is processing/handling regulated, sensitive or other customer data with the LLM as part of an operational process. An example that I'm thinking of is for a customer who wants to more efficiently resolve or route IT incidents to the right place. The incident d…

Oh man, thanks for taking the time to reply. I feel a bit better now, lol. If you read my rant linked previously, yeah... we are on the same page. As another user pointed out in that thread, the issue here is that even on Bedrock and Azure Foundry, now with Fable 5, Anthropic inserts themselves as an additional data subprocessor that we would have to consider and certainly disclose, correct? That kind of destroys the…

Yeah tbh I may have read past some of your previous post :) What you’re saying is what makes me nervous.

It was definitely sold as “anthropic IP, thorough your old pals at the hyper scaler”. And it’s turning into something else — I’m having lunch with AWS and this other guy showed up with them.

Re: Cybersecurity researchers aren't happy about the guardrails on Anthropic's Fable

#534

Post by barnabee »

mschuster91 wrote: post #444
ungovernableCat wrote: post #186

Wait a few months and a competitor will release a similarly powerful model with less guardrails, if they steal sufficient market share Anthropic will reverse policies. This is why I’m immensely hoping the Chinese don’t stop with their open sourced local models. None of these companies are your friend.

> This is why I’m immensely hoping the Chinese don’t stop with their open sourced local models. None of these companies are your friend. The Chinese aren't your friend either [1]. [1] https://www.hks.harvard.edu/centers/carr-ryan/our-work/carr-...

Indeed, but the weights they release are valuable in a way that precisely counteracts the fact that none of these companies are your friend.

Re: Cybersecurity researchers aren't happy about the guardrails on Anthropic's Fable

#535

Post by consumer451 »

Spooky23 wrote: post #533
consumer451 wrote: post #520

Earlier quoted context omitted.

Oh man, thanks for taking the time to reply. I feel a bit better now, lol. If you read my rant linked previously, yeah... we are on the same page. As another user pointed out in that thread, the issue here is that even on Bedrock and Azure Foundry, now with Fable 5, Anthropic inserts themselves as an additional data subprocessor that we would have to consider and certainly disclose, correct? That kind of destroys the…

Yeah tbh I may have read past some of your previous post :) What you’re saying is what makes me nervous. It was definitely sold as “anthropic IP, thorough your old pals at the hyper scaler”. And it’s turning into something else — I’m having lunch with AWS and this other guy showed up with them.

No worries :) What this showed me is the power/velocity/inertia that Anthropic can hold over the 3rd party providers. Like, they should have pushed back on this, as it must have been clear to the 3rd parties that this change was a big deal to their customers... and yet, it went how Anthropic wanted it to go.

Re: Cybersecurity researchers aren't happy about the guardrails on Anthropic's Fable

#536

Post by echelon »

lbreakjai wrote: post #406
echelon wrote: post #221

Earlier quoted context omitted.

These coding agent models only started getting useful in January. Before that they were difficult to control autocomplete, and not very smart. January was an inflection point, and no open weights model has crossed over that same threshold. This is definitely recursive self improvement territory, except that we're prohibited from participating. It feels like the capability gap is wider than before.

Have you tried deepseek V4? It costs pennies and is as good as Opus 4.6 (I found 4.7 to be a downgrade, and cancelled my claude subscription before 4.8). The threshold has definitely been crossed.

It is not as good as Opus. I've tried to write Rust with it (and Codex for that matter), and it's awful.

Re: Cybersecurity researchers aren't happy about the guardrails on Anthropic's Fable

#537

Post by kmeisthax »

intended wrote: post #388
gck1 wrote: post #360

Earlier quoted context omitted.

OpenAI has a real opportunity to do some sort of "we don't maliciously alter your prompt and nerf the model" with some form of verification, when they release the next model. But if Anthropic gets their way with regulatory capture, this could be the only future we'll see. To think that they didn't expect the backlash speaks volumes about how much shady things they're doing which is not publicly known.

Eh, I expect open Ai to follow suit. I suspect this is surprising to folk because they aren’t the ones busy figuring out how to use LLMs for illegal acts. In general, HN users focus on making stuff, and not the safety side of things, or the scale of harms being enabled via LLMs and generative AI. If you are on the safety side of things the ratio of misuse to fair use is inverted and everything is at scale. Transparen…

Yes, but there is a very specific subset of things AI companies will and won't cite safety for as a concern, and that subset intersects neatly with things the companies consider to be business risks. Like, the main reason why AI companies are so willing to poison the well is because there's no money in selling to the kinds of people who want to write malware[0].

The correlation between how bad an AI safety risk actually is and how much the companies in question will actually talk about it is almost perfectly negative. The poster child of this is AI superintelligence; companies love to talk about how dangerous the AI they are actively trying to build is. But superintelligence is also a really vague concept without a clear definition. If we naively define it as "an AI system that is better than a human in some aspect", then it already exists. These models already read and write at superhuman speed.

"That's not real superintelligence!" you say. But that's exactly the capability you need in order to flood every online forum with an unending tide of AI slop. And I don't remember, say, OpenAI saying they were shutting down Sora because it was destroying or defacing human culture[1]. They shut down Sora because it was way too expensive to run.

Meanwhile, Sam Altman went and bragged about how he wants ChatGPT to make erotica. Y'know, as if we don't already know that character.ai gooning is about as safe for your mental health as Action Park was for your physical health. But porn is also a huge market, so obviously he and all the other AI companies want in on it, even though the "sexy suicide coach" is already a well-documented harm of AI.

And the idea that distillation is an attack is laughable. Like, I get the logic - if someone can ask the AI to make another AI then they get to change the guardrails - but it's still ultimately just Anthropic objecting to their own conduct when it happens to them. All their models are trained on nonconsensually harvested data. There is no moral or legal principle where Anthropic gets to use my data without permission but I don't get to use theirs.

Furthermore, AI safetyism runs up against "Freedom Zero", a core tenet of the Free Software ethos: you should be allowed to use software in any way you choose. This is not a call for more people using AI for evil, but a call to recognize that people should be allowed to use their property as they wish. Making software disobey its owner is malicious behavior. And every single time safety considerations are brought up it is to justify further attacks on Freedom Zero. And these justifications are always self-serving. There is no context in the world where a frontier AI lab asking someone else's AI about AI research is intrinsically harmful; especially not to the point where we need to make Claude deliberately sabotage your work. That is malware. Anthropic shipped malware. This is inexcusable.

[0] Digital or biological.

[1] https://www.youtube.com/watch?v=YCPAIg7RUq8

Re: Cybersecurity researchers aren't happy about the guardrails on Anthropic's Fable

#538

Post by DyslexicAtheist »

brookst wrote: post #462
saidnooneever wrote: post #449

Malware authors are pretty excited about guard-rails. you can add prompts to your malware to get LLM scanners to hit guard-rails and stop their runs. New shai-hulud npm worm campaign for example includes prompts to request biological weapon schematics/creation etc. to ensure LLM scanners probing NPM packages refuse to scan it. These AI places have 0 clue about how threat actors actually work. None of their mitigation…

I’ve never understood the “if I don’t enable bad behavior, someone else will, so I might as well enable bad behavior” argument. Can you elaborate? From where I sit it seems reasonable for Anthropic to not want their product used to create malware, even if they can’t solve the entire problem globally for every model. What’s wrong with that position? What should they do differently?

the problem is that the guardrails prevent us from performing real security work which is friction that is incurred by the legitimate user but not by a moderately sophisticated threat-actor.

for example in my org it is part of the culture that security has no seat at the table. that is a separate problem, but the number of orgs like mine are more numerous than the number of orgs where security isn't a cost-center.

we find lots of stuff because low-hanging fruit is everywhere. hecking heck: I'm a fruit.

and when the cost of fixing is even the slightest inconvenience to devs we will not fix it, but continue sitting on the risk until the cows come home. In such a place a new critical finding isn't even novel. Instead our job moves to to combining different vulns that we already have, and try to show managers how bad it is.

the common retort from management is: proof to me why this is an issue, and why engineering should divert their attention to it. And unless my team can proof why X can be exploited, or Y can be bypassed, or Z can gain persistence, ... the vulnerabilities will remain. I have been in discussions where the business demanded to see an exploit so they can justify the cost of fixing it. low-cyber-maturity doesn't even describe it. we are not a mom and pop shop but have 110K employees worldwide. and again - we are not uniquely insecure.

so these guardrails aren't helping because the moment the chat has any offsec artifacts, or even just a single wrongly worded phrase anywhere in the workspace, the session is flagged, you need to downgrade the model.

what adds insult to injury, is that the guardrail is just a way to funnel users into the Ai company's "cyber marketing" program: "your chat has been flagged, please proof your identity and hand over your passport data so you can sign up to our TrustedCyber program". Bitch please you have my payment information, use that??

if you consider bug-density (security defect density) per LoC, it is even more of a sh1t show: no restrictions apply for developers to push their buggy code, but the security team needs to somehow proof that they aren't the malicious party?

totally off - considering the right way to build defensive/offsec/malicious tooling with AI isn't by using frontier models ... but run a serious of agents on tightly scoped tasks. see https://securitycryptographywhatever.com/2026/03/25/ai-bug-f... and https://aisle.com/blog/ai-cybersecurity-after-mythos-the-jag... - this shuts out the average joe who works in an org where cyber security maturity is poor. joe does not know about how to orchestrate a fleet of agents and give them muppet names. all he knows is that the good guys are losing the fight.

Re: Cybersecurity researchers aren't happy about the guardrails on Anthropic's Fable

#539

Post by dontlikeyoueith »

brookst wrote: post #483
saidnooneever wrote: post #480

Earlier quoted context omitted.

all their decisions are based on sales. like other corporations especially those going for IPO. thats absolutely true. Any messaging outbound will be for that purpose mostly from a business perspective, regardless of what opinions or ideals the involved persons hold personally. Its good to keep that in mind indeed when looking at these things. People arent evil, but business incentives can definitely paint such a pic…

> all their decisions are based on sales That’s the edgy cynical thing, and too reductive to be meaningful. For one thing, it assumes perfect knowledge of how a decision will impact sales, which I assure you is not remotely the case. Agreed on incentives, but it’s not binary. I’ve been involved in plenty of decisions in multiple Fortune 500’s where the deciding factors were taste, wanting or not wanting to work with…

It's an optimal first order approximation.

Anything anyone with a capital-C in their job title says in public should be assumed to be marketing material.

Post reply on HN