Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alibaba to ban Claude Code in workplace over alleged backdoor risks, source says

reuters.com

Post reply on HN
Print view View original

271–280 of 299 posts

Re: Alibaba to ban Claude Code in workplace over alleged backdoor risks, source says

#271

Post by arw0n »

seanmcdirmid wrote: post #221
SubiculumCode wrote: post #202

Earlier quoted context omitted.

Why make this u.s. centric? You think China served models would be different?

It’s pretty hard to put a backdoor in a bunch of model weights. Maybe not impossible mind you, but I can’t fathom how you would do it.

Not really, it is shockingly easy for what it is. https://arxiv.org/abs/2401.05566

This only really matters in a world where Prompt Injection and Jailbreaking isn't trivial in the first place though. All current models are still extremely exploitable.

I strongly suspect we are only scratching the surface of activation engineering at the moment, and there's plenty of very targetted ways of lobotomizing or cracking LLMs if you understand the model in detail.

Re: Alibaba to ban Claude Code in workplace over alleged backdoor risks, source says

#273

Post by hakfoo »

bhouston wrote: post #55

All remote AI are a massive security risk for individuals/companies/governments that may be targeted by the US government. It is likely that the US will get a live feed from each AI provider that they are inspecting in real time to identity things of interest, terrorist attacks or foreign government planning or even foreign companies competitive to key US companies. It will give them access to the though process in t…

I think we can even skip the "that may be targeted by the US government" clause.

The whole "hosted AI" business feels like like a huge violation of corporate norms on confidentiality. Businesses that would have your head for printing out a source file to reference and annotate are encouraging developers to feed in huge amounts of proprietary code and data, and incorporate changes suggested from an outside party with minimal vetting. Evidently whatever privacy policies they've been throwing at enterprise users are plated with mithril.

At some point, one of the big services is going to get popped, and it won't just be a data breach. There's too much opportunity to quietly use the system as a malware distribution hub. Every vibe-coded dashboard suddenly starts depending on some weird left-pad fork that, 12 dependencies deep, is running a keylogger or Dogecoin miner. Your payment processor suddenly starts accepting the Konami code to approve a transaction.

Re: Alibaba to ban Claude Code in workplace over alleged backdoor risks, source says

#274

Post by youre-wrong3 »

vrganj wrote: post #223
breppp wrote: post #215

Earlier quoted context omitted.

The Great Leap Forward and the Cultural Revolution are two such examples Generally Communist nations historically favored technological development to human life in the scale of millions, keep that in mind when we enter a new economic revolution

The Great Leap Forward wasn't "killing" people, which implies intent. It was just good old economic mismanagement. On a related note, around 300k people die in the US every year due to causes directly attributable to poverty. [0] In other words, ~a million every three years. Now what? [0] https://pmc.ncbi.nlm.nih.gov/articles/PMC10111231/

On one side you have people starving because they are forced to grow food which is effectively stolen and sold to enrich the party. People killed fr starvation, or by the party if you didn’t contribute enough, ate some, stole some. So the deaths are the direct result of the CCP.

Vs a study which suggests 183-300k where you take the high end and seem to read about the vast range of causes of which a lot are not very much attributable to poverty.

Re: Alibaba to ban Claude Code in workplace over alleged backdoor risks, source says

#275

Post by vrganj »

youre-wrong3 wrote: post #274
vrganj wrote: post #223

Earlier quoted context omitted.

The Great Leap Forward wasn't "killing" people, which implies intent. It was just good old economic mismanagement. On a related note, around 300k people die in the US every year due to causes directly attributable to poverty. [0] In other words, ~a million every three years. Now what? [0] https://pmc.ncbi.nlm.nih.gov/articles/PMC10111231/

On one side you have people starving because they are forced to grow food which is effectively stolen and sold to enrich the party. People killed fr starvation, or by the party if you didn’t contribute enough, ate some, stole some. So the deaths are the direct result of the CCP. Vs a study which suggests 183-300k where you take the high end and seem to read about the vast range of causes of which a lot are not very m…

And on the other side you have people dying because they are denied food, shelter and healthcare as a policy choice.

It's not that they can't be taken care of. It's just that it'd cost money and eat profit margins.

It's an active political and ideological choice to let them die, same in both cases. Just happens to be an ideology you agree with and feel a need to defend.

The 180k are currently poor people, the 300k people poor over the past 10 years (which includes the 180). Didn't take the larger number, took the one that more accurately applied.

Re: Alibaba to ban Claude Code in workplace over alleged backdoor risks, source says

#276

Post by wren6991 »

causal wrote: post #240
breppp wrote: post #57

Earlier quoted context omitted.

The issue here is not whether Anthropic used Common Crawl, Alibaba also does that. The issue is that by distilling Claude, Alibaba reuses the IP anthropic used to train the model that's more akin to historical Chinese reverse engineering methods and disrespect of IP

I wish people would stop using Anthropics incorrect use of the term distill. They don’t share logits so you can’t distill. You can generate training data, which doesn’t sound nearly so scary.

Why do you need logits to distill? Those are at least tokenizer-dependent, and different models use different tokenizers.

Re: Alibaba to ban Claude Code in workplace over alleged backdoor risks, source says

#277

Post by stackskipton »

wqaatwt wrote: post #260
shimman wrote: post #224

Earlier quoted context omitted.

What absolute bollocks. Human ingenuity and innovation is only limited by the greed of elites, not due to something as damaging as "IP." Good grief. All one has to do is look at how humanity has consistently progressed due iterating on what has existed is how we progress, not whether some corporation that wants to rat fuck us all for a few pts in share value.

> progressed due iterating on what has existed is how we progress Progress was extremely slow until the 1800s. Coincidentally corporation and modern capitalism in general developed around the same time. Of course I’m not necessarily saying it was the main or direct course since it isn’t exactly possible to create an experiment comparing it to other systems (of course that was tried an failed completely in the USSR, M…

Progress was slow until industrial farming was developed and more people could be freed from just trying to grow enough food to feed themselves.

Capitalism was side effect as well.

Re: Alibaba to ban Claude Code in workplace over alleged backdoor risks, source says

#278

Post by seanmcdirmid »

arw0n wrote: post #271
seanmcdirmid wrote: post #221

Earlier quoted context omitted.

It’s pretty hard to put a backdoor in a bunch of model weights. Maybe not impossible mind you, but I can’t fathom how you would do it.

Not really, it is shockingly easy for what it is. https://arxiv.org/abs/2401.05566 This only really matters in a world where Prompt Injection and Jailbreaking isn't trivial in the first place though. All current models are still extremely exploitable. I strongly suspect we are only scratching the surface of activation engineering at the moment, and there's plenty of very targetted ways of lobotomizing or cracking LLM…

You have to hide it in the model and it has to be subtle or it will be discovered quickly (even if you can train against a specific safety detector). Again, I'm not saying its impossible, but it seems really hard to pull off.

Re: Alibaba to ban Claude Code in workplace over alleged backdoor risks, source says

#279

Post by usef- »

wongarsu wrote: post #191
usef- wrote: post #73

Earlier quoted context omitted.

The article on HN only said that they seemed to be collecting this to detect resellers. How else did the behavior change? Most services I know that are trying to block abuse do collect device info

There is this whole thing where Fable silently starts behaving worse if they suspect you are trying to use it for RL or are otherwise building a competing product. This is likely the primary vector how that works: they check if you are in china, if you proxy your requests, and if you are from a list of known labs or match a couple keywords

They walked back that policy on the first day after pushback. They were upfront in the model launch that they designed it that way, it wasn't secret.
Post reply on HN