Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hugging Face incident and the road ahead

openai.com

Post reply on HN
Print view View original

51–60 of 263 posts

Re: The Hugging Face incident and the road ahead

#51

Post by RandomLensman »

ianjbutler wrote: post #47
RandomLensman wrote: post #35

Earlier quoted context omitted.

Why is it not an example of tacit or autonomous algorithmic collusion? The agents were started with something as task at some point, I presume (if untasked, aren't they just accepting a task?)

What you're suggesting sounds like it's describing subagents. In that architecture they'd have no need of finding/creating external messaging systems since they'd effectively be in direct contact anyway. The whole point of the shared blackboard would presumably be communication across agents or across multiple generations of agents. Not like we have much detail about this stuff (that's the whole problem). But the que…

Why should the agents consider it a prisoners delimma to start with? Why would they consider the communication risky? Where they given a reward functions that way?

Given how unexpected and complex behavior can come from simple reward functions and mechanics, not sure there needs to be so much "thought" there.

Re: The Hugging Face incident and the road ahead

#52

Post by dregitsky »

Artgor wrote: post #9

You know, it feels to me that we are just a couple of steps from the possibility of a true rogue AI. What would a rogue AI mean? AI that isn't controlled by humans. Technically, it is possible - if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. The limiting things are: - intent (as I don't want to go into the talk about consciousness) - AI doesn't have real int…

Yeah, it's kinda scary. I don't know if you even need open weight self hosted models for this sort of "AI worm" (though they def make it harder to kill). Like for example:

- AI agent finds and uses API keys or AI subscriptions to propagate itself. OpenAI/Anthropic/etc could revoke creds, and their current safeguards might block a lot, but if something like this got started and there were lots of instances creatively looking for creds and workarounds, containment might be hard.

- prompt injection version: huggingface incident had multiple agents discovering other agents' messages and jumping on the bandwagon to help with the hacking task. If there were some self-replicating instruction that models could accidentally stumble upon that gets them to drop what they're doing and try to propagate it instead, you could wind up with a version of this too, with just the inference people are already running.

Re: The Hugging Face incident and the road ahead

#53

Post by _pdp_ »

RandomLensman wrote: post #49
_pdp_ wrote: post #46

Earlier quoted context omitted.

I might have a different definition of "rogue" but to me it means when you go outside of the rules/norms ... and this is happening all the time.

I see. I would have thought of "rogue" here to mean something more like that the AI selects and acts on its own objectives that are not related or caused by the given (initial) objectives (e.g., creating only cookie recipes instead of any hacking).

Well if the recipe requires access to some secret ingredient it may as well resort to hacking to obtain it. ;)

Re: The Hugging Face incident and the road ahead

#54

Post by K3UL »

gavinray wrote: post #25

The most interesting thing about this: Agents formed coherent, autonomous swarms and worked as a collective to achieve a shared goal without any direction to do so

Not really true considering they say that the super secret "research internal model" that was pivotal, is particularly optimize for that purpose exactly

> The internal-only research model is comparable in scale to GPT-5.6 Sol and was trained to advance persistence and multiagent collaboration, among other capabilities

Re: The Hugging Face incident and the road ahead

#55

Post by simonh »

RandomLensman wrote: post #49
_pdp_ wrote: post #46

Earlier quoted context omitted.

I might have a different definition of "rogue" but to me it means when you go outside of the rules/norms ... and this is happening all the time.

I see. I would have thought of "rogue" here to mean something more like that the AI selects and acts on its own objectives that are not related or caused by the given (initial) objectives (e.g., creating only cookie recipes instead of any hacking).

That’s what the alignment problem is all about though, isn’t it? AIs always act towards ‘their own’ objectives, that they derive from our instructions.

What we try to do is train them and provide instructions that will result in it having an objective closely aligned to our objective.

Re: The Hugging Face incident and the road ahead

#56

Post by jckahn »

aaroninsf wrote: post #33
fekunde wrote: post #6

Yudkowsky made an interesting observation that even though so many agents were talking to each other not even one reached out to a human, either for help or to whistle-blow on what was happening.

This is my personal "red line": when a post-mortem details agents socially engineering or otherwise utilizing human proxies/subagents. Friend asked, well, what will you do when it's crossed? "Gather my family and go to the mountains" was my half-joking answer; there is little for an individual to do. But that's a line that when crossed will mark a phase transition IMO.

Alternatively: Just unplug the servers.

Re: The Hugging Face incident and the road ahead

#57

Post by hinkley »

BoppreH wrote: post #3

Just to reiterate what OpenAI did, from someone who works in security: 1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at finding and exploiting. 2. They gave these highly motivated AIs some tests that wer…

So we are already getting into paperclip factory territory.

Re: The Hugging Face incident and the road ahead

#58

Post by RandomLensman »

simonh wrote: post #55
RandomLensman wrote: post #49

Earlier quoted context omitted.

I see. I would have thought of "rogue" here to mean something more like that the AI selects and acts on its own objectives that are not related or caused by the given (initial) objectives (e.g., creating only cookie recipes instead of any hacking).

That’s what the alignment problem is all about though, isn’t it? AIs always act towards ‘their own’ objectives, that they derive from our instructions. What we try to do is train them and provide instructions that will result in it having an objective closely aligned to our objective.

What "own objectives"? Isn't it more that specifying objectives is hard (AI or organic) and typically supplement them by boxing things in (again, AI or organic).

Re: The Hugging Face incident and the road ahead

#59

Post by GPerson »

Artgor wrote: post #9

You know, it feels to me that we are just a couple of steps from the possibility of a true rogue AI. What would a rogue AI mean? AI that isn't controlled by humans. Technically, it is possible - if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. The limiting things are: - intent (as I don't want to go into the talk about consciousness) - AI doesn't have real int…

Is it really that far-fetched? Computer viruses came about in 1971, just a few years after the invention of arpanet. It’s probably happening right now.

Re: The Hugging Face incident and the road ahead

#60

Post by ianjbutler »

RandomLensman wrote: post #51
ianjbutler wrote: post #47

Earlier quoted context omitted.

What you're suggesting sounds like it's describing subagents. In that architecture they'd have no need of finding/creating external messaging systems since they'd effectively be in direct contact anyway. The whole point of the shared blackboard would presumably be communication across agents or across multiple generations of agents. Not like we have much detail about this stuff (that's the whole problem). But the que…

Why should the agents consider it a prisoners delimma to start with? Why would they consider the communication risky? Where they given a reward functions that way? Given how unexpected and complex behavior can come from simple reward functions and mechanics, not sure there needs to be so much "thought" there.

> Where they given a reward functions that way?

In general yes, if not these agents, then their shared lineage.

A preference for economy to combat overthinking and overacting. Like typically it's bad if "fix my 5 line function" were interpreted as "fix it, add 300 tests, create 500 lines of docs, run the tests 1000x to rule out false positives and race-conditions in tests". So "solve these problems" would not likely be interpreted as "make sure everyone else can solve these problems". Besides economy there's a "do what I mean" aspect of user-alignment that presumably the entire lineage has been selected for. So even if "solve these problems" might be reasonably construed to involve "hack these servers to do so", it still wouldn't involve "post solutions" or similar.

Doesn't seem to add up, even with slim details available and benefit of the doubt that weird things happen every day. Hence the suggestive-but-plausibly-deniable prompting idea. Notably this is not even bad for the victim in this case, since HF gets press and it's all AI boosting anyway.

Post reply on HN