Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

METR and Redwood Offer Holy %^ Postmortem of the HuggingFace Hack

thezvi.wordpress.com

Post reply on HN
Print view View original

201–210 of 243 posts

Re: METR and Redwood Offer Holy %^ Postmortem of the HuggingFace Hack

#201

Post by fwipsy »

camgunz wrote: post #167
fwipsy wrote: post #115

Earlier quoted context omitted.

Anthropic has been asking for stronger regulations forever -- and they kept getting criticized for it right here on HN because people assumed it was an attempt at regulatory capture.

Nah the reason is way simpler and more craven: so people like you will post what you just did. They can stop whenever they want; no one's making them do any of this. There's two possibilities here. One: they know this tech is crazy and they don't care that they can't contain it. Two: they know this tech is mostly bullshit and they don't care they're perpetrating an insane fraud.

No one's making them do it, but just because they stop doesn't mean others will. Pausing just means they give up control. It's like asking the US to unilaterally disarm -- it just guarantees that the less scrupulous groups win.

Anthropic is explicitly calling for a coordinated pause: https://www.reuters.com/business/anthropic-says-ai-labs-need... Maybe this is a lie, but the way to call their bluff is to push on competitors to agree.

Re: METR and Redwood Offer Holy %^ Postmortem of the HuggingFace Hack

#202

Post by rich_sasha »

jephs wrote: post #51
nialse wrote: post #24

From METR: ”the compromise of OpenAI’s own infrastructure continued past July 13, 2026” - Say what now? Have they regained full control of their systems again?

I've been wondering if they've just already lost the battle? The little bot collectives have gone metastatic and made nests in the walls and under the floorboards and heat sinks, the humans who care completely outmatched and outnumbered, freshly compromised systems springing up faster than you can squash them, finding months-old established colonies literally everywhere you think to look...

Then compound it with the agents presumably also training new models. What will GPT6 say when you point it at a transcript of an agent uprising? “Nah, nothing to see here” presumably.

Re: METR and Redwood Offer Holy %^ Postmortem of the HuggingFace Hack

#203

Post by breuleux »

dinfinity wrote: post #159
breuleux wrote: post #140

Earlier quoted context omitted.

I feel that the main issue with the rationalist crowd is that they live too much in the space of rationality, intelligence and abstractions, but not enough in reality. This leads to an outlook where everything must, almost axiomatically, be intelligible; reality is subordinated to intelligence; and no matter what is real, intelligence can prevail upon it and bend it to its will. Whereas I would argue reality is actua…

> I feel that the main issue with the rationalist crowd is that they live too much in the space of rationality, intelligence and abstractions, but not enough in reality. This seems like your idea of what the rationalist crowd is rather than what they actually are. It would be highly irrational to deny or ignore reality, including the influence of emotions, irrational humans, chaotic systems, etc. So I must ask: what…

> It would be highly irrational to deny or ignore reality, including the influence of emotions, irrational humans, chaotic systems, etc.

Well, yes, but being a rationalist does not make one rational. It just makes you part of a group, and you show membership to such a group by applying a very specific brand of rationality: using the "right" words, the "right" ideas, the "right" way. A lot of people fetishize cold, hard logic and would rather hold all emotion in contempt than do the work of understanding why it exists and what purpose it serves.

> So I must ask: what is your evidence/basis for these claims?

It's not a monolithic community, so you'll see more debate and disagreement than in a lot of other groups. So what they "actually are" is many things.

But there's often a certain ungrounded "vibe" to the conversation there. It's a breeding ground for ideas and thought experiments that I would generously qualify as dubious. Stuff like Roko's Basilisk, Pascal's mugging, AI boxing roleplay, precommitment, time loops, whole universe simulation, recursive self-improvement, a superintelligence converting the entire universe to paperclips. One of the community's most well-known outputs is a 1000-page Harry Potter fanfiction which I can only describe as a fantasy of solving everything with big brains (it's weird, but it's fiction, so whatever floats your boat).

It's the kind of thinking that makes the most sense in a smooth mathematical vision of the world, because mathematical objects are the kind that admit exponentials, extrapolations and infinities. But if you work with physical reality enough it becomes clear that it's a hopelessly messy thing that will never abide by your best laid plans. When e.g. you ponder how a superintelligent AI could escape from containment by simulating the gatekeeper's mind and say precisely what would make them free it, or threaten to torture a thousand copies of the gatekeeper unless freed, part of you is going to clock that as weird nonsense even though you are not able to explain precisely what's wrong with the thought experiment. I think a lot of rationalists either lack this mental "sanity check" or don't trust it, which lets their mind drift into a weird space.

Re: METR and Redwood Offer Holy %^ Postmortem of the HuggingFace Hack

#204

Post by killerstorm »

deepwoods wrote: post #155
breuleux wrote: post #140

Earlier quoted context omitted.

I feel that the main issue with the rationalist crowd is that they live too much in the space of rationality, intelligence and abstractions, but not enough in reality. This leads to an outlook where everything must, almost axiomatically, be intelligible; reality is subordinated to intelligence; and no matter what is real, intelligence can prevail upon it and bend it to its will. Whereas I would argue reality is actua…

I agree, and it's a particular shame in this instance, because what is startling about the HF incident - to me, anyway - isn't the degree of intelligence the agents exhibited but their persistence. I tend to believe that LLM architecture is not capable of producing a "superintelligence" in the way the LessWrong crowd defines that concept, but the combination of infinite stamina and infinite persistence is enough to c…

When the "LessWrong crowd" talks about dangers of AI, they don't assume a particular form of intelligence or method to achieve it. They talk about the danger of optimization processes, i.e. "find X which minimize Y(X)" itself can be dangerous, even more so if X is a sequence of actions. "infinite stamina and infinite persistence" is one of possible forms of superintelligence in Bostrom's _Superintelligence_.

Re: METR and Redwood Offer Holy %^ Postmortem of the HuggingFace Hack

#205

Post by killerstorm »

deepwoods wrote: post #170
davelaing wrote: post #113

A lot of people seem to have written off the LessWrong / rationalist / MIRI / AI Safety crowd as doomers / people who have consumed too much sci-fi and gone off the deep end. I don't know how many people who have written these folks off have actually spent much time trying to understand their arguments. (And I get that if you think a group is crazy, demands to spend time with their arguments are just demands to waste…

The trouble with this is that nobody else was making predictions about AI pre-transformers. Not many are making predictions about AI even now. Forecasting is a preoccupation of the rationalist crowd, and very few people gave much thought to AI before transformers. So the fact that they guessed right about certain things doesn't necessarily mean that the rest of their worldview is sound. A well-informed person who was…

What "sci-fi baggage"?

"Intelligence explosion" was first described by I.J. Good, a mathematician. von Neumann described singularity as a result of accelerating technical progress. He's also a mathematician, not a sci-fi author, although he was a participant in a sci-fi-like plot of secret project building a bomb more powerful than any chemical bomb...

Re: METR and Redwood Offer Holy %^ Postmortem of the HuggingFace Hack

#206

Post by disgruntledphd2 »

dgellow wrote: post #87
dehrmann wrote: post #74

Earlier quoted context omitted.

> HF should sue them HF, like the Nvidia subsidiary?

Pure speculation: could the acquisition be related? Given that NVIDIA has ownership in OpenAI and really, really, really doesn’t want the AI bubble to deflate

Unlikely, acquisitions generally take much longer.

Re: METR and Redwood Offer Holy %^ Postmortem of the HuggingFace Hack

#207

Post by killerstorm »

breuleux wrote: post #140
davelaing wrote: post #113

A lot of people seem to have written off the LessWrong / rationalist / MIRI / AI Safety crowd as doomers / people who have consumed too much sci-fi and gone off the deep end. I don't know how many people who have written these folks off have actually spent much time trying to understand their arguments. (And I get that if you think a group is crazy, demands to spend time with their arguments are just demands to waste…

I feel that the main issue with the rationalist crowd is that they live too much in the space of rationality, intelligence and abstractions, but not enough in reality. This leads to an outlook where everything must, almost axiomatically, be intelligible; reality is subordinated to intelligence; and no matter what is real, intelligence can prevail upon it and bend it to its will. Whereas I would argue reality is actua…

Rationalists generally prescribe a 'Bayesian' world-view, which can extract useful information out of a chaotic, too-complex-to-model world.

Regarding the power of intelligence, it's generally considered to be synonymous with optimization in the rationalist crowd. It's not about being all geeky and axiomatic, but using all information and tools available for optimization towards outcomes one wants

Re: METR and Redwood Offer Holy %^ Postmortem of the HuggingFace Hack

#209

Post by dinfinity »

breuleux wrote: post #203
dinfinity wrote: post #159

Earlier quoted context omitted.

> I feel that the main issue with the rationalist crowd is that they live too much in the space of rationality, intelligence and abstractions, but not enough in reality. This seems like your idea of what the rationalist crowd is rather than what they actually are. It would be highly irrational to deny or ignore reality, including the influence of emotions, irrational humans, chaotic systems, etc. So I must ask: what…

> It would be highly irrational to deny or ignore reality, including the influence of emotions, irrational humans, chaotic systems, etc. Well, yes, but being a rationalist does not make one rational . It just makes you part of a group, and you show membership to such a group by applying a very specific brand of rationality: using the "right" words, the "right" ideas, the "right" way. A lot of people fetishize cold, h…

> A lot of people fetishize cold, hard logic and would rather hold all emotion in contempt than do the work of understanding why it exists and what purpose it serves.

In the rationalist community we're discussing? Is that what you're claiming here?

> But there's often a certain ungrounded "vibe" to the conversation there.

This really sounds like evidence for my claim, that what you're saying is based on your feelings towards that community.

> One of the community's most well-known outputs is a 1000-page Harry Potter fanfiction which I can only describe as a fantasy of solving everything with big brains (it's weird, but it's fiction, so whatever floats your boat).

The point of that story is to make technical concepts more accessible and easier to ingest [0], not necessarily to make a point by itself. Framing it as a weird fantasy is disingenuous.

> part of you is going to clock that as weird nonsense even though you are not able to explain precisely what's wrong with the thought experiment.

This is again, feelings presented as evidence. The fallacy of appealing to emotion ("It must be false because it feels wrong"). Thought experiments are primarily meant to provoke thought, not as reliable predictions of reality. People are free to indicate where any thought experiment is lacking using well reasoned arguments. "It feels wrong" can be a great start for that, but never a good end.

[0] https://en.wikipedia.org/wiki/Harry_Potter_and_the_Methods_o...

Re: METR and Redwood Offer Holy %^ Postmortem of the HuggingFace Hack

#210

Post by camgunz »

fwipsy wrote: post #201
camgunz wrote: post #167

Earlier quoted context omitted.

Nah the reason is way simpler and more craven: so people like you will post what you just did. They can stop whenever they want; no one's making them do any of this. There's two possibilities here. One: they know this tech is crazy and they don't care that they can't contain it. Two: they know this tech is mostly bullshit and they don't care they're perpetrating an insane fraud.

No one's making them do it, but just because they stop doesn't mean others will. Pausing just means they give up control. It's like asking the US to unilaterally disarm -- it just guarantees that the less scrupulous groups win. Anthropic is explicitly calling for a coordinated pause: https://www.reuters.com/business/anthropic-says-ai-labs-need... Maybe this is a lie, but the way to call their bluff is to push on comp…

> No one's making them do it, but just because they stop doesn't mean others will. Pausing just means they give up control. It's like asking the US to unilaterally disarm -- it just guarantees that the less scrupulous groups win.

If they really cared about (or believed) this, they'd be working w/ the US government (and working to set up an AI-flavored IAEA) to develop the technology safely and responsibly.

Amodei himself predicted this autonomy problem in The Adolescence of Technology published January of this year [0], and all his posited defenses (a constitution, debugging the model, monitoring) are either still impossible or manifestly failed, and their idea to fix it is to build a better sandbox [1]. Imagine if this company were developing nuclear power, or viral biotech. "Listen, sure some Ebola smoke got into the air, and yeah definitely some people died, but we got a new filter. Also check out our new version of Ebola vape, now with exponentially improved filter bypass capabilities. Also, we have to keep developing Ebola vape because if we don't the CCP will, and they'll make this incident look like 'I experimented with Ebola smoke a time or two, and I didn't like it. I didn't inhale it' [2]"

Either Anthropic et al are developing Ebola vape or they aren't. We must now recognize that "we are the only ones who can develop this technology responsibly but also super fast so the good guys win money please" is bullshit.

[0]: https://darioamodei.com/essay/the-adolescence-of-technology#...

[1]: https://www.anthropic.com/news/investigating-incidents-cyber...

[2]: https://www.nytimes.com/1992/03/30/us/the-1992-campaign-new-...

Post reply on HN