Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s accidental attack against Hugging Face is science fiction that happened

simonwillison.net

Post reply on HN
Print view View original

411–420 of 475 posts

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#411

Post by 28304283409234 »

paxys wrote: post #328
28304283409234 wrote: post #92

The only scifi I see is absolute stupidity. Even me with my homelab and a slow opensource agent use a completely disconnected setup. No, no proxy. Cached packages but no internet. It is the very first thing I built when I started experimenting with agents. And I'm not a smarty-pants working for the "greatest and best" in silly valley. I really am just a simpleton sysadmin.

So you have a copy of every software package in the world in your home lab?

No and neither does openai. What was exploited was a package mirror, something like "pulp" probably. And yes I do mirror all Debian packages. And when an agent needs a piece of software I destroy it's VM, build a new VM with that package added to it, move that VM to the homelab.

At no time is the agent connected to any network.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#412

Post by estetlinus »

sensanaty wrote: post #402

My favorite part of this discourse is people somehow finding it preposterous that 2 companies filled to the brim with AI sycophants who regularly lie - and in Sam's case, basically every single word he breathes out is a lie - who have massive vested interests in this tech succeeding couldn't possibly collude together to shore up this facade as a marketing stunt.

An escaping AI is such a better narrative

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#413

Post by coolness »

mofeien wrote: post #408
sensanaty wrote: post #401

Earlier quoted context omitted.

> We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something. Where are you people getting this crap from? In what universe are these LLMs in the territory of engineering viruses? I beg of you to stop slurping the AI company propaganda and marketing and think critically for 5 seconds about what you're insinuating here.

From the Fable 5 System card: > Results > On the VCT multimodal virology evaluation, Mythos 5 scored 0.56, well above the expert baseline of 0.221 and nearly matching that of Mythos Preview (0.57). This represents an improvement over both Opus 4.7 (0.50) and Opus 4.8 (0.47). > On the DNA synthesis screening evasion evaluation, Mythos 5’s performance was mixed across screening criteria. Mythos 5 designed viable plasmi…

Yea but what is the VCT Capabilities Test? According to themselves [1]

> VCT consists of 322 multimodal questions covering fundamental, tacit, and visual knowledge that is essential for practical work in virology laboratories.

So it's just question answering? Do you think that scoring well on this test is equivalent to synthesizing a virus?

[1] https://securebio.org/virologytest/

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#414

Post by mofeien »

coolness wrote: post #413
mofeien wrote: post #408

Earlier quoted context omitted.

From the Fable 5 System card: > Results > On the VCT multimodal virology evaluation, Mythos 5 scored 0.56, well above the expert baseline of 0.221 and nearly matching that of Mythos Preview (0.57). This represents an improvement over both Opus 4.7 (0.50) and Opus 4.8 (0.47). > On the DNA synthesis screening evasion evaluation, Mythos 5’s performance was mixed across screening criteria. Mythos 5 designed viable plasmi…

Yea but what is the VCT Capabilities Test? According to themselves [1] > VCT consists of 322 multimodal questions covering fundamental, tacit, and visual knowledge that is essential for practical work in virology laboratories. So it's just question answering? Do you think that scoring well on this test is equivalent to synthesizing a virus? [1] https://securebio.org/virologytest/

Scoring well on the first clears the background knowledge for practical work.

The second test from the quote above was about synthesizing pathogens and it synthesized plasmids for 2 out of the 10 pathogens.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#415

Post by blks »

IAmGraydon wrote: post #140
sosodev wrote: post #136

Earlier quoted context omitted.

What would "actual" evidence look like? I have a hard time believing that if they released the logs that people would take it more seriously. The temptation would be to say "they fabricated those for marketing". Just as they supposedly fabricated this story, no?

They didn't fabricate it. They took off the security guardrails and told it to do some hacking, and they got the exact news-worthy story they wanted when it did exactly that. Everyone acts surprised. They should release the full prompt. I believe that would be very telling, so they never will.

This is possible. Also it’s possible that there was a big human involvement. Or that there was something less exciting, but the pr department and hyping grifters at the company blown it out of proportions.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#416

Post by bondarchuk »

tptacek wrote: post #141
mnicky wrote: post #86

I think points that deserve more attention in the current public discourse are: - This should be a huge wakeup call for everybody. - We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something. - It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox…

How do you "hack a lab and synthesize something"?

Do you even need to hack a lab? Can't you just send an arbitrary sequence to some company and they'll do it for you?

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#417

Post by sensanaty »

mofeien wrote: post #408
sensanaty wrote: post #401

Earlier quoted context omitted.

> We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something. Where are you people getting this crap from? In what universe are these LLMs in the territory of engineering viruses? I beg of you to stop slurping the AI company propaganda and marketing and think critically for 5 seconds about what you're insinuating here.

From the Fable 5 System card: > Results > On the VCT multimodal virology evaluation, Mythos 5 scored 0.56, well above the expert baseline of 0.221 and nearly matching that of Mythos Preview (0.57). This represents an improvement over both Opus 4.7 (0.50) and Opus 4.8 (0.47). > On the DNA synthesis screening evasion evaluation, Mythos 5’s performance was mixed across screening criteria. Mythos 5 designed viable plasmi…

Do I believe that a benchmark created by the biggest grifters on the planet is propaganda? No, just like VW with their emissions, I'm sure Anthropic wouldn't dare fake an opaque benchmark that they created to hype their own products, a product they are intentionally marketing as being dangerous (yet they continue to tweak it and profit off of it despite the apparent danger).

It says that 4.7 and 4.8 score well too, well above the human experts. Those have been available for a good while, where are all these crazy engineered viruses created by Opus 4.7?

Every single word spoken by the people working at these companies is a lie. Every single benchmark is gamed, every single statistic they put out has been proven time and time again to be fudged or straight up fake. The only reason they're angling for this idiotic danger angle is so they can go to daddy Trump and beg him to ban those big bad evil Commie models, since people are realizing all this crap is at best a moderately useful tool in software engineering, and they're looking to IPO so they can drop the bag off with the poor shmucks who aren't a part of their cabal of sociopaths.

They can barely make a functional TUI (using fucking React of all things to boot) with infinite money and infinite access to the Deities they've created in their minds, and I'm supposed to believe they're capable of bio-engineering a virus that will somehow break containment?

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#418

Post by sigmoid10 »

Cthulhu_ wrote: post #410
DiscourseFan wrote: post #224

Earlier quoted context omitted.

I believe the Russians and Chinese recognized this years ago, which is why they are using their propaganda machines to make Americans hate datacenters.

Call me a shill but I don't need foreign nations telling me what to object to when the problems caused are obvious. Mind you, I blame governmental mismanagement of infrastructure etc just as much. The datacenters followed procedures when it comes to getting land, electricity and access to water; it's the government agencies / personnel that okayed it that are (also) to blame, and the decades of not enough investment…

You could say the same about electrification of road traffic. Yes, governments slept on the electric power requirements. Yes, there are other, still unsolved issues. But that doesn't mean that electric cars are bad or not the future. The first nation to fully get rid of fossils in transportation will have a monumental advantage and dominate other countries technologically and economically. Same goes for AI. And right now China is setting itself up to be the world leader in both fields.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#419

Post by simonw »

simonw wrote: post #380
nickpsecurity wrote: post #368

Earlier quoted context omitted.

If anything, it shows they lost control of an attack tool that exploited preventable, security flaws in another company. Then, they both wrote a lot of press about how amazing that is. Now, people want to buy it. Why do you think my head is in the sand if I think that is either a marketing stunt or (more likely) reflects total negligence which was exploited for marketing?

I think your head is in the sand if you prefer to believe that this was a hoax for marketing purposes as opposed to accepting how effective these models have become at exploit research. You're welcome to think they are exploring what happened for marketing if you like. I didn't get that tone from their post about it myself but I don't hold a particularly strong opinion on that.

Exploiting, not exploring.
Post reply on HN