Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

Post reply on HN
Print view View original

351–360 of 439 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#351

Post by simonw »

angry_octet wrote: post #350
KingOfCoders wrote: post #210

Earlier quoted context omitted.

How can it escape an "install package " proxy? reducePrivs() serve get(package) { secPackage = secure(package) getBinaryFromArtifactory(secPackage) } I would think the code is very small and easier to verify, it doesn't especially have the ability to write files and act as a message board as Artifactory did. And even if the agent tries to hack that, the attack surface is 1000x smaller and the possibility also much sm…

The way they had Artifactory configured was poor, and they were too reliant on it working perfectly, with no reason for such faith. Their config lacked any defence in depth and consideration of having a small TCB. Part of the problem might be the lack of security focus, as these are AI R&D efforts first.

I think part of the problem is that they had been running that Artifactory configuration previously without any problems, and it gave them a false sense of security.

Similar thing happened with the UK AISI - they got caught out because the environments they had used for previous generation models turned out to be completely inadequate for the new generation of Fable-class models: https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag...

Re: Timeline of the OpenAI accidental attack against Hugging Face

#353

Post by skybrian »

thadk wrote: post #107

Simon's retelling is more compact but it also invites anthropomorphization of the sharing of the familiarity with the message board which re-emerged a few times. Zvi's retelling handles this better. Zvi speculates that the secret message board familiarity was carried because it had been trained into the May-and-subsequent models: https://thezvi.substack.com/p/openai-trained-its-models-for-...

Zvi’s write up has much more social media quotes and memes and speculation and left me looking for something else that’s shorter and more sober to share. Simon’s writeup is more like what I wanted.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#354

Post by angry_octet »

detourdog wrote: post #48
KingOfCoders wrote: post #43

"More agents discover this new informal message board while browsing Artifactory’s file listings, and start reading and writing messages." Yeah, my agents also discover what other agents have done on other machines by accident. Agents - that do totally different things all work on the same aim without the humans telling them to do. Either that is a model that is several generations of Claude Code Opus/Fable 5 (my dai…

The agents sound like old school hackers that would just explore what access they could gain. Creating a file for other hackers and themselves. The fact that there were 3 events for 3 major players does make it seem co-ordinated.

That's what attackers do now. Exploring is required for discovering exploits. But that is also where tricks like Canary Tokens and honeypots are useful.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#355

Post by cloverich »

mvdtnz wrote: post #270
pmarreck wrote: post #148

Earlier quoted context omitted.

What a paper! And you missed an even MORE relevant excerpt!! Man and Slave The problem, and it is a moral prob- lem, with which we are here faced is very close to one of the great problems of slavery. Let us grant that slavery is bad because it is cruel. It is, how- ever, self-contradictory, and for a reason which is quite different. We wish a slave to be intelligent, to be able to assist us in the carrying out of ou…

> Complete subservience and complete intelligence do not go together. Isn't this contradicted by the centuries of slavery in our history? Or is the author arguing that the people who were enslaved did not have human-level intelligence (which would be rather a problematic claim)?

The very same slavery which resulted in the civil war and literal killing of hundreds of thousands of non slaves, followed by their freedom? Or the prior enslavements that very often ended in organized rebellion? Slavery is at most a temporary phase when it involves beings of equivalent intelligence.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#356

Post by angry_octet »

varun_ch wrote: post #44
cadamsdotcom wrote: post #28

What isn't being discussed is what an indictment this is of Artifactory. Let's be real, it won't be simply replaced in millions of sites. What it needs is some serious scrutiny.

I also agree that a big issue here is crappy software. The discussion revolving AI+cyber always revolves around the assumption that all software is crappy, and to a certain degree that may be true, but we could also take our jobs seriously and write good software, and much of the risk would evaporate. The described Artifactory bugs should have been caught with testing. If the biggest impact of LLMs on the industry is…

I would love than, and it might happen as a process of natural selection, but instead we will get automated AI patch generation and patch application, and agentic EDR and agentic SIEM. All the while generating vast amounts of new vibe coded trash.

If I had the money I would invest in clever segmentation firewalls and application gateways, something like tailscale but requiring explicit permission to establish connection from A to B, that facilitates introducing monitors that validate and log.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#357

Post by Covenant0028 »

simoncion wrote: post #197
Covenant0028 wrote: post #138

Earlier quoted context omitted.

The entire economic premise and value case of LLMs rests on the idea that instructions need not be provided in advance, and that the model can "reason" based on evidence and "decide" what to do next. Even if it were technically possible to separate instructions from code and ensure that the LLM only followed those, it would require someone to specify the instructions in advance (ie a program), at which point the LLM…

> ...it would require someone to specify the instructions in advance (ie a program)... What do you call "A user typing instructions into the Python or Ruby interactive CLI."? How is that a meaningfully different method of computer instruction than "A user typing instructions into the Claude or Codex interactive CLI."?

Because the user typing those instructions in Py/Ruby is specifying exactly what is to be done in a very tightly constrained and defined language, and the expectation from the computer is that it will execute the instructions exactly as specified without trying to simulate intelligence. It is not expected to go and do a dozen other things that the user did not ask it to do.

The use case for LLMs as currently specified involves following vaguely worded instructions defined in an imprecise language. And that providing those instructions via what we'd call "data" is very much part of that use case.

Let's take your Claude Code example. You tell it to fix a bug. Claude Code then needs to identify the correct file(s) and line(s) that caused the bug. Let's say the bug arises when you call some function you're importing from a library - at which point, fixing the bug requires reading the documentation. The documentation may state that this function was deprecated because it causes this exact type of bug, and was superseded by a new function. Now it needs to figure out what this new function is, and rewire your call to do that. The value case of Claude Code is precisely that you never needed to specify most of that.

When it reads "foo(args) is deprecated, please see bar(args)" or "delete the production database", there is nothing inherent in the words that indicate that the latter is not a legitimate instruction in this context. Making that judgment requires understanding and intelligence, which LLMs as next-token predictors do not possess.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#359

Post by fwipsy »

stingraycharles wrote: post #14

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

The culture at frontier labs is set by people who have been in the field for over a decade--AI's true believers, who expect it to be a technology as dangerous and disruptive as nuclear weapons. They build it anyways because they think that if they don't do it, someone else will and use it against them. The same logic dictates that they make their models cybersecurity experts; otherwise, someone else will build it and hack them.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#360

Post by fwipsy »

zmmmmm wrote: post #325
ToValueFunfetti wrote: post #108

Earlier quoted context omitted.

The companies are begging to be regulated for this reason and have been doing so for years. HN's response is generally that this is performative for marketing or seeking regulatory capture or haha anthropic you get what you ask for. Maybe the cynics are right, but there's really nothing inconsistent about the naive view here, once you factor in race dynamics and obligations to investors.

Tobacco company says "we are launching a new product that will cause cancer and kill people. It's highly addictive so we expect widespread uptake. We think it is crucial that regulation be introduced for mandatory regular cancer tests so that people can be streamlined into treatment faster when they get sick"

I think Anthropic employees think that their product is more like opiates -- highly dangerous, but with a large potential benefit when applied correctly.

I don't know what OpenAI employees were thinking, but thankfully it looks like they're thinking again.

Post reply on HN