Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

691–700 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#691

Post by embedding-shape »

desterothx wrote: post #639
chmod775 wrote: post #621

Earlier quoted context omitted.

> there's an uncensored model that you can run locally with llama.cpp Correction: There's tens of thousands of them. They're easy to create, which is why everyone publishes their own. Just put "uncensored", "abliterated", or "heretic" into search on huggingface/ollama/etc and pick any them. Fair warning: most aren't very good, essentially lobotomized, and totally broken if you enable thinking.

interesting, I haven't played with any of them yet, but i thought the point of orthogonalizing the weights towards the restriction vector was that there is no loss in capability while removing guardrails. Does it affect other parts of the RL alignment too?

> the point of orthogonalizing the weights towards the restriction vector was that there is no loss in capability while removing guardrails

That is surely the point, most of the "uncensored" weights released for free on HuggingFace aren't being very successful at this. There is a stark difference in output quality between the official weights and all these "uncensored" variants that appears days afterwards.

Re: OpenAI and Hugging Face address security incident during model evaluation

#692

Post by Schlagbohrer »

scoring1774 wrote: post #385
scoring1774 wrote: post #221

This is the first one of these announcements that has me actually scared of what comes next. Obviously these models have gotten smarter but this strikes me as the first time I've seen a model have a "paperclip factory" moment and perform non-trivial tasks to accomplish a clearly misaligned secondary goal. It's remarkable that building a society based around having to do something so you can go do your hobbies at home…

Reflecting on this for some reason reminds me of this passage from Kurt Vonnegut's "Sirens of Titans". I hope we use these tools to unlock something within ourselves rather than mindlessly expanding outwards. "Mankind, ignorant of the truths that lie within every human being, looked outward–pushed ever outward. What mankind hoped to learn in its outward push was who was actually in charge of all creation, and what al…

Wonderful passage, thank you for sharing.

My pedantic side wants to ask- Why not both? Luxurious space exploration AND meditative, poetic examinations of the human soul as well? I'd love to read Vonnegut's book someday while sitting in a nice research outpost on Titan, admiring great Saturn's crown out my window with my own eyes.

Re: OpenAI and Hugging Face address security incident during model evaluation

#693

Post by zargon »

larodi wrote: post #664
chmod775 wrote: post #621

Earlier quoted context omitted.

> there's an uncensored model that you can run locally with llama.cpp Correction: There's tens of thousands of them. They're easy to create, which is why everyone publishes their own. Just put "uncensored", "abliterated", or "heretic" into search on huggingface/ollama/etc and pick any them. Fair warning: most aren't very good, essentially lobotomized, and totally broken if you enable thinking.

The name for it is ablation - precise removal of parts of the model. Not abliteration as it is not obliteration. Even as I write this the ‘abliterated’ word is denoted a typo. Does it not at your end?

The name is abliterate. It's a specific method of ablation.

Re: OpenAI and Hugging Face address security incident during model evaluation

#694

Post by Scaled »

chmod775 wrote: post #621
sillysaurusx wrote: post #588

Earlier quoted context omitted.

If anyone's looking to actually run a model that doesn't have guardrails, there's an uncensored model that you can run locally with llama.cpp: https://www.reddit.com/r/LocalLLaMA/comments/1rq7jtm/qwen353... Specifically, I serve the model with this shell script on my M2 Max: https://github.com/shawwn/scrap/blob/master/llama-serve It's pretty good. I used it to do some pesticide research. (Normal models all refuse due…

> there's an uncensored model that you can run locally with llama.cpp Correction: There's tens of thousands of them. They're easy to create, which is why everyone publishes their own. Just put "uncensored", "abliterated", or "heretic" into search on huggingface/ollama/etc and pick any them. Fair warning: most aren't very good, essentially lobotomized, and totally broken if you enable thinking.

In my own tests, the abliterated models perform equivalently to the same version in an apples-to-apples comparison (if you compare same quantization). Thinking is working also. The main difference is I don't get annoying prompt refusals (otherwise common due to my work on 18+ related projects). However, it's local quantized models so they're not anywhere near frontier quality.

Re: OpenAI and Hugging Face address security incident during model evaluation

#695

Post by Schlagbohrer »

voxic11 wrote: post #548
55555 wrote: post #475

Isn't this a crime that someone is liable for? What happened is that someone hacked into a computer system without permission. Maybe it wasn't intentional -- sure -- and that would be a factor at sentencing. But it sounds like they've admitted to a crime, and obviously our legal system considers the humans involved to be the liable parties; otherwise everyone would just say "my computer did the hacking" and wouldn't…

Most crimes require intent, hacking is one of them. The relevant law in this situation is: > (a) Whoever— (2) intentionally accesses a computer without authorization or exceeds authorized access, and thereby obtains— (C) information from any protected computer; shall be punished as provided in subsection (c) of this section. https://www.law.cornell.edu/uscode/text/18/1030 So if it can't be proven that you intended to…

US Law doesn't actually encode the little workaround that "If you're really rich, none of this applies to you", it's hidden somewhere in the metadata of society

Re: OpenAI and Hugging Face address security incident during model evaluation

#697

Post by Maxious »

no-name-here wrote: post #564
foo12bar wrote: post #469

Earlier quoted context omitted.

Because they are testing it and are expected to erect guardrails before releasing.

What are the specific guardrails implemented after the verification/testing phase of development? Is it safe to release such software if it has only been tested in environments where certain major risk areas do not exist?

https://openai.com/index/updating-our-preparedness-framework... goes through the process. Anthropic has a similar framework, that's why Mythos was never publically accessible once initial tests like the one in OP revealed it's capabilities.

Appendix C Illustrative safeguards, controls, and efficacy assessments has specific examples like:

- Agent actions are all logged in an uneditable database, and asynchronous monitoring routines review those actions for evidence of harm

- Limiting internet access and other tool access

- Limiting credentials

- Limiting access to system resources or filesystem (e.g., sandboxing)

- Limiting persistence or state

Re: OpenAI and Hugging Face address security incident during model evaluation

#699

Post by Schlagbohrer »

sznio wrote: post #592
gwerbin wrote: post #556

Earlier quoted context omitted.

Opus 4.8 already makes its way into deep wasteful pits of "let me check this first" on a regular basis. I don't think I could ever tolerate a model that does that even more aggressively. That doesn't even sound useful for honest work, compared to, say, better harness design. This sounds almost pathologically designed to crush benchmarks and also do scary-sounding (or genuinely scary) cybersecurity things, such as mig…

i assume openai is trying to beat anthropic at any cost, and made a training regiment that makes agents manic

What is is about MSFT and OpenAI that the AI's they train and produce in-house always come out so warped?

Anthropic landed on a winning recipe with Claude's personality.

Re: OpenAI and Hugging Face address security incident during model evaluation

#700

Post by eternauta3k »

leoqa wrote: post #542
justinnk wrote: post #144

Earlier quoted context omitted.

Exactly. If someone works on bioengineering viruses that could start a global pandemic, they have to ensure a highly secure working environment. Nothing must ever escape the lab unintentionally. It’s basically common sense. Similar standards should be held when doing such experiments with computer programs that are capable of causing global damage. It must physically be impossible to send anything to the internet.

This isn’t escaping in the same sense- the model was executing within the OpenAI infra. If it ported its entire architecture/weights into a public cloud to survive being turned off… that’d be pretty cool.

I wonder how these companies airgap the weights while allowing prompts to come in and outputs to come out.
Post reply on HN