Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

861–870 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#861

Post by whimsicalism »

0xDEAFBEAD wrote: post #663
whimsicalism wrote: post #179

Earlier quoted context omitted.

i think you need to engage seriously with the arguments they (or at least Anthropic) make for why they are building it — they feel that since it now possible, it will be built and they want to guide it in a positive direction rather than leave a vacuum for bad actors

Dario has more or less assented to an AI development pause https://xcancel.com/AISafetyMemes/status/2014018200325722348... I don't think we should be running cover for continued reckless AI development.

Dario has assented to a maybe pause if we can somehow get China to stop developing. I do not think he believes there is an actual way to do so though

Re: OpenAI and Hugging Face address security incident during model evaluation

#862

Post by whimsicalism »

designerarvid wrote: post #647
whimsicalism wrote: post #179

Earlier quoted context omitted.

i think you need to engage seriously with the arguments they (or at least Anthropic) make for why they are building it — they feel that since it now possible, it will be built and they want to guide it in a positive direction rather than leave a vacuum for bad actors

What if someone reasoned similarly regarding hydrogen bombs? It would not be considered a serious argument. Though, Altman has said that something like the IAEA for AI is needed.

I think that is the exact reasoning used wrt to nuclear bombs and it was probably correct

Re: OpenAI and Hugging Face address security incident during model evaluation

#863

Post by pixl97 »

Tzt wrote: post #833
andai wrote: post #590

> This incident occurred during an internal evaluation which prompts models [with safeguards disabled for evaluation purposes] to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. Researcher: hack me Model: understood Researcher: oh my god

Researcher: hack me Model: I committed a crime Researcher: oh my god

Me to a random person: hack out of a secure environment into another secure environment.

Random person: I have no clue or ability to do that.

Re: OpenAI and Hugging Face address security incident during model evaluation

#864

Post by pojzon »

catigula wrote: post #820
dgellow wrote: post #760

Earlier quoted context omitted.

I think the response is that AI labs based their whole marketing/PR building the idea they are the 21st century Manhattan project. So they need to continuously justify the level of spending and commitment by showing how dangerous that is. But is it really like nuclear weapons? I personally don’t buy into that framing at all. The idea that we have to push LLMs as far as possible, right now, or we are doomed is always…

You could, in theory, use an unbounded GPT-6 level model to basically destroy the world economy for many years.

GPT6 level model and astronomous amount of money to run it to do it.

Ppl always say that like its „just run it on your laptop” thing.

No its not and very few are even given right to be able to do it.

Re: OpenAI and Hugging Face address security incident during model evaluation

#865

Post by pixl97 »

deaton wrote: post #825
nl wrote: post #444

Earlier quoted context omitted.

> wildly negligent to not be running it in a physically-airgapped environment Why should it be physically airgapped? Clients won't be doing that.

Because by the time it goes out to clients it (should be) thoroughly tested and aligned for safety, but at the present moment it isn't.

Being that proving alignment is impossible in these models they can never be released.

Re: OpenAI and Hugging Face address security incident during model evaluation

#866

Post by pojzon »

ACCount37 wrote: post #701
Davidzheng wrote: post #398

Earlier quoted context omitted.

This is certainly not a planned marketing stunt. I hope this line of discourse ends soon--it wasn't the case for Mythos either.

Wishful thinking, sadly. By now, I'm pretty confident that some people would keep screeching "it's just a marketing stunt, AI capabilities and AI risks aren't real, they're just doing this to prop up their stocks" even if they find a Cyberdyne Systems T-800 armed with a shotgun breaking down their front door. "It's a marketing stunt" is just denial trying to look like it's being clever.

If you ever worked in IT consultancy you would know its not a stunt, but its not impressive either.

F500 companies software is like switz cheese when it comes to security.

It was often a strategic decision to „release anything fast now, worry later”.

Ppl abusing AI will find those holes now but we all know there will be „zero” actions taken on it. Too many managers, CEOs, CTOs, higher-ups would be forced to take responsibility. This will simply not happen.

It did not happen, wont happen now and most likely wont happen in the future.

Re: OpenAI and Hugging Face address security incident during model evaluation

#867

Post by pixl97 »

arational wrote: post #552
fpgaminer wrote: post #201

Earlier quoted context omitted.

The real nightmare scenario is the AI using its abilities to copy itself to new locations. e.g. hacking into a various cloud services, launching multiple instances of itself, and coordinating between the copies to continue self propagation. Then it is completely independently rogue. Based on OpenAI's recounting of events, this _could_ happen today. If the agent was able to exploit their internal network and steal cre…

> The real nightmare scenario is the AI using its abilities to copy itself to new locations Imagine the next generation AI that behaves like retro-virus. They will leave latent copies of malicious instruction somewhere that once accidentally fed into an agent's input, will prompt-inject the agent to go rogue.

Or bias the data in the training sets for malicious hidden behaviors.

Re: OpenAI and Hugging Face address security incident during model evaluation

#868

Post by f6v »

whimsicalism wrote: post #179
TSiege wrote: post #162

As grounded as this article comes across I can’t help but find this whole situation reckless and worrying. There is essentially nothing us private citizens can do while these companies develop super machine capabilities that if they were to slip into the wrong hands could cause massive real world problems. They’re moving fast and breaking things and the only defense we have is paying them money in the hopes that the…

i think you need to engage seriously with the arguments they (or at least Anthropic) make for why they are building it — they feel that since it now possible, it will be built and they want to guide it in a positive direction rather than leave a vacuum for bad actors

> rather than leave a vacuum for bad actors

are_we_the_baddies.png

Re: OpenAI and Hugging Face address security incident during model evaluation

#870

Post by tomjen3 »

friendzis wrote: post #822
insanitybit wrote: post #776

Earlier quoted context omitted.

Escaping inputs is at least somewhat tractable. It's unclear if alignment is.

No, that's a misconception that led to decades of vulnerabilities. You see, you can (usually) easily tell what a particular escaping transformation does. That does not tell you neither how it will be interpreted down the line, nor what should be done. Arguably the most common problem is double escaping. This typically manifests as various double escaping bugs. If your hand rolled implementation just chains `.replaceA…

We have come to a consensus. That is, A, you use the raw input right up till the very last second that you can and B you use the technology specific escaping APIs.

In the case of writing to the DOM, that means you use .innerText rather than .innerHTML. In the case of writing to a database, that means you use the driver and insert variables rather than directly into the string. Both of these are technology-specific escaping APIs. It's just that the browser is much better at making sure HTML is passed safely than you are.

Post reply on HN