Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Preparedness Challenge

openai.com

Post reply on HN
Print view View original

121–130 of 160 posts

Re: OpenAI Preparedness Challenge

#121

Post by two_in_one »

c7b wrote: post #108
jameshart wrote: post #95

How do we know this challenge page hasn’t been posted by a malevolent AI, which has overcome its creators and obtained access to the internet, and is now looking for ideas for how to maximize the harm it can do?

While it is a joke, I assume, I think it hints at a crucial problem: it's relatively easy to imagine an Auto-GPT-style agent with some sort of CLI access on an internet-connected machine turning into a paperclip machine, no matter how harmless the original task.

It has no long term memory. Everything what happens within one session is forgotten. With limited 'window' it keeps forgetting even within the session. There are no interconnections between sessions. The result: it cannot execute long plans or have permanent 'life'. At least for now. This will be fixed in more complex AI systems, I believe 'soon'. There is strong demand from military here and 'there'. Plus there are many other uses for embodied AI. Like space travel with speed of light.

Re: OpenAI Preparedness Challenge

#122

Post by haltist »

nomel wrote: post #90
haltist wrote: post #63

Earlier quoted context omitted.

Sam Altman is on the record about his belief that OpenAI is going to create a general intelligence system that can solve any well-posed challenge. That belief is based on the current success of LLMs as syntax co-pilots. So if you can formally specify what it means to have a zero-day exploit then presumably OpenAI's general intelligence system will then understand and "solve" it. Many people have compared OpenAI to a…

> Many people have compared OpenAI to a cult and it is easy to see why. Could you help me understand why it's "easy"? Do you have the actual quote? If it was an "eventually" statement, I don't think anything "cult" is required to think AGI will eventually happen. Was the claim that they would be first? It's an eventual goal of many of the wealthiest organizations, with many very smart people working towards it. I thi…

I'll probably write something more elaborate at some point but in the mean time I recommend Melanie Mitchell's book on AI as a good reference for counter-arguments and answers to several of the posted questions. For learning more about the limits of formal systems like LLMs it helps to have basic understanding of basic model theory and formal systems of logic like simple type theory.

Understanding the compactness theorem is a good conceptual checkpoint for whoever decides to follow the above plan. The gist of the argument comes down to compostionality and "emergence" of properties like consciousness/sentience/self-awareness/&etc. There is a lot of money to be made in the AI business and that's already a very problematic ethical dilemma for the people working on this for monetary gain. One might call this a misalignment of values and incentives designed to achieve them, a very pernicious kind of misalignment problem.

Re: OpenAI Preparedness Challenge

#123

Post by startupsfail »

TaylorAlexander wrote: post #109
MeImCounting wrote: post #105

Earlier quoted context omitted.

This gave me a proper chuckle. Truly though I think movies like The Terminator and The Matrix really did a number on the societal consciousness and capacity to think clearly when it comes to anything called "AI".

As a robotics engineer and someone who was interested in robotics since I was 11 years old, and I am SO VERY TIRED of people making the "haha they will kill us all" jokes. It's just the only thing that 99% of people can think about when it comes to robotics and AI! The Terminator came out the year I was born, 39 years ago, and it seems to be all people can think about to this day. When some powerful and wealthy perso…

If you’ve been around for a while, then you should remember that the current state of affairs, where an AI could chat and generate code is relatively recent - Unreasonable Effectiveness paper came out only in 2015. And Deep Learning was there only from 2010. With just a few people who were working on it, instead of using discriminative models.

These same people (I belong to that group), who had a vision back then, to work on a right thing, are now saying clearly that super intelligent machines are nearly there. And likely this is a potential change and a challenge, as we’ve seen many examples when superior intelligence doesn’t care that much about inferior.

As to “become self aware” - it’s super simple to SFT a model that is “self aware”. There is nothing magical in self awareness. There is also not much use in it, so no one is bothering to do it.

Re: OpenAI Preparedness Challenge

#124

Post by jiggawatts »

jerbear4328 wrote: post #36
jiggawatts wrote: post #29

Earlier quoted context omitted.

That is essentially what happened! The FBI asked researchers and university professors precisely this question. They then used the proposed attack vectors to formulate a plan to protect the nation. This was all supposed to be done in secret. After all, we don’t want to “give the terrorists ideas.” The reason I know about this at all is because someone found one such paper was accidentally published on a public FTP si…

...is there a link? I am very curious now.

This was about twenty years ago, and I lost the file to disk corruption.

I do remember some of the proposed attacks.

The most of scary one was if the terrorists have a decent number of people is to drive around and destroy transformers at electrical substations.

Most of those locations are unmanned and have minimal security.

The risk is that there just aren’t that many spare transformers available globally above a certain size and they take months to build.

If you take out enough of them fast enough, you can cripple any modern energy-dependent economy.

This tactic very nearly worked for Russia in Ukraine.

Re: OpenAI Preparedness Challenge

#125

Post by two_in_one »

MeImCounting wrote: post #105
jameshart wrote: post #95

How do we know this challenge page hasn’t been posted by a malevolent AI, which has overcome its creators and obtained access to the internet, and is now looking for ideas for how to maximize the harm it can do?

This gave me a proper chuckle. Truly though I think movies like The Terminator and The Matrix really did a number on the societal consciousness and capacity to think clearly when it comes to anything called "AI".

Just like clowns became evil thanks to Hollywood. Couple of years ago every celebrity, , thought they must warn us about the danger of AI.

Re: OpenAI Preparedness Challenge

#126

Post by uoaei »

ben_w wrote: post #78
ekidd wrote: post #53

Earlier quoted context omitted.

OpenAI is irresponsible in a really curious way according to their own beliefs about AI . If you pay attention to OpenAI's social circles, lots of those people really do believe that we're less than 20-30 years away making humans intellectually obsolete. Specifically, they believe that we may build something much smarter than us, something that's capable of real-world planning. Basically, "We believe our corporate pl…

"[W]e're less than 20-30 years away making humans intellectually obsolete" is neither necessary nor sufficient to get to the conclusion "20% chance of killing literally everybody". A super-virus that blends the common cold with rabies would kill approximately everybody; that doesn't need human-level intellect to happen. Conversely, humans are human-level intellect, and we're mostly sympathetic to each other's plights…

It's still a lack of imagination to assume that AIs will display behaviors that align whatsoever with pathologies we identify in humans. AIs could be completely incomprehensible or even imperceptible yet have strong influence on our lives.

Re: OpenAI Preparedness Challenge

#127

Post by RandomLensman »

startupsfail wrote: post #123
TaylorAlexander wrote: post #109

Earlier quoted context omitted.

As a robotics engineer and someone who was interested in robotics since I was 11 years old, and I am SO VERY TIRED of people making the "haha they will kill us all" jokes. It's just the only thing that 99% of people can think about when it comes to robotics and AI! The Terminator came out the year I was born, 39 years ago, and it seems to be all people can think about to this day. When some powerful and wealthy perso…

If you’ve been around for a while, then you should remember that the current state of affairs, where an AI could chat and generate code is relatively recent - Unreasonable Effectiveness paper came out only in 2015. And Deep Learning was there only from 2010. With just a few people who were working on it, instead of using discriminative models. These same people (I belong to that group), who had a vision back then, to…

It's fair to say "we believe X will happen soon", but if the dangerous super intelligent machines don't happen any time soon, will those same people compensate societies for wasting political and economic resources on worrying about it? The view of a rather imminent danger has real consequences even if it turns out incorrect.

Re: OpenAI Preparedness Challenge

#128

Post by uoaei »

londons_explore wrote: post #76

What exactly would I use $25k of openAI credits for? My personal use of it rarely comes to more than $10 per month, despite using it multiple times per day. (most recently: "Write me a bash command to send data at a given rate to an arbitrary IP address.")

Pretty sure this is for researchers to develop and implement a plan that uses their APIs to test hypotheses and run studies without having to pay out of their own pocket...

Re: OpenAI Preparedness Challenge

#129

Post by IshKebab »

TheCaptain4815 wrote: post #7

While I applaud how much OpenAi fears these negatives, given the current state of Ai trajectory, it won't be long until a future open source model gets "uncensored" and is easily usable for tons and tons of malicious intent. There already exists a fantastic "uncensored" model with the newly released Dolphin Mistral 7b. I saw some results from others where the model could easily give explosives recipes from existing p…

Come on. Racist poems and hallucinated bomb recipes aren't the risk.

The big risks are that AI can automate harmful things that are possible today but require a human.

For example

* Surveillance of social media (e.g. like the Department of Education in the UK recently did)

* Social engineering fraud. Especially via phone calls. Imagine if scammers could literally call all the grannies and talk to them using their children's voices, automatically.

Re: OpenAI Preparedness Challenge

#130

Post by lucubratory »

two_in_one wrote: post #121
c7b wrote: post #108

Earlier quoted context omitted.

While it is a joke, I assume, I think it hints at a crucial problem: it's relatively easy to imagine an Auto-GPT-style agent with some sort of CLI access on an internet-connected machine turning into a paperclip machine, no matter how harmless the original task.

It has no long term memory. Everything what happens within one session is forgotten. With limited 'window' it keeps forgetting even within the session. There are no interconnections between sessions. The result: it cannot execute long plans or have permanent 'life'. At least for now. This will be fixed in more complex AI systems, I believe 'soon'. There is strong demand from military here and 'there'. Plus there are…

Personally I think multi-year scale memory is possible with currently available research, if we just put it all together. What happens if we combine very long context lengths, dedicated summarising LLMs, RAG, MemGPT, sparse MoE, and a perennial Constitutional AI overseer? I don't see any reason why these systems couldn't work together. It just hasn't really been that much time, particularly considering that large training runs can take months, and if you want the best performance you often need to train with your specific architecture in mind. As well as time, it's not like all of these advances are coming from the same group of people, they're from AI researchers spread across the globe. Get them all in a room with a CERN-level budget for compute resources and I think they could do it.
Post reply on HN