Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI

scottaaronson.blog

Post reply on HN
Print view View original

111–120 of 158 posts

Re: OpenAI

#111

Post by tome »

deadpannini wrote: post #90
astrange wrote: post #40

Earlier quoted context omitted.

> It is clearly imaginable that a very intelligent agent could end humanity if its objective would require so. This is quite possible. Indeed, I don't believe this is exclusive to superintelligence or requires it at all. Compare to the closest thing we have to "inventing AGI" - having babies. People do that all the time and there isn't a mathematical guarantee that baby won't end humanity, but we don't do much to sto…

"Mainly, why would it want to [end humanity]?" The anthropomorphism is misleading. No one expects that an AGI would "want to" in the commonplace sense of being motivated by animosity, fear, or desire. The problem is that the best path to satisying its reward function could have adverse-to-extinction level consequences for humanity, because alignment is hard, or maybe impossible.

But now you have appealed to anthropomorphism (“intelligence”) to pose a problem yet forbidden anthropomorphism in an attempted counter argument. That doesn’t seem quite fair.

Re: OpenAI

#112

Post by tome »

marvin wrote: post #91
csmpltn wrote: post #55

Earlier quoted context omitted.

> "Most of these AGI doom-scenarios require no self-awareness at all. AGI is just an insanely powerful tool that we currently wouldn't know how to direct, control or stop if we actually had access to it." You're talking about "doomsday scenarios". Can you actually provide a few concrete examples?

Over the course of years, we figure out how to create AI systems that are more and more useful, to the point where they can be run autonomously and with very little supervision produce economic output that eclipses that of the most capable humans in the world. With generality, this obviously includes the ability to maintain and engineer similar systems, so human supervision of the systems themselves can become redund…

> they can be run autonomously and with very little supervision produce economic output that eclipses that of the most capable humans in the world.

What’s the evidence for this?

> Perverse instantiation of AI systems was accidentally demonstrated in the lab decades ago

What are you referring to?

Re: OpenAI

#113

Post by gitfan86 »

theptip wrote: post #105
gitfan86 wrote: post #97

The debate around "What is AGI?" is becoming increasingly irrelevant. If in two iterations of DallE it can do 30% of graphic design work just as well as a human, who cares if it really "understands" art. It is going to start making an impact on the world. Same thing with self driving. If the car doesn't "understand" a complex human interaction, but still achieves 10x safety at 5% of the cost of a human, it is going t…

It’s worth being clear about what AI risk is. This has nothing to do with “AI may do some harm by putting lots of people out of work”. The idea is that there is _existential risk_ (ie species-extinction) once an AI can self-modify to improve itself, therefore increasing its own power. A powerful AI can change the world however it wants, and if this AI is not aligned to human interests it can easily decide to make hum…

I agree that putting lots of people out of work isn't the problem. The problem is that these Non-General models become very powerful and they can be programmed by humans to do very impactful things. So much so that even if AGI comes into existence just a few years later it will be of minimal impact to the world.

Re: OpenAI

#114

Post by nopinsight »

ma2rten wrote: post #109
theptip wrote: post #102

Earlier quoted context omitted.

It’s worrying to see very smart guys like LeCun failing to grok the paper clip maximizer issue (or coffee maximizer as Russell phrases it), which is like the one paragraph summary or elevator pitch for AI risk. I think there are plenty of other valid objections to a high E-risk estimate but that one is non-sensical to me. I think Robin Hanson has the most cogent objection to high E-risk estimates, which is basically…

Have you considered that it's not LeCun who is missing something? The AI safety community seems to be unfortunately almost completely separate from the actual AI research community and be making some strong assumptions about how AGI is going to work. Note that LeCun had a reply in the thread and there was a lot more discussion which GP didn't quote.

Russell did have good replies to Lecun’s replies.

From my reading of the full article, Bengio who was/is also well-versed in the latest deep learning research was leaning more toward the Russell argument as well.

Re: OpenAI

#115

Post by nopinsight »

mrow84 wrote: post #110
nopinsight wrote: post #100

Earlier quoted context omitted.

Most people would probably agree the latest models generalize better than flatworms. Mouse-level intelligence is more challenging and the comparison is unclear. Flatworms first appeared 800+ million years ago, while mouse lineage diverged from humans only 70-80 million years ago. If our AGI development timeline roughly follows the proportion it took natural evolution, it might be much too late to begin seriously thin…

> Most people would probably agree the latest models generalize better than flatworms. > Flatworms first appeared 800+ million years ago Surviving for 800 million years seems to me like a pretty good indicator of meaningful generalisation.

Water, rocks, and other minerals have been around much longer than that.

Our concern is not the survivability or adaptability over evolutionary timescale but the capabilities to affect the world in human timescale.

Re: OpenAI

#116

Post by preommr »

gitfan86 wrote: post #97

The debate around "What is AGI?" is becoming increasingly irrelevant. If in two iterations of DallE it can do 30% of graphic design work just as well as a human, who cares if it really "understands" art. It is going to start making an impact on the world. Same thing with self driving. If the car doesn't "understand" a complex human interaction, but still achieves 10x safety at 5% of the cost of a human, it is going t…

> "What is AGI?" is becoming increasingly irrelevant.

It's always been irrelevant in the practical sense. It's just an interesting conversation piece particularly among the general public where they're not going to discuss specific solutions like algorithms or techniques.

Re: OpenAI

#117

Post by deadpannini »

tome wrote: post #111
deadpannini wrote: post #90

Earlier quoted context omitted.

"Mainly, why would it want to [end humanity]?" The anthropomorphism is misleading. No one expects that an AGI would "want to" in the commonplace sense of being motivated by animosity, fear, or desire. The problem is that the best path to satisying its reward function could have adverse-to-extinction level consequences for humanity, because alignment is hard, or maybe impossible.

But now you have appealed to anthropomorphism (“intelligence”) to pose a problem yet forbidden anthropomorphism in an attempted counter argument. That doesn’t seem quite fair.

I don't intend to forbid anything - I just think the language of motivation and desire makes it harder to see the risks, because it introduces irrelevant questions into the the conversation like "how can machines want something?"

Conversely, at least in this discussion, the term "intelligence" seems pretty neutral.

Re: OpenAI

#118

Post by nopinsight »

tome wrote: post #112
marvin wrote: post #91

Earlier quoted context omitted.

Over the course of years, we figure out how to create AI systems that are more and more useful, to the point where they can be run autonomously and with very little supervision produce economic output that eclipses that of the most capable humans in the world. With generality, this obviously includes the ability to maintain and engineer similar systems, so human supervision of the systems themselves can become redund…

> they can be run autonomously and with very little supervision produce economic output that eclipses that of the most capable humans in the world. What’s the evidence for this? > Perverse instantiation of AI systems was accidentally demonstrated in the lab decades ago What are you referring to?

By the time you find such evidence, it could already be close to game over for humanity. It’s important to get this right before that.

We already have significant warnings. See for yourself if latest models like Imagen, Gato, Chinchilla have economic values and can potentially cause harm.

Re: OpenAI

#119

Post by cillian64 »

pas wrote: post #70
nlqrt wrote: post #57

"[...] where the misuse of AI for spambots, surveillance, propaganda, and other nefarious purposes is already a major societal concern [...]" I'm curious what he will do and whether for example he approves of the code laundering CoPilot tool. I also hope he'll resist being used as an academic promoter of such tools, explicitly or implicitly (there are many ways, his mere association with the company buys goodwill alr…

what's wrong with copilot as a concept and as the concrete implementation? it's a fancy autocomple. we had stack overflow based autocomplete before. this got a bigger training data set.

The objection here is that its training was based on code on github without paying any attention to the license of that code. It’s generally considered ok for people to learn from code and then produce new code without the new code being considered a derived work of what they learned from (I’m not sure if there is a specific fair use clause covering this). But it’s not obvious that copilot should be able to ignore the licenses of the code it was trained on, especially given it sometimes outputs code from the training set verbatim. One could imagine a system very similar to copilot which reads in GPL or proprietary code and writes functionally equivalent code while claiming it’s not a derived work of the original and so isn’t subject to its licensing constraints.

Re: OpenAI

#120

Post by lgas »

__john wrote: post #6
jks wrote: post #4

Earlier quoted context omitted.

In academia, a "sabbatical" means you take some time off from teaching courses, advising students and doing administrative work so you can concentrate on your research. So in order to stay on sabbatical, he'd need to get the AI to do that other stuff.

Not so impractical, don't most undergrads write at a GPT3 level anyways? =)

No, but with a little help I'm sure they could learn to.
Post reply on HN