Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI

scottaaronson.blog

Post reply on HN
Print view View original

101–110 of 158 posts

Re: OpenAI

#101

Post by newbye4 »

nopinsight wrote: post #93
chis wrote: post #7

Seeing someone as trustworthy as Scott choose to work on AI safety is a pretty good sign for the state of the field IMO. It seems like a lot of studious people agree AI alignment is important but then end up shoehorning the problem into whatever framework they are most expert in. When all you have is a hammer etc... I feel like he has good enough taste to avoid this pitfall. Semi-related - I'd want to see some actual…

I think this debate on AGI safety between major AI researchers is quite relevant to those who are non-expert in the area. Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More https://www.lesswrong.com/posts/WxW6Gc6f2z3mzmqKs/debate-on-... Note that it was in 2019 when we didn’t yet see the capabilities of current models like Chinchilla, Gato, Imagen and DALL-E-2. Sample: “Yann LeCun: "do…

Interesting, also anyone could modify the GAI so to disable the safety measures, just ask the GAI how could a bad actor change the code to allow you become evil?

Re: OpenAI

#102

Post by theptip »

nopinsight wrote: post #93
chis wrote: post #7

Seeing someone as trustworthy as Scott choose to work on AI safety is a pretty good sign for the state of the field IMO. It seems like a lot of studious people agree AI alignment is important but then end up shoehorning the problem into whatever framework they are most expert in. When all you have is a hammer etc... I feel like he has good enough taste to avoid this pitfall. Semi-related - I'd want to see some actual…

I think this debate on AGI safety between major AI researchers is quite relevant to those who are non-expert in the area. Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More https://www.lesswrong.com/posts/WxW6Gc6f2z3mzmqKs/debate-on-... Note that it was in 2019 when we didn’t yet see the capabilities of current models like Chinchilla, Gato, Imagen and DALL-E-2. Sample: “Yann LeCun: "do…

It’s worrying to see very smart guys like LeCun failing to grok the paper clip maximizer issue (or coffee maximizer as Russell phrases it), which is like the one paragraph summary or elevator pitch for AI risk. I think there are plenty of other valid objections to a high E-risk estimate but that one is non-sensical to me.

I think Robin Hanson has the most cogent objection to high E-risk estimates, which is basically that the chances of a runaway AI are low because if N is the first power level that can self-modify to improve, nation-states (and large corporations) will all have powerful AIs at power level N-1, and so you’d have to “foom” really hard from N to N+10 before anyone else increased power in order to be able to overpower the other non-AGI AIs. So it’s not that we get one crack at getting alignment right; as long as most of the nation-state AIs end up aligned, they should be able to check the unaligned ones.

I can see this resulting in a lot of conflict though, even if it’s not Eleizer’s “kill all humans in a second” scale extinction event. I think it’s quite plausible we’ll see a Butlerian Jihad, less plausible we’ll see an unexpected extinction event from a runaway AGI. Still think it’s worth studying but I’m not convinced we are dramatically underfunding it at this stage.

Re: OpenAI

#103

Post by abeppu »

gitfan86 wrote: post #97

The debate around "What is AGI?" is becoming increasingly irrelevant. If in two iterations of DallE it can do 30% of graphic design work just as well as a human, who cares if it really "understands" art. It is going to start making an impact on the world. Same thing with self driving. If the car doesn't "understand" a complex human interaction, but still achieves 10x safety at 5% of the cost of a human, it is going t…

Why does the question of defining AGI even need to enter into this?

Aaronson's post only sort of obliquely touches on AGI, via OpenAI's stated founding mission, and Yudkowsky's very dramatic views. Most of the post is on there being signs that the field is ready for real progress. AI safety can be an interesting, important, fruitful area without AI approaching AGI, or even surpassing human performance on some tasks. We would still like to be able to establish confidently that a pretty dumb delivery drone won't decide to mow down pedestrians to shorten its delivery time, right?

Re: OpenAI

#104

Post by nopinsight »

hansword wrote: post #58

'OpenAI, of course, has the word “open” right in its name, ... don’t expect me to share any proprietary information' Yeah, Mr Aaronson just lost quite a bit of respect from my side. Going into AI is a great move, moving to the ClosedAI corporation.......? Why? (Edit: Removed an outdated reference to Elon Musk, thanks @pilaf !)

The latest huge AI models that may germinate into an AGI all come from private corporations, largely because of the requisite resources and currently there’re very few if any public or nonprofit AI-focused organizations with such resources.

Re: OpenAI

#105

Post by theptip »

gitfan86 wrote: post #97

The debate around "What is AGI?" is becoming increasingly irrelevant. If in two iterations of DallE it can do 30% of graphic design work just as well as a human, who cares if it really "understands" art. It is going to start making an impact on the world. Same thing with self driving. If the car doesn't "understand" a complex human interaction, but still achieves 10x safety at 5% of the cost of a human, it is going t…

It’s worth being clear about what AI risk is. This has nothing to do with “AI may do some harm by putting lots of people out of work”.

The idea is that there is _existential risk_ (ie species-extinction) once an AI can self-modify to improve itself, therefore increasing its own power. A powerful AI can change the world however it wants, and if this AI is not aligned to human interests it can easily decide to make humans extinct.

Scott said in the OP that he now sees AGI as potentially close enough that one can do meaningful research into alignment, ie it’s plausible that this powerful AI could arrive in our lifetimes.

So he is claiming the opposite of you; AGI is more relevant than ever, hence the career change.

I agree with your premise that non-General AI will continue to improve and add lots of value, but I don’t think your conclusion follows from that premise.

Re: OpenAI

#106

Post by nopinsight »

theptip wrote: post #102
nopinsight wrote: post #93

Earlier quoted context omitted.

I think this debate on AGI safety between major AI researchers is quite relevant to those who are non-expert in the area. Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More https://www.lesswrong.com/posts/WxW6Gc6f2z3mzmqKs/debate-on-... Note that it was in 2019 when we didn’t yet see the capabilities of current models like Chinchilla, Gato, Imagen and DALL-E-2. Sample: “Yann LeCun: "do…

It’s worrying to see very smart guys like LeCun failing to grok the paper clip maximizer issue (or coffee maximizer as Russell phrases it), which is like the one paragraph summary or elevator pitch for AI risk. I think there are plenty of other valid objections to a high E-risk estimate but that one is non-sensical to me. I think Robin Hanson has the most cogent objection to high E-risk estimates, which is basically…

My issue with the Hanson objection as stated above (link to the original would be appreciated) is that it rests on the assumption that the N-1 level AIs still under human control can somehow completely eliminate or suppress the self-modifying AGI long enough until alignment research is complete. Meanwhile, the unaligned AGI could multiply, hide, and accumulate power covertly.

Humanity would also need time to align AGI before any AI reaches the N+10 power level. The existence of all those N-1 level AIs in multiple organizations only means there are more chances of an AGI reaching the critical power level.

Re: OpenAI

#107

Post by d--b »

csee wrote: post #99
d--b wrote: post #23

AI is not going to become self aware and destroy the world. AI is going to cause something like the industrial revolution of the 19th century: massive changes in who is rich, massive changes in the labor market, massive changes in how people make war, etc. It’s already started really. What worries me most is that as long as society is capitalist, AI will be used to optimize for self-enrichment, likely causing an even…

"AI is not going to ... destroy the world." Bare assertion fallacy? This question is hotly debated and I don't believe it can be so easily dismissed like that. It is not obvious that aligning something much smarter than us will be a piece of cake.

Should I really add “in my opinion” to all the sentences I write? We are a smart bunch here. We can figure out when statements lack nuance in order to provoke some reaction.

We’re talking about the future here and a fairly complex one at that. So obviously I don’t know more than the next guy.

Re: OpenAI

#108

Post by d--b »

sgillen wrote: post #48
davesque wrote: post #41

Earlier quoted context omitted.

Agree with basically all of your points. I have huge concerns that the humanity of the future will basically split into two different species: the technocracy and the underlings. Sounds like science fiction but it honestly feels like we're headed in that direction. Even today, the privilege afforded by a life in technology and among technologists seems to set a person apart from the rest of the world to such an exten…

>> Even today, the privilege afforded by a life in technology and among technologists seems to set a person apart from the rest of the world to such an extent that they almost forget it exists. I agree on your second point, but those in medicine, finance, or law enjoy similar salaries and quality of life to those in tech. Furthermore to really set yourself apart and join the global super rich you can’t really do that…

Medicine, finance and law are three disciplines that are being heavily threatened by AI.

Re: OpenAI

#109

Post by ma2rten »

theptip wrote: post #102
nopinsight wrote: post #93

Earlier quoted context omitted.

I think this debate on AGI safety between major AI researchers is quite relevant to those who are non-expert in the area. Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More https://www.lesswrong.com/posts/WxW6Gc6f2z3mzmqKs/debate-on-... Note that it was in 2019 when we didn’t yet see the capabilities of current models like Chinchilla, Gato, Imagen and DALL-E-2. Sample: “Yann LeCun: "do…

It’s worrying to see very smart guys like LeCun failing to grok the paper clip maximizer issue (or coffee maximizer as Russell phrases it), which is like the one paragraph summary or elevator pitch for AI risk. I think there are plenty of other valid objections to a high E-risk estimate but that one is non-sensical to me. I think Robin Hanson has the most cogent objection to high E-risk estimates, which is basically…

Have you considered that it's not LeCun who is missing something? The AI safety community seems to be unfortunately almost completely separate from the actual AI research community and be making some strong assumptions about how AGI is going to work.

Note that LeCun had a reply in the thread and there was a lot more discussion which GP didn't quote.

Re: OpenAI

#110

Post by mrow84 »

nopinsight wrote: post #100
nradov wrote: post #88

Earlier quoted context omitted.

If they could produce an AGI as smart as, let's say a mouse, that would be good evidence that they're on the right track. So far nothing is even close to that level. Depending on how you measure, they're not even really at the flatworm level yet. All the AI technology produced so far has been domain specific and doesn't represent meaningful progress towards true generalized intelligence.

Most people would probably agree the latest models generalize better than flatworms. Mouse-level intelligence is more challenging and the comparison is unclear. Flatworms first appeared 800+ million years ago, while mouse lineage diverged from humans only 70-80 million years ago. If our AGI development timeline roughly follows the proportion it took natural evolution, it might be much too late to begin seriously thin…

> Most people would probably agree the latest models generalize better than flatworms.

> Flatworms first appeared 800+ million years ago

Surviving for 800 million years seems to me like a pretty good indicator of meaningful generalisation.

Post reply on HN