Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI

scottaaronson.blog

Post reply on HN
Print view View original

141–150 of 158 posts

Re: OpenAI

#141

Post by astrange »

jononor wrote: post #136
astrange wrote: post #133

Earlier quoted context omitted.

> If, in that MDP, there is another "human" who has some probability, however small, of switching the agent off, and if the agent has available a button that switches off that human, the agent will necessarily press that button as part of the optimal solution for fetching the coffee. This is anthropomorphization - "turning off" = "death" is a concept limited to biological creatures, and isn't necessarily true for oth…

The human "turning off (the agent)" could be substituted with "removing a necessary resource to complete the specified task". Say the electricity, either of the agent, or even just the coffee machine.

Sounds like an OSHA violation, but not a new or different one. You can already get run over by a forklift if you're standing in front of it. There's various things we do about that, but they're boring real-life things, not fun logic-puzzle things, so they're just not mentioned in the problem. There isn't a way to categorically prevent machines from accidentally killing people though.

Re: OpenAI

#142

Post by sgillen »

d--b wrote: post #108
sgillen wrote: post #48

Earlier quoted context omitted.

>> Even today, the privilege afforded by a life in technology and among technologists seems to set a person apart from the rest of the world to such an extent that they almost forget it exists. I agree on your second point, but those in medicine, finance, or law enjoy similar salaries and quality of life to those in tech. Furthermore to really set yourself apart and join the global super rich you can’t really do that…

Medicine, finance and law are three disciplines that are being heavily threatened by AI.

You can add many tech disciplines including large swathes of programming to the list of things being threatened by AI.

Re: OpenAI

#143

Post by nopinsight »

astrange wrote: post #135
newbye4 wrote: post #101

Earlier quoted context omitted.

Interesting, also anyone could modify the GAI so to disable the safety measures, just ask the GAI how could a bad actor change the code to allow you become evil?

How did you get a "limited" "AGI" in the first place? If you had a human that was "limited" to be unable to even imagine doing evil (fsvo evil), that would seem to make them less than generally intelligent and there'd be quite a lot of things it wouldn't be able to learn or do. This field is fairly silly because it just involves people making up a lot of incoherent concepts and then asserting they're both possible (b…

Most, perhaps all, AI alignment researchers do not suggest that we limit the AGI’s capabilities. Rather, it becomes clear that we need to engineer a very capable AGI which aligns with us and use it to help control the emergence of unaligned AGIs, because nothing else likely suffices.

Your public mischaracterization of the whole field composed of many very smart people only shows your ignorance.

Note that Yann LeCun didn’t do that in the debate.

Re: OpenAI

#144

Post by astrange »

nopinsight wrote: post #143
astrange wrote: post #135

Earlier quoted context omitted.

How did you get a "limited" "AGI" in the first place? If you had a human that was "limited" to be unable to even imagine doing evil (fsvo evil), that would seem to make them less than generally intelligent and there'd be quite a lot of things it wouldn't be able to learn or do. This field is fairly silly because it just involves people making up a lot of incoherent concepts and then asserting they're both possible (b…

Most, perhaps all, AI alignment researchers do not suggest that we limit the AGI’s capabilities. Rather, it becomes clear that we need to engineer a very capable AGI which aligns with us and use it to help control the emergence of unaligned AGIs, because nothing else likely suffices. Your public mischaracterization of the whole field composed of many very smart people only shows your ignorance. Note that Yann LeCun d…

> Most, perhaps all, AI alignment researchers do not suggest that we limit the AGI’s capabilities. Rather, it becomes clear that we need to engineer a very capable AGI which aligns with us and use it to help control the emergence of unaligned AGIs, because nothing else likely suffices.

Alternate wording: Mr. Yud has invented a religion that comes with a predefined Satan (evil AGI) and life work (invent God to beat it). A religion with no deity but only an anti-deity is a bit unique but there's probably historical examples.

Although that's not really what he says in the post. He says we've already failed to do it and are now doomed. Of course, saying we're all doomed (millenarianism) is what preachers have always done at some point.

> Your public mischaracterization of the whole field composed of many very smart people only shows your ignorance.

https://en.wikipedia.org/wiki/Courtier's_reply

Note, something getting a lot of smart-looking posts online actually isn't evidence that this is the state of the field. As we know from Yud's own post (https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...) the thing he's upset about is that people who actually run AI research orgs like FAIR don't believe him. And as we know from an HN post a few days ago (…which I forgot the title of), once you go offline you find most smart people out there aren't publicly posting anything, don't necessarily agree with the consensus opinion online about anything is, and don't know there is one.

…I wasn't talking about Yud though. He has a good reason to care about this, it being his job. I'm just saying people posting about it as if it's a certain risk are listening to him because it appeals to nerds. And, of course, if you value your own "intelligence" and thinks it gives you superpowers then a theory that says something with even more "intelligence" can exist and gets even better superpowers is going to be scary to you.

Re: OpenAI

#145

Post by c1ccccc1 »

astrange wrote: post #140
c1ccccc1 wrote: post #123

Earlier quoted context omitted.

This is your friendly physics reminder that perpetual motion machines have nothing to do with this. It's hard to turn the whole planet into paperclips because paperclips are mostly made of iron, while the planet contains many other elements. Of course, with a high enough level of technology, it might be possible to fuse together the non-iron elements, so that you would end up with just a bunch of iron nuclei. This wo…

It's less than perpetual, but a planet is a lot of raw material to work through for a machine without it breaking down, if the machine's also eaten all the people who can repair it.

Yes, self replicating machines would probably be required. But self replicators are definitely physically possible.

Re: OpenAI

#146

Post by nopinsight »

astrange wrote: post #144
nopinsight wrote: post #143

Earlier quoted context omitted.

Most, perhaps all, AI alignment researchers do not suggest that we limit the AGI’s capabilities. Rather, it becomes clear that we need to engineer a very capable AGI which aligns with us and use it to help control the emergence of unaligned AGIs, because nothing else likely suffices. Your public mischaracterization of the whole field composed of many very smart people only shows your ignorance. Note that Yann LeCun d…

> Most, perhaps all, AI alignment researchers do not suggest that we limit the AGI’s capabilities. Rather, it becomes clear that we need to engineer a very capable AGI which aligns with us and use it to help control the emergence of unaligned AGIs, because nothing else likely suffices. Alternate wording: Mr. Yud has invented a religion that comes with a predefined Satan (evil AGI) and life work (invent God to beat it…

My first paragraph was quite substantive which you didn’t really address, other than asserting in the last sentence that one’s intelligence does not give one power in the world. Perhaps the intelligence of an individual does not mean much in most cases, but we already have ample evidence that a sufficiently intelligent species (when we include social intelligence in the definition) can dominate all others which are stronger, faster, or multiply faster.

Reminder: An AGI will be much faster at communicating and (if not successfully contained) multiplying than humans ever could.

Major AI research organizations including DeepMind and OpenAI have AI safety programs and people working full-time on it.

My second paragraph in GP was a reply in kind to your…

“This field is fairly silly because it just involves people making up a lot of incoherent concepts and then asserting they're both possible (because they seem logical after 5 seconds of thought) and likely (because anything you've decided is possible could eventually happen). When someone brings it up, rather than debate it, it'd be a better use of time to tell them they're being a nerd again.”

In retrospect, I shouldn’t have said it. But it’s also quite disappointing that your several paragraphs of reply largely doubled down on ad hominem attack to anyone who disagrees with you (eg by implying they all follow a prophet without thinking; I’d say many would be capable of reaching similar conclusions on their own).

Even Yann LeCun and other top researchers who disagree with the current AI safety programs were not so dismissive of the concerns. Note that many other top AI researchers do have concerns themselves. Bengio and Russell are some examples. I’ll stop here since it’s likely unproductive to continue.

Re: OpenAI

#147

Post by tlringer »

I'm not sure how Scott ended up buying the party line of the weird AGI Doomsday Cult but so be it. In any case, none of the things he says about verifying AI in this post make any sense at all, and if OpenAI actually cares about verifying AI and not just about hiring people who believe in the AGI Doomsday party line, probably they should hire verification people. Alas, that is not the point.

Re: OpenAI

#148

Post by tlringer »

sigmoid10 wrote: post #36
astrange wrote: post #28

Earlier quoted context omitted.

He mostly studies computational complexity. Quantum computing is a part of that, but there's other subfields. Though the kind of AI safety described in this post seems more like an extremely fancy version of program verification, so out of CS bloggers you'd expect John Regehr to get into it.

>the kind of AI safety described in this post seems more like an extremely fancy version of program verification It kind of is. The field of AI safety is actually much more advanced than most people realise, with actual, real techniques to e.g. make sure neural networks are aligned with certain goals even under fluctuating parameters. Granted, we're still far from soothing an AGI before it can do something bad, but t…

If you're interested in verification you should probably talk to people who actually work on verification, for example, literally anyone from our research community: https://www.floc2022.org/

Re: OpenAI

#149

Post by tlringer »

csee wrote: post #99
d--b wrote: post #23

AI is not going to become self aware and destroy the world. AI is going to cause something like the industrial revolution of the 19th century: massive changes in who is rich, massive changes in the labor market, massive changes in how people make war, etc. It’s already started really. What worries me most is that as long as society is capitalist, AI will be used to optimize for self-enrichment, likely causing an even…

"AI is not going to ... destroy the world." Bare assertion fallacy? This question is hotly debated and I don't believe it can be so easily dismissed like that. It is not obvious that aligning something much smarter than us will be a piece of cake.

It's a really absurd opinion that AI will destroy the world, and one that does not deserve serious consideration in any research community. It's only in strange Rationalist corners and the companies in Silicon Valley that echo those corners that this is considered at all "hotly debated."

Re: OpenAI

#150

Post by hansword »

nopinsight wrote: post #104
hansword wrote: post #58

'OpenAI, of course, has the word “open” right in its name, ... don’t expect me to share any proprietary information' Yeah, Mr Aaronson just lost quite a bit of respect from my side. Going into AI is a great move, moving to the ClosedAI corporation.......? Why? (Edit: Removed an outdated reference to Elon Musk, thanks @pilaf !)

The latest huge AI models that may germinate into an AGI all come from private corporations, largely because of the requisite resources and currently there’re very few if any public or nonprofit AI-focused organizations with such resources.

> huge AI models that may germinate into an AGI

are science fiction.

Post reply on HN