Live data from Hacker News

Effective Altruism’s obsession with AI safety helps bury bad behavior

bloomberg.com

261–270 of 329 posts

Re: Effective Altruism’s obsession with AI safety helps bury bad behavior

#261
post #126

Earlier quoted context omitted.

What scientists?

A recent poll of published AI researchers showed a median estimate of a 5% chance AI causes human extinction, with half believing the chance is greater than 10%. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ A poll of NLP researchers showed 37% agreed that catastrophic outcomes of AI comparable to nuclear war were plausible. https://arxiv.org/abs/2208.12852

it's a little annoying that the "AI Impacts" paper seems to be three MIRIs in a trenchcoat, considering that this article is about EA's AI agenda and by extension Singularity/MIRI. There is a slight conflict of interest here - as the publishers of the paper are also taking funding specifically to "deal with the threat" they're publishing on.

Re: Effective Altruism’s obsession with AI safety helps bury bad behavior

#262

Earlier quoted context omitted.

Personally, I find the AI safety arguments sound pretty far-fetched, but I can’t come up with solid arguments against them. Also, when I look for a good ”debunking”, I don’t find anything. Mostly all I find are mockery, ad-hominem and the like. Frankly, the people who argue climate change isn’t real do a better job than the people who argue AI risk isn’t real, and they’re wrong!

From what I've seen, the AI safety arguments basically rely on assuming that the singularity is not merely real, but guaranteed: that we will create a full-fledged general AI, that it will be smarter than us, and that it will be fully capable of upgrading itself to be far beyond our control before we have any ability to prevent it from doing so. Every step of this relies on assumptions that are not merely questionabl…

Do you think it's a strike against these points to call them unfalsifiable? They seem no more so than any claim of technological possibility, such as "nuclear fission is possible", "fission chain reactions are possible", "nuclear weapons are possible", "nuclear fusion is possible", "nuclear fusion can be made economical", etc. Unless a technology violates laws of physics it's very hard to prove through math or experiment that something can't be done.

But the AI safety groups don't just assume these without any justification. The existence of the human brain itself is either a proof and very strong launching point for most of these. In 1930, maybe you could have convinced someone that a nuclear bomb is impossible or at least an unfalsifiable worry, because one had never yet been built, but your reasoning is like trying to cast doubt on the possibility of artificial digestion, when there are already billions of stomachs roaming the earth.

The idea that a computer can't possibly be made to accomplish whatever a brain can is losing plausibility with every passing day. And as for surpassing it: a human brain is powerful, but has so many surmountable limitations, like: requiring 20 years of education to get up to speed with existing experts; dying after 70-90 years; not being able to run copies of oneself in parallel; etc. We only need to imagine removing these limitations, and doing so violates no known scientific principles.

It's like having a 50-megaton warhead in your lab, and meanwhile the rocket scientists are gradually figuring out how to make rockets fly, and you're saying "yes this is a big bomb, but the idea that one of these could be more dangerous if mounted on a missile is an unfalsifiable assumption!"

Re: Effective Altruism’s obsession with AI safety helps bury bad behavior

#263

Earlier quoted context omitted.

Personally, I find the AI safety arguments sound pretty far-fetched, but I can’t come up with solid arguments against them. Also, when I look for a good ”debunking”, I don’t find anything. Mostly all I find are mockery, ad-hominem and the like. Frankly, the people who argue climate change isn’t real do a better job than the people who argue AI risk isn’t real, and they’re wrong!

The ai safety arguments rely on something called the “orthoganality thesis” which is a huge, unintuitive assumption. In real life, intelligence is associated with picking better goals. Generally, an entity with higher intelligence will pick different goals than a being with lower intelligence. The orthoganility thesis is an unproven assumption that intelligence and goals are not corellated, meaning an intelligent bei…

> The orthoganility thesis is an unproven assumption that intelligence and goals are not corellated, meaning an intelligent being can pursue stupid goals. States like that, it’s obviously wrong and laughable.

To me it's obviously wrong when stated that way because 'stupid' is not an appropriate metric for goals. We may consider goals good or bad within our value system, but that has little do to with 'stupid' or 'intelligent.' e.g. a body builder may have a goal to get as ripped as possible, a VC to make as much money as possible, an ascetic to deny the flesh as much as possible. Which of these are stupid or intelligent goals? I don't think that's a question that makes sense.

We may consider the goals of the Athenians (to expand their power) "worse" than the goals of the Melians (to be left alone)[0], but I don't see how they were "stupider."

[0]: https://en.wikipedia.org/wiki/Siege_of_Melos#The_Melian_Dial...

Re: Effective Altruism’s obsession with AI safety helps bury bad behavior

#264

Earlier quoted context omitted.

"Apocalypse cults are always wrong, AI safety is an apocalypse cult, therefore it is (probably) wrong" is an argument which really only betrays the ignorance of the speaker regarding the specifics of the AI doomer argument. Why are apocalypse cults always wrong? Do the same specific reasons apply to the doomers here?

I am well aware of the AI doomer argument, but you appear to assume that argumentation can be trusted just because it is convincing*. I believe you are not factoring in the possibility of epistemological error. *for what it's worth I do not find it convincing but I am arguing "even if I found it convincing, I wouldn't believe it"

What's your approach? Do you only believe arguments that sound unconvincing? Or just ignore all arguments, and try to adopt the beliefs that the popular kids in the schoolyard talk about?

Re: Effective Altruism’s obsession with AI safety helps bury bad behavior

#265

Earlier quoted context omitted.

I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions. Since very smart human beings have in history done a lot of damage, but…

>I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions. This has been discussed under "warning shots" https://www.lesswrong.co…

That blog post makes some giant unsupported assumptions, like that the reason for the failure of a few activists to stop gain-of-function research was that the government is fundamentally bad at addressing risks, and not that biologists might know more about the risk/benefit tradeoff than amateurs.

Re: Effective Altruism’s obsession with AI safety helps bury bad behavior

#266

Earlier quoted context omitted.

The scenario described in this article comes the closest, detailing a mechanism that may have been responsible for the end-Permian mass extinction, wherein warming oceans become anoxic and begin to release huge quantities of hydrogen sulfide gas. The gas is directly lethal to most life in the oceans and on land, and destroys the ozone layer as a bonus. It's hard for me to imagine any way in which an agricultural huma…

By "climate change" people mean the sort of change that will be realistically caused by humans within the next few hundred years, not the sort of extinction caused by massive volcano activity that takes 10,000 years and happens once every 100 million years. Obviously if the climate changes drastically enough fast enough then everyone dies but nobody is suggesting that's the case for preset day human activity.

In terms of timeline, the key sentence would be "The so-called thermal extinction at the end of the Paleocene began when atmospheric CO2 was just under 1,000 parts per million (ppm)". The end-Permian was likely around 2500 ppm[1]. It doesn't really matter whether that carbon comes from supervolcanoes or from human emissions.

When the article was written, CO2 concentration was 385 ppm. Today it's 421 ppm. If CO2 concentrations were to rise linearly, the timeline for reaching 1,000 ppm would be 250 years. Achieving that would require emissions to stabilize at ~2014 levels. If emissions keep rising, then perhaps it's closer to 150 years. If strong feedbacks kick in, like methane from melting permafrost, then maybe 100 years or less, or maybe we'll reach that happy 2500 ppm mark and see what a real mass-extinction looks like.

Of course the ocean has a big thermal mass, so it will probably take quite some time to heat up enough to trigger the disaster even after we reach that level of atmospheric carbon. Hopefully everything will be fine?

[1] https://www.nature.com/articles/s41467-021-22298-7

Re: Effective Altruism’s obsession with AI safety helps bury bad behavior

#267
post #92

Earlier quoted context omitted.

I'm assuming you read it, so why does Harry Potter (a 11 year old in 1991/1992) think polyamory might (or probably would be, looking at the text right now) be a good idea in a future where people live a long time, in said fanfic? I think it's pretty dubious, but the other 11 and 12 year old girls in the same fanfic also seem to be fans of the idea of yaoi like situations, so all semblances of actual historical events…

Harry has voldemort's soul in him the HPMOR which is the explanation given for why he acts like an evil adult.

Yes, spoilers though, but I don't actually think that explains it. Are there any indications in cannon that Tom Riddle was ever interested in polyamory? I'm pretty sure voldemort is not interested in other people at all after his soul gets all divided, but I don't know about before that. Riddle is not from the US west coast either, so it's kind of dubious.

Re: Effective Altruism’s obsession with AI safety helps bury bad behavior

#268

Earlier quoted context omitted.

My strong belief is that HPMOR is a Harry Potter fanfic not for any calculated reason, but just because the author was in a demographic maximally interested in making Harry Potter fanfic.

he has literally stated that's why he did it as a Harry Potter fanfic. "There's a large number of potential readers who would enter at least moderately familiar with the Harry Potter universe." He was also reading a lot of HP fic himself, so he knew the stuff.

Let me slightly revise my claim:

I think that Eliezer is very well-situated by both age and social niche to find HP fanfiction vastly more appealing than most human beings do. I think that he genuinely finds the idea of "Harry Potter but more as a hard fantasy with a hero bent on breaking the setting by using everything without regard for genre convention" interesting as a story concept. And when he set out to write up his ideas about rationality in the form of something other than dry nonfiction essays, I think that HP fanfiction sounded good to him.

I don't think that he correctly found the medium that is maximally appealing to the potential target audience for his ideas (like, I definitely do not want to read a super long Harry Potter fanfic, and I think I'm joined in that by essentially everyone who's not a very geeky millennial), and I don't think that he gritted his teeth and wrote a HP fanfic in order to increase the audience for his ideas despite not liking HP fanfics.

Re: Effective Altruism’s obsession with AI safety helps bury bad behavior

#269
post #147
post #74

Earlier quoted context omitted.

Yeah, I can't really argue against the whole idea. My "P(doom)"(language from the article), at least as related to agentic AI, is pretty low. About 7 or 8 percent before I die, and I'm not really old. There's probably not a lot I could do about it anyway, given that most AI jobs are alleged to just make it worse. That said, I can't really go around telling people with different numbers they should be working on antiv…

"Roll a D20 and if you get 1 humanity as you know it will be destroyed in your lifetime" seems bad. Why do you consider 7 or 8 percent low?

I'm not one of those "all value in the universe" people, P(doom) for me does not have to be that bad. I would actually consider the inability to manufacture or unwillingness to manufacture certain things at scale in the US to count. Generally, my point is that agentic AI is not something I can really do much about, and other things have a combined P(doom) probability higher than it. The unwillingness in certain groups to use what they consider to be toxic or "forever" chemicals in the manufacture of physical products at scale that can make them world a better and less fragile place is something that could be worked in instead.

Re: Effective Altruism’s obsession with AI safety helps bury bad behavior

#270

Earlier quoted context omitted.

>I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions. This has been discussed under "warning shots" https://www.lesswrong.co…

That blog post makes some giant unsupported assumptions, like that the reason for the failure of a few activists to stop gain-of-function research was that the government is fundamentally bad at addressing risks, and not that biologists might know more about the risk/benefit tradeoff than amateurs.

Was there even a serious policy discussion of banning gain-of-function research?

Gain of function research is supposed to help with pandemics, right? Did it help with COVID?

Post reply on HN