Earlier quoted context omitted.
What scientists?
A recent poll of published AI researchers showed a median estimate of a 5% chance AI causes human extinction, with half believing the chance is greater than 10%. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ A poll of NLP researchers showed 37% agreed that catastrophic outcomes of AI comparable to nuclear war were plausible. https://arxiv.org/abs/2208.12852
Effective Altruism’s obsession with AI safety helps bury bad behavior
261–270 of 329 posts
Re: Effective Altruism’s obsession with AI safety helps bury bad behavior
#262Earlier quoted context omitted.
Personally, I find the AI safety arguments sound pretty far-fetched, but I can’t come up with solid arguments against them. Also, when I look for a good ”debunking”, I don’t find anything. Mostly all I find are mockery, ad-hominem and the like. Frankly, the people who argue climate change isn’t real do a better job than the people who argue AI risk isn’t real, and they’re wrong!
From what I've seen, the AI safety arguments basically rely on assuming that the singularity is not merely real, but guaranteed: that we will create a full-fledged general AI, that it will be smarter than us, and that it will be fully capable of upgrading itself to be far beyond our control before we have any ability to prevent it from doing so. Every step of this relies on assumptions that are not merely questionabl…
But the AI safety groups don't just assume these without any justification. The existence of the human brain itself is either a proof and very strong launching point for most of these. In 1930, maybe you could have convinced someone that a nuclear bomb is impossible or at least an unfalsifiable worry, because one had never yet been built, but your reasoning is like trying to cast doubt on the possibility of artificial digestion, when there are already billions of stomachs roaming the earth.
The idea that a computer can't possibly be made to accomplish whatever a brain can is losing plausibility with every passing day. And as for surpassing it: a human brain is powerful, but has so many surmountable limitations, like: requiring 20 years of education to get up to speed with existing experts; dying after 70-90 years; not being able to run copies of oneself in parallel; etc. We only need to imagine removing these limitations, and doing so violates no known scientific principles.
It's like having a 50-megaton warhead in your lab, and meanwhile the rocket scientists are gradually figuring out how to make rockets fly, and you're saying "yes this is a big bomb, but the idea that one of these could be more dangerous if mounted on a missile is an unfalsifiable assumption!"
Re: Effective Altruism’s obsession with AI safety helps bury bad behavior
#263Earlier quoted context omitted.
Personally, I find the AI safety arguments sound pretty far-fetched, but I can’t come up with solid arguments against them. Also, when I look for a good ”debunking”, I don’t find anything. Mostly all I find are mockery, ad-hominem and the like. Frankly, the people who argue climate change isn’t real do a better job than the people who argue AI risk isn’t real, and they’re wrong!
The ai safety arguments rely on something called the “orthoganality thesis” which is a huge, unintuitive assumption. In real life, intelligence is associated with picking better goals. Generally, an entity with higher intelligence will pick different goals than a being with lower intelligence. The orthoganility thesis is an unproven assumption that intelligence and goals are not corellated, meaning an intelligent bei…
To me it's obviously wrong when stated that way because 'stupid' is not an appropriate metric for goals. We may consider goals good or bad within our value system, but that has little do to with 'stupid' or 'intelligent.' e.g. a body builder may have a goal to get as ripped as possible, a VC to make as much money as possible, an ascetic to deny the flesh as much as possible. Which of these are stupid or intelligent goals? I don't think that's a question that makes sense.
We may consider the goals of the Athenians (to expand their power) "worse" than the goals of the Melians (to be left alone)[0], but I don't see how they were "stupider."
[0]: https://en.wikipedia.org/wiki/Siege_of_Melos#The_Melian_Dial...
Re: Effective Altruism’s obsession with AI safety helps bury bad behavior
#264Earlier quoted context omitted.
"Apocalypse cults are always wrong, AI safety is an apocalypse cult, therefore it is (probably) wrong" is an argument which really only betrays the ignorance of the speaker regarding the specifics of the AI doomer argument. Why are apocalypse cults always wrong? Do the same specific reasons apply to the doomers here?
I am well aware of the AI doomer argument, but you appear to assume that argumentation can be trusted just because it is convincing*. I believe you are not factoring in the possibility of epistemological error. *for what it's worth I do not find it convincing but I am arguing "even if I found it convincing, I wouldn't believe it"
Re: Effective Altruism’s obsession with AI safety helps bury bad behavior
#265Earlier quoted context omitted.
I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions. Since very smart human beings have in history done a lot of damage, but…
>I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions. This has been discussed under "warning shots" https://www.lesswrong.co…
Re: Effective Altruism’s obsession with AI safety helps bury bad behavior
#266Earlier quoted context omitted.
The scenario described in this article comes the closest, detailing a mechanism that may have been responsible for the end-Permian mass extinction, wherein warming oceans become anoxic and begin to release huge quantities of hydrogen sulfide gas. The gas is directly lethal to most life in the oceans and on land, and destroys the ozone layer as a bonus. It's hard for me to imagine any way in which an agricultural huma…
By "climate change" people mean the sort of change that will be realistically caused by humans within the next few hundred years, not the sort of extinction caused by massive volcano activity that takes 10,000 years and happens once every 100 million years. Obviously if the climate changes drastically enough fast enough then everyone dies but nobody is suggesting that's the case for preset day human activity.
When the article was written, CO2 concentration was 385 ppm. Today it's 421 ppm. If CO2 concentrations were to rise linearly, the timeline for reaching 1,000 ppm would be 250 years. Achieving that would require emissions to stabilize at ~2014 levels. If emissions keep rising, then perhaps it's closer to 150 years. If strong feedbacks kick in, like methane from melting permafrost, then maybe 100 years or less, or maybe we'll reach that happy 2500 ppm mark and see what a real mass-extinction looks like.
Of course the ocean has a big thermal mass, so it will probably take quite some time to heat up enough to trigger the disaster even after we reach that level of atmospheric carbon. Hopefully everything will be fine?
Re: Effective Altruism’s obsession with AI safety helps bury bad behavior
#267Earlier quoted context omitted.
I'm assuming you read it, so why does Harry Potter (a 11 year old in 1991/1992) think polyamory might (or probably would be, looking at the text right now) be a good idea in a future where people live a long time, in said fanfic? I think it's pretty dubious, but the other 11 and 12 year old girls in the same fanfic also seem to be fans of the idea of yaoi like situations, so all semblances of actual historical events…
Harry has voldemort's soul in him the HPMOR which is the explanation given for why he acts like an evil adult.
Re: Effective Altruism’s obsession with AI safety helps bury bad behavior
#268Earlier quoted context omitted.
My strong belief is that HPMOR is a Harry Potter fanfic not for any calculated reason, but just because the author was in a demographic maximally interested in making Harry Potter fanfic.
he has literally stated that's why he did it as a Harry Potter fanfic. "There's a large number of potential readers who would enter at least moderately familiar with the Harry Potter universe." He was also reading a lot of HP fic himself, so he knew the stuff.
I think that Eliezer is very well-situated by both age and social niche to find HP fanfiction vastly more appealing than most human beings do. I think that he genuinely finds the idea of "Harry Potter but more as a hard fantasy with a hero bent on breaking the setting by using everything without regard for genre convention" interesting as a story concept. And when he set out to write up his ideas about rationality in the form of something other than dry nonfiction essays, I think that HP fanfiction sounded good to him.
I don't think that he correctly found the medium that is maximally appealing to the potential target audience for his ideas (like, I definitely do not want to read a super long Harry Potter fanfic, and I think I'm joined in that by essentially everyone who's not a very geeky millennial), and I don't think that he gritted his teeth and wrote a HP fanfic in order to increase the audience for his ideas despite not liking HP fanfics.
Re: Effective Altruism’s obsession with AI safety helps bury bad behavior
#269Earlier quoted context omitted.
Yeah, I can't really argue against the whole idea. My "P(doom)"(language from the article), at least as related to agentic AI, is pretty low. About 7 or 8 percent before I die, and I'm not really old. There's probably not a lot I could do about it anyway, given that most AI jobs are alleged to just make it worse. That said, I can't really go around telling people with different numbers they should be working on antiv…
"Roll a D20 and if you get 1 humanity as you know it will be destroyed in your lifetime" seems bad. Why do you consider 7 or 8 percent low?
Re: Effective Altruism’s obsession with AI safety helps bury bad behavior
#270Earlier quoted context omitted.
>I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions. This has been discussed under "warning shots" https://www.lesswrong.co…
That blog post makes some giant unsupported assumptions, like that the reason for the failure of a few activists to stop gain-of-function research was that the government is fundamentally bad at addressing risks, and not that biologists might know more about the risk/benefit tradeoff than amateurs.
Gain of function research is supposed to help with pandemics, right? Did it help with COVID?