Weird. Seems like this is a plagiarism of this article:
https://www.strangeloopcanon.com/p/agi-strange-equation
(Or maybe it's an independent reinvention of the same idea, and something's just in the water.)
AGI Doom and the Drake Equation
11–20 of 113 posts
Re: AGI Doom and the Drake Equation
#12One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren…
“AGI doomers mistakenly believe that intelligence lets you find the right ethics”
“That’s why they can’t see that superhuman AGI will be so smart it will choose to find and settle on the right ethics, like me!”
Re: AGI Doom and the Drake Equation
#13One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren…
I think you're misinterpreting the argument. The paperclip maximizer scenario is not "over-optimizing" anything, it's an example of that same ethical pluralism you mention. The paperclip maximizer believes that maximizing paperclips is the highest possible good. There is only one set of facts about reality, and rationalism aims to find that set, but it makes no claims about what should be done with that information. It's descriptive, not normative.
The fact that there's so much possible variance in ethical norms is what makes recursively self-improving AI so dangerous. Human ethical norms are highly complex, and are the result of a long evolutionary history that will not be shared by any AI. The chances of any arbitrary set of ethical values being compatible with human life is very low.
Re: AGI Doom and the Drake Equation
#14> Point 3 [improvement is not limited by computing power] is one that I’m skeptical about. Intelligence is expensive and it requires a lot of energy. There is obvious advantage and efficiency in letting ChatGPT manage cloud instances, which means it will happen, which means these resources could be requisitioned. (I don’t think LLMs pose a Bostrom threat. But the author’s arguments aren’t convincing.)
ChatGPT doesn't have a bank account/credit card that's footing the bill, though.
Re: AGI Doom and the Drake Equation
#15> Point 3 [improvement is not limited by computing power] is one that I’m skeptical about. Intelligence is expensive and it requires a lot of energy. There is obvious advantage and efficiency in letting ChatGPT manage cloud instances, which means it will happen, which means these resources could be requisitioned. (I don’t think LLMs pose a Bostrom threat. But the author’s arguments aren’t convincing.)
> There is obvious advantage
You have a different definition of “obvious” than I have.
Re: AGI Doom and the Drake Equation
#16One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren…
I think you're misinterpreting the argument. The paperclip maximizer scenario is not "over-optimizing" anything, it's an example of that same ethical pluralism you mention. The paperclip maximizer believes that maximizing paperclips is the highest possible good. There is only one set of facts about reality, and rationalism aims to find that set, but it makes no claims about what should be done with that information.…
Implicit in creating paper clips is its belief that it should create paper clips, which is a normative conclusion.
Re: AGI Doom and the Drake Equation
#17One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren…
I think you're misinterpreting the argument. The paperclip maximizer scenario is not "over-optimizing" anything, it's an example of that same ethical pluralism you mention. The paperclip maximizer believes that maximizing paperclips is the highest possible good. There is only one set of facts about reality, and rationalism aims to find that set, but it makes no claims about what should be done with that information.…
> Human ethical norms are highly complex, and are the result of a long evolutionary history that will not be shared by any AI. The chances of any arbitrary set of ethical values being compatible with human life is very low.
If AI is trained by a huge corpus of human language, it may very well share our norms/values.
Re: AGI Doom and the Drake Equation
#18> Point 3 [improvement is not limited by computing power] is one that I’m skeptical about. Intelligence is expensive and it requires a lot of energy. There is obvious advantage and efficiency in letting ChatGPT manage cloud instances, which means it will happen, which means these resources could be requisitioned. (I don’t think LLMs pose a Bostrom threat. But the author’s arguments aren’t convincing.)
ChatGPT doesn't have a bank account/credit card that's footing the bill, though.
> ChatGPT doesn't have a bank account/credit card that's footing the bill
This is phishing, which LLMs should be uniquely capable of.
Re: AGI Doom and the Drake Equation
#19> Point 3 [improvement is not limited by computing power] is one that I’m skeptical about. Intelligence is expensive and it requires a lot of energy. There is obvious advantage and efficiency in letting ChatGPT manage cloud instances, which means it will happen, which means these resources could be requisitioned. (I don’t think LLMs pose a Bostrom threat. But the author’s arguments aren’t convincing.)
> There is obvious advantage You have a different definition of “obvious” than I have.
> have a different definition of “obvious” than I have
Sysadmins and the engineers who manage clouds are expensive. They’re ultimately a translation layer between instructions in language and their tools. It’s profitable to supplement and replace them, and possible, a combination which to me makes something obviously likely to occur.
Re: AGI Doom and the Drake Equation
#20The nonexistence of grey goo (von neumann probes) is strong prior for safe agi. AI xrisk is woo. Paperclip maximizers are p-zombies. They can't exist. Chicken littles see apocalypses on every horizon even though they dont understand the technology at all. "I can imagine this destroying the world" is their justification. Even though their "imagination" is 16x16 greyscale.
I’ve got a theory on the fermi paradox stuff I need to flesh out (and research, am assuming this isn’t original), among like a dozen other things I’ve been meaning to expand on that I haven’t: I think we severely underestimate how much we’re optimized to see what’s proximal to us.
I think there’s a strong possibility there may be “gray goo” all over the place, beings far bigger than us we’re inside of, physics that sits “parallel” to ours in whatever you’d call “space” in some construction we can’t comprehend, etc.
In short, I think the universe seems empty precisely because it’s distant, both evolutionarily and in terms of physical space.
Donald Hoffman’s been talking about a lot of stuff pointing in this direction.