Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are there so many rationalist cults?

asteriskmag.com

Post reply on HN
Print view View original

761–770 of 947 posts

Re: Why are there so many rationalist cults?

#761

Post by rsynnott »

socalgal2 wrote: post #610
ar-nelson wrote: post #200

Earlier quoted context omitted.

I find Yudowsky-style rationalists morbidly fascinating in the same way as Scientologists and other cults. Probably because they seem to genuinely believe they're living in a sci-fi story. I read a lot of their stuff, probably too much, even though I find it mostly ridiculous. The biggest nonsense axiom I see in the AI-cult rationalist world is recursive self-improvement. It's the classic reason superintelligence tak…

> it assumes that soon LLMs will gain the capability of assisting humans No, it does not. It assumes there will be progress in AI. It does not assume that progress will be in LLMs It doesn't require AI to be better than humans for AI to take over because unlike a human an AI can be cloned. You have have 2 AIs, then 4, then 8.... then millions. All able to do the same things as humans (the assumption of AGI). Build ca…

> No, it does not. It assumes there will be progress in AI. It does not assume that progress will be in LLMs

I mean, for the specific case of the 2027 doomsday prediction, it really does have to be LLMs at this point, just given the timeframes. It is true that the 'rationalist' AI doomerism thing doesn't depend LLMs, and in fact predates transformer-based models, but for the 2027 thing, it's gotta be LLMs.

Re: Why are there so many rationalist cults?

#762

Post by rsynnott »

jmoggr wrote: post #597

I think the comments here have been overly harsh. I have friends in the community and have visited the LessWrong "campus" several times. They seemed very welcoming, sincere, and were kind and patient even when I was basically asserting that several of their beliefs were dumb (in hopefully somewhat respectful manner). As for the AI doomerism, many in the community have more immediate and practical concerns about AI, h…

> They seemed very welcoming, sincere, and were kind and patient even when I was basically asserting that several of their beliefs were dumb

I mean, I'm not sure what that proves. A cult which is reflexively hostile to unbelievers won't be a very effective cult, as that would make recruitment almost impossible.

Re: Why are there so many rationalist cults?

#763

Post by rsynnott »

joe_the_user wrote: post #499
Jtsummers wrote: post #387

> Many of them also expect that, without heroic effort, AGI development will lead to human extinction. > These beliefs can make it difficult to care about much of anything else: what good is it to be a nurse or a notary or a novelist, if humanity is about to go extinct? Replace AGI causing extinction with the Rapture and you get a lot of US Christian fundamentalists. They often reject addressing problems in the envir…

A lot of people also believe that global warming will cause terrible problems. I think that's a plausible belief but if you combine people believing one or another of these things, you've a lot of the US. Which is to say that I don't think just dooming is going on. Especially, the belief in AGI doom has a lot of plausible arguments in its favor. I happen not to believe in it but as a belief system it is more similar…

> A lot of people also believe that global warming will cause terrible problems. I think that's a plausible belief but if you combine people believing one or another of these things, you've a lot of the US.

They're really quite different; precisely nobody believes that global warming will cause the effective end of the world by 2027. A significant chunk of AI doomers do believe that, and even those who don't specifically fall in with the 2027 timeline are often thinking in terms of a short timeline before an irreversible end.

Re: Why are there so many rationalist cults?

#764

Post by rsynnott »

JKCalhoun wrote: post #235

> Many of them also expect that, without heroic effort, AGI development will lead to human extinction. Odd to me. Not biological warfare? Global warming? All-out nuclear war? I guess The Terminator was a formative experience for them. (For me perhaps it was The Andromeda Strain .)

I mean, this is the religion/philosophy which produced Roko's Basilisk (and not one of their weird offshoot murder-cults, either, it showed up on LessWrong, and was taken at least somewhat seriously by people there, to the point that Yudkowsky censored it. Their beliefs about AI are... out there.

Re: Why are there so many rationalist cults?

#765

Post by BaseBaal »

JohnMakin wrote: post #266
kergonath wrote: post #241

Earlier quoted context omitted.

Another annoying one is the simulation theory group. They know just enough about Physics to build sophisticated mental constructs without understanding how flimsy the foundations are or how their logical steps are actually unproven hypotheses.

Agreed. This one is especially annoying to me and dear to my heart, because I enjoy discussing the philosophy behind this, but it devolves into weird discussions and conclusions fairly quickly without much effort at all. I particularly enjoy the tenets of certain sects of buddhism and how they view these things, but you'll get a lot of people that are doing a really pseudo-intellectual version of the Matrix where the…

Which sects of Buddhism? Just curious to read further about them.

Re: Why are there so many rationalist cults?

#766

Post by rsynnott »

SirFatty wrote: post #25
mlinhares wrote: post #4

> One is Black Lotus, a Burning Man camp led by alleged rapist Brent Dill, which developed a metaphysical system based on the tabletop roleplaying game Mage the Ascension. What the actual f. This is such an insane thing to read and understand what it means that i might need to go and sit in silence for the rest of the day. How did we get to this place with people going completely nuts like this?

Came to ask a similar question, but also has it always been like this? The difference is now these people/groups on the fringe had no visibility before the internet? It's nuts.

I mean, cults have constantly shown up for all of recorded human history. Read a history of Scientology and you'll see a lot of commonalities, say. Rationalism is probably the first major cult/new religion to emerge in the internet era (Objectivism may be a marginal case, as its rise overlapped with USENET a bit), which does make it especially visible.

Re: Why are there so many rationalist cults?

#767

Post by ptero »

lexandstuff wrote: post #578
qcnguy wrote: post #514

Earlier quoted context omitted.

[flagged]

The point of wearing a mask is to protect other people from your respiratory droplets. Please wear a mask when you're sick.

Indoors. There were decades of research leading to the recommendations of mask wearing when symptomatic and only indoors.

All that fell by the wayside when mask wearing became a covid-time cult. A friend (with a degree in epidemiology) told me that if she tried to argue those points and doubts outdoor mask mandates she will be the immediately out of her job.

The covid-time environment of shutting down scientific discussions because policymakers decided that we had enough science to reach a conclusion should not be forgotten, it was a reasonable concern turned into a cult. My 2c.

Re: Why are there so many rationalist cults?

#768

Post by rsynnott »

JTbane wrote: post #20
mlinhares wrote: post #4

> One is Black Lotus, a Burning Man camp led by alleged rapist Brent Dill, which developed a metaphysical system based on the tabletop roleplaying game Mage the Ascension. What the actual f. This is such an insane thing to read and understand what it means that i might need to go and sit in silence for the rest of the day. How did we get to this place with people going completely nuts like this?

I don't know how you can call yourself a "rationalist" and base your worldview on a fantasy game.

I mean see also the Democratic People's Republic of Korea. You can't really take what groups call themselves too seriously.

Re: Why are there so many rationalist cults?

#769

Post by empiricus »

godelski wrote: post #504
empiricus wrote: post #442

Earlier quoted context omitted.

soo many things make no sense in this comment that I feel like 20% chance this a mid quality gpt. and so much interpolation effort, but starting from hearsay instead of primary sources. then the threads stop just before seeing the contradiction with the other threads. I imagine this is how we all reason most of the time, just based on vibes :(

Sure, I wrote a lot and it's a bit scattered. You're welcome to point to something specific but so far you haven't. Ironically, you're committing the error you're accusing me of. I'm also not exactly sure what you mean because the only claim I've made is that they've made assumptions where there are other possible, and likely, alternatives. It's much easier to prove something wrong than prove it right (or in our case…

And about this second comment, I agree that intelligence is bounded. We can discuss how much more intelligence is theoretically possible, but even if limit ourselves to extrapolation from human variance (agency of musk, math smart of von neumann, manipulative as trump, etc), and add a little more speed and parallelism (100 times faster, 100 copies cooperating), then we can get pretty far.

Also I agree we are all pretty fucking dumb, and cannot make this kind of predictions, which is actually one very important point in the rationalist circles: doom is not certain, but p(doom) looks uncomfortably high though. How lucky do you feel?

Re: Why are there so many rationalist cults?

#770

Post by rsynnott »

rglover wrote: post #116
mlinhares wrote: post #4

> One is Black Lotus, a Burning Man camp led by alleged rapist Brent Dill, which developed a metaphysical system based on the tabletop roleplaying game Mage the Ascension. What the actual f. This is such an insane thing to read and understand what it means that i might need to go and sit in silence for the rest of the day. How did we get to this place with people going completely nuts like this?

I came to comments first. Thank you for sharing this quote. Gave me a solid chuckle. I think people are going nuts because we've drifted from the dock of a stable civilization. Institutions are a mess. Economy is a mess. Combine all of that together with the advent of social media making the creation of echo chambers (and the inevitable narcissism of "leaders" in those echo chambers) effortless and ~15 years later, w…

> I think people are going nuts because we've drifted from the dock of a stable civilization.

When was stable period, exactly? I'm 40; the only semi-stable bit I can think of in my lifetime was a few years in the 90s (referred to, sometimes unironically, as "the end of history" at the time, before history decided to come out of retirement).

Everything's always been unstable, people sometimes just take a slightly rose-tinted view of the past.

Post reply on HN