Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are there so many rationalist cults?

asteriskmag.com

Post reply on HN
Print view View original

751–760 of 947 posts

Re: Why are there so many rationalist cults?

#751

Post by torben-friis »

pavlov wrote: post #398

A very interesting read. My idea of these self-proclaimed rationalists was fifteen years out of date. I thought they’re people who write wordy fan fiction, but turns out they’ve reached the point of having subgroups that kill people and exorcise demons. This must be how people who had read one Hubbard pulp novel in the 1950s felt decades later when they find out he’s running a full-blown religion now. The article see…

As someone who started reading without knowing about rationalists, I actually came out without knowing much more. Lots of context is assumed I guess.

Some main figures and rituals are mentioned but I still don’t know how the activities and communities arise from the purported origin. How do we go from “let’s rationally analyze how we think and get rid of bias” to creating a crypto, or being hype focused on AI, or summoning demons? Why did they raise this idea of matching confrontation always with escalation? Why the focus on programming, is this a Silicon Valley thing?

Also lesswrong is mentioned but no context is given about it. I only know the name as a forum, just like somethingawful or Reddit, but I don’t know how it fits into the picture.

Re: Why are there so many rationalist cults?

#752

Post by rsynnott »

> The Sequences [posts on LessWrong, apparently] make certain implicit promises. There is an art of thinking better, and we’ve figured it out. If you learn it, you can solve all your problems, become brilliant and hardworking and successful and happy, and be one of the small elite shaping not only society but the entire future of humanity.

Ooh, a capital S and everything. I mean, I feel like it is fairly obvious, really. 'Rationalism' is a new religion, and every new religion spawns a bunch of weird, generally short-lived, cults. You might as well ask, in 100AD, "why are there so many weird Christian cults all of a sudden"; that's just what happens whenever any successful new religion shows up.

Rationalism might be particularly vulnerable to this because it lacks a strong central authority (much like early Christianity), but even with new religions which _did_ have a strong central authority from the start, like Mormonism or Scientology, you still saw this happening to some extent.

Re: Why are there so many rationalist cults?

#753

Post by heavyset_go »

skybrian wrote: post #541
pavlov wrote: post #398

A very interesting read. My idea of these self-proclaimed rationalists was fifteen years out of date. I thought they’re people who write wordy fan fiction, but turns out they’ve reached the point of having subgroups that kill people and exorcise demons. This must be how people who had read one Hubbard pulp novel in the 1950s felt decades later when they find out he’s running a full-blown religion now. The article see…

The WHO didn't declare a global pandemic until March 11, 2020 [1]. That's a little slow and some rationalists were earlier than that. (Other people too.) After reading a warning from a rationalist blog, I posted a lot about COVID news to another forum and others there gave me credit for giving the heads-up that it was a Big Deal and not just another thing in the news. (Not sure it made all that much difference, thoug…

Shitposting comedy forums were ahead of the WHO when it came to this, it didn't take a genius to understand what was going on before shit completely hit the fan.

Re: Why are there so many rationalist cults?

#754

Post by housecarpenter »

rpcope1 wrote: post #189
nathan_compton wrote: post #12

Thinking too hard about anything will drive you insane but I think the real issue here is that rationalists simply over-estimate both the power of rational thought and their ability to do it. If you think of people who tend to make that kind of mistake you can see how you get a lot of crazy groups. I guess I'm a radical skeptic, secular humanist, utilitarianish sort of guy, but I'm not dumb enough to think throwing a…

Even the real progenitors of a lot of this sort of thought, like E.T. Jaynes, expoused significantly more skepticism than I've ever seen a "rationalist" use. I would even imagine if you asked almost all rationalists who E.T. Jaynes was (if they weren't well versed in statistical mechanics) they'd have no idea who he was or why his work was important to applying "Bayesianism".

It would surprise me if most rationalists didn't know who Jaynes was. I first heard of him via rationalists. The Sequences talk about him in adulatory tones. I think Yudkowsky would acknowledge him as one of his greatest influences.

Re: Why are there so many rationalist cults?

#755

Post by rcxdude »

wredcoll wrote: post #530
pizzadog wrote: post #445

I have a lot of experience with rationalists. What I will say is: 1) If you have a criticism about them or their stupid name or how "'all I know is that I know nothing' how smug of them to say they're truly wise," rest assured they have been self flagellating over these criticisms 100x longer than you've been aware of their group. That doesn't mean they succeeded at addressing the criticisms, of course, but I can tel…

But are they scotsmen?

this isn't really a 'no true scotsman' thing, because I don't think the comment is saying 'no rationalist would go crazy', in fact they're very much saying the opposite, just claiming there's a large fraction which are substantially more moderate but also a lot less visible.

Re: Why are there so many rationalist cults?

#756

Post by tim333 »

tempfile wrote: post #713
doubleunplussed wrote: post #476

Earlier quoted context omitted.

On the other hand, I'm baffled to encounter recursive self-improvement being discussed as something not only weird to expect, but as damning evidence of sloppy thinking by those who speculate about it. We have an existence proof for intelligence that can improve AI: humans. If AI ever gets to human-level intelligence, it would be quite strange if it couldn't improve itself. Are people really that sceptical that AI wi…

Consider that even the named phenomenon is sloppy: "recursive self improvement" does not imply "self improvement without bounds". This is the "what if you hit diminishing returns and never get past it" claim. Absolutely no justification for the jump, ever, among AI boosters. > If AI ever gets to human-level intelligence This picture of intelligence as a numerical scale that you just go up or down, with ants at the bo…

>"recursive self improvement" does not imply "self improvement without bounds"

I was thinking that. I mean if you look at something like AlphaGo it was based on human training and then they made one I think called AlphaZero which learned by playing against itself and got very good but not infinitely good as it was still constrained by hardware. I think with Chess the best human is about 2800 on the ELO scale and computers about 3500. I imagine self improving AI would be like that - smarter than humans but not infinitely so and constrained by hardware.

Also like humans still play chess even if computers are better, I imagine humans will still do the usual kind of things even if computers get smarter.

Re: Why are there so many rationalist cults?

#757

Post by throwanem »

aa-jv wrote: post #737
ImaCake wrote: post #490

Earlier quoted context omitted.

Not sure why you got down voted for this. The opening paragraph of the article reads as suspicious to the observant outsider: >The rationalist community was drawn together by AI researcher Eliezer Yudkowsky’s blog post series The Sequences, a set of essays about how to think more rationally. Anyone who had just read a lot about Scientology would read that and have alarm bells ringing.

Out of curiosity, why would the bells be ringing in this case? Is it just the fact that a single person is exerting influence over their followers by way of essays?

Even a marginal familiarity with the history of Scientology is an excellent curative for the idea that you can think yourself into superpowers, or that you should ever trust anyone who promises to teach you how.

The consequences of ignorance on this score are all drearily predictable to anyone with a modicum of both good sense and world knowledge, which is why they've come as such a surprise to Yudkowsky.

Re: Why are there so many rationalist cults?

#758

Post by astrange »

ufmace wrote: post #345
ar-nelson wrote: post #200

Earlier quoted context omitted.

I find Yudowsky-style rationalists morbidly fascinating in the same way as Scientologists and other cults. Probably because they seem to genuinely believe they're living in a sci-fi story. I read a lot of their stuff, probably too much, even though I find it mostly ridiculous. The biggest nonsense axiom I see in the AI-cult rationalist world is recursive self-improvement. It's the classic reason superintelligence tak…

I agree. There's also the point of hardware dependance. From all we've seen, the practical ability of AI/LLMs seems to be strongly dependent on how much hardware you throw at it. Seems pretty reasonable to me - I'm skeptical that there's that much out there in gains from more clever code, algorithms, etc on the same amount of physical hardware. Maybe you can get 10% or 50% better or so, but I don't think you're going…

I think you can get a few more gigantic step functions' worth of improvement on the same hardware. For instance, LLMs don't have any kind of memory, short or long term.

Re: Why are there so many rationalist cults?

#759

Post by rcxdude »

optimalsolver wrote: post #3

Pertinent Twitter comment: "Rationalism is such an insane name for a school of thought. Like calling your ideology correctism or winsargumentism" https://x.com/growing_daniel/status/1893554844725616666

It's not particularly unusual, though. See the various kinds of 'Realist' groups, for example, which have a pretty wild range of outlooks. (both Realist and Rationalist also have the neat built-in shield of being able to say "look, I don't particularly like the conclusions I'm coming to, they just are what they are", so it's a convenient framing for unpalatable beliefs)

Re: Why are there so many rationalist cults?

#760

Post by rsynnott »

azangru wrote: post #733
pavlov wrote: post #398

A very interesting read. My idea of these self-proclaimed rationalists was fifteen years out of date. I thought they’re people who write wordy fan fiction, but turns out they’ve reached the point of having subgroups that kill people and exorcise demons. This must be how people who had read one Hubbard pulp novel in the 1950s felt decades later when they find out he’s running a full-blown religion now. The article see…

> Rationalists came to correct views about the COVID-19 pandemic while many others were saying masks didn’t work I wonder what views about covid-19 are correct. On masks, I remember the mainstream messaging went through the stages that were masks don't work, some masks work, all masks work, double masking works, to finally masks don't work (or some masks work; I can't remember where we ended up).

> to finally masks don't work (or some masks work; I can't remember where we ended up).

Most masks 'work', for some value of 'work', but efficacy differs (which, to be clear, was ~always known; there was a very short period when some authorities insisted that covid was primarily transmitted by touch, but you're talking weeks at most). In particular I think what confused people was that the standard blue surgical masks are somewhat effective at stopping an infected person from passing on covid (and various other things), but not hugely effective at preventing the wearer from contracting covid; for that you want something along the lines of an n95 respirator.

The main actual point of controversy was whether it was airborne or not (vs just short-range spread by droplets); the answer, in the end, was 'yes', but it took longer than it should have to get there.

Post reply on HN