Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are there so many rationalist cults?

asteriskmag.com

Post reply on HN
Print view View original

671–680 of 947 posts

Re: Why are there so many rationalist cults?

#671

Post by elliotto »

username332211 wrote: post #413
vehemenz wrote: post #344

I get the impression that these people desperately want to study philosophy but for some reason can't be bothered to get formal training because it would be too humbling for them. I call it "small fishbowl syndrome," but maybe there's a better term for it.

The reason why people can't be bothered to get formal training is that modern philosophy doesn't seem that useful. It was a while ago, but take the infamous story of the 2006 rape case in Duke University. If you check out coverage of that case, you get the impression every member of faculty that joined in the hysteria was from some humanities department, including philosophy. And quite a few of them refused to change…

Couldn't you take this same line of reasoning and apply it to the rationalist group from the article who killed a bunch of people, and conclude that you shouldn't become a rationalist because you probably kill people?

Re: Why are there so many rationalist cults?

#672

Post by marcus_holmes »

What is it about San Francisco that makes it the global center for this stuff?

Reading this, I was reminded of the 60's hippy communes, that generally centered around SF, and the problems they reported. So similar, especially around that turning-inward group emotional dynamics problem, that such groups tend to become dysfunctional (as TFA says) by blowing up internal emotional group politics into huge problems that need the entire group to be involved in trying to heal (as opposed to, say, accepting that a certain amount of interpersonal conflict is inevitable in human groups and ignoring it). It's fascinating that the same kind of groups seem to encounter the same kind of problems despite being ~60 years apart and armed with a lot more tech and knowledge.

And, yeah, why SF?

Re: Why are there so many rationalist cults?

#673

Post by elliotto »

samdoesnothing wrote: post #577
guerrilla wrote: post #460

Earlier quoted context omitted.

I'm someone who has read all of that and much more, including intense study of SEP and some contemporary papers and textbooks, and I would say that I am absolutely not qualified to produce philosophy of the quality output by analytic philosophy over the last century. I can understand a lot of it, and yes, this is better than being completely ignorant of the last 2500 years of philosophy as most rationalists seem to b…

Oh I don't mean to imply reading some classical lit prepares you for a career producing novel works in philosophy, simply that if one wants to understand themselves, others, and the world better they don't need to go to university to do it. They can just read.

I think you are understating how difficult this is to do. I suspect there are a handful of super-geniuses who can read the philosophical canon and understand it, without some formal guidance. Plato and Dostoevsky might be possible (Socrates would be a bit difficult), but getting to Hegel and these newer more complex authors is almost impossible to navigate unless you are a savant.

I suspect a lot of the rationalists have gotten stuck here, and rather than seek out guidance or slowing down, changed tack entirely and decided to engage with the philosophers du jour, which unfortunately is a lot of slop running downstream from Thiel.

Re: Why are there so many rationalist cults?

#674

Post by stickfigure »

pavlov wrote: post #398

A very interesting read. My idea of these self-proclaimed rationalists was fifteen years out of date. I thought they’re people who write wordy fan fiction, but turns out they’ve reached the point of having subgroups that kill people and exorcise demons. This must be how people who had read one Hubbard pulp novel in the 1950s felt decades later when they find out he’s running a full-blown religion now. The article see…

You're falling into some sort of fallacy; maybe a better rationalist than I could name it.

The "they" you are describing is a large body of disparate people spread around the world. We're reading an article that focuses on a few dysfunctional subgroups. They are interesting because they are so dysfunctional and rare.

Or put it this way: Name one -ism that _doesn't_ have sub/splinter groups that kill people. Even Pacifism doesn't get a pass.

Re: Why are there so many rationalist cults?

#676

Post by doubleunplussed »

morleytj wrote: post #354
ar-nelson wrote: post #200

Earlier quoted context omitted.

I find Yudowsky-style rationalists morbidly fascinating in the same way as Scientologists and other cults. Probably because they seem to genuinely believe they're living in a sci-fi story. I read a lot of their stuff, probably too much, even though I find it mostly ridiculous. The biggest nonsense axiom I see in the AI-cult rationalist world is recursive self-improvement. It's the classic reason superintelligence tak…

The built in assumptions are always interesting to me, especially as it relates to intelligence. I find many of them (though not all), are organized around a series of fundamental beliefs that are very rarely challenged within these communities. I should initially mention that I don't think everyone in these communities believes these things, of course, but I think there's often a default set of assumptions going int…

I think you'll indeed find, if you were to seek out the relevant literature, that those claims are more or less true, or at least, are the currently best-supported interpretation available. So I don't think they're assumptions so much as simply current state of the science on the matter, and therefore widely accepted among those who for whatever reason have looked into it (or, more likely, inherited the information from someone they trust who has read up on it).

Interestingly, I think we're increasingly learning that although most aspects of human intelligence seem to correlate with each other (thus the "singular factor" interpretation), the grab-bag of skills this corresponds to are maybe a bit arbitrary when compared to AI. What evolution decided to optimise the hell out of in human intelligence is specific to us, and not at all the same set of skills as you get out of cranking up the number of parameters in an LLM.

Thus LLMs continuing to make atrocious mistakes of certain kinds, despite outshining humans at other tasks.

Nonetheless I do think it's correct to say that the rationalists think intelligence is a real measurable thing, and that although in humans it might be a set of skills that correlate and maybe in AIs it's a different set of skills that correlate (such that outperforming humans in IQ tests is impressive but not definitive), that therefore AI progress can be measured and it is meaningful to say "AI is smarter than humans" at some point. And that AI with better-than-human intelligence could solve a lot of problems, if of course it doesn't kill us all.

Re: Why are there so many rationalist cults?

#677

Post by elliotto »

sien wrote: post #567
dragonwriter wrote: post #545

Earlier quoted context omitted.

> It's just that you also want to write about what you're thinking in response to reading such people and ideally have what you write critiqued by smart people. Perhaps an AI could do some of that these days. An AI can neither write about what you are thinking in your place nor substitute for a particularly smart critic, but might still be useful for rubber-ducking philosophical writing if used well.

Errrf. That was poor writing on my part. I meant use the AI to critique what you have written in response to reading the suggested authors. Yes, a particularly smart critic would be better. But an LLM is easily available.

I find using an AI to understand complex philosophical topics one of my most unexpected use cases. Previously, I would get stuck scrolling through wikipedia full of incredibly opaque language, that assumes a background I don't have. But I can tell a bot what my background is, and it can make an explanation that is in the right level of complexity.

As an example, I'm reading a book on Buddhism, and I'm comfortable with Kant, and AI is useful for explaining to me a lot of the ideas they have as they relate to transcendental idealism (Kant).

On the other hand, I still don't know what a body without organs is.

Re: Why are there so many rationalist cults?

#678

Post by tintor »

godelski wrote: post #450
robocat wrote: post #399

Earlier quoted context omitted.

I saw an article recently that talked about stringing likely inferences together but ending up with an unreliable outcome because enough 0.9 probabilities one after the other lead to an unlikely conclusion. Edit: Couldn't find the article, but AI referenced Baysian "Chain of reasoning fallacy".

I think you have this oversimplified. Stringing together inferences can take us in either direction. It really depends on how things are being done and this isn't always so obvious or simple. But just to show both directions I'll give two simple examples (real world holds many more complexities) It is all about what is being modeled and how the inferences string together. If these are being multiplied, then yes, this…

Assuming your vampire tests are independent.

Re: Why are there so many rationalist cults?

#679

Post by doubleunplussed »

throw0101a wrote: post #340
thedudeabides5 wrote: post #304

Purity Spirals + Cheap Talk = irrational rationalists

> Purity Spirals This is an interesting idea (phenomenon?): > A purity spiral is a theory which argues for the existence of a form of groupthink in which it becomes more beneficial to hold certain views than to not hold them, and more extreme views are rewarded while expressing doubt, nuance, or moderation is punished (a process sometimes called "moral outbidding").[1] It is argued that this feedback loop leads to me…

Certainly something they're aware of - the same concept was discussed as early as in 2007 on Less Wrong under the name "evaporative cooling of group beliefs"

https://www.lesswrong.com/posts/ZQG9cwKbct2LtmL3p/evaporativ...

Re: Why are there so many rationalist cults?

#680

Post by elliotto »

eschaton wrote: post #336
JKCalhoun wrote: post #235

> Many of them also expect that, without heroic effort, AGI development will lead to human extinction. Odd to me. Not biological warfare? Global warming? All-out nuclear war? I guess The Terminator was a formative experience for them. (For me perhaps it was The Andromeda Strain .)

It makes a lot of sense when you realize that for many of the “leaders” in this community like Yudkowsky, their understanding of science (what it is, how it works, and its potential) comes entirely from reading science fiction and playing video games. Sad because Eli’s dad was actually a real and well-credentialed researcher at Bell Labs. Too bad he let his son quit school at an early age to be an autodidact.

I'm not at all a rationalist or a defender, but big yud has an epistemology that takes the form of the rationalist sacred text mentioned in the article (the sequences). A lot of it is well thought out, and probably can't be discarded as just coming from science fiction and video games. Yud has a great 4 hour talk with Stephen Wolfram where he holds his own.
Post reply on HN