Live data from Hacker News

Why are there so many rationalist cults?

asteriskmag.com

411–420 of 947 posts

Re: Why are there so many rationalist cults?

#411

Earlier quoted context omitted.

I feel this way about some of the more extreme effective altruists. There is no room for uncertainty or recognition of the way that errors compound. - "We should focus our charitable endeavors on the problems that are most impactful, like eradicating preventable diseases in poor countries." Cool, I'm on board. - "I should do the job that makes the absolute most amount of money possible, like starting a crypto exchang…

"I should do the job that makes the absolute most amount of money possible, like starting a crypto exchange, so that I can use my vast wealth in the most effective way." Has always really bothered me because it assumes that there are no negative impacts of the work you did to get the money. If you do a million dollars worth of damage to the world and earn 100k (or a billion dollars worth of damage to earn a million d…

> If you do a million dollars worth of damage to the world and earn 100k (or a billion dollars worth of damage to earn a million dollars), even if you spend all of the money you earned on making the world a better place, you arent even going to fix 10% of the damage you caused (and thats ignoring the fact that its usually easier/cheaper to break things than to fix them).

You kinda summed up a lot of the world post industrial revolution there, at least as far as stuff like toxic waste (Superfund, anyone?) and stuff like climate change, I mean for goodness sake let's just think about TEL and how they knew Ethanol could work but it just wasn't 'patentable'. [0] Or the "We don't even know the dollar amount because we don't have a workable solution" problem of PFAS.

[0] - I still find it shameful that a university is named after the man who enabled this to happen.

Re: Why are there so many rationalist cults?

#412

> Many of them also expect that, without heroic effort, AGI development will lead to human extinction. Odd to me. Not biological warfare? Global warming? All-out nuclear war? I guess The Terminator was a formative experience for them. (For me perhaps it was The Andromeda Strain .)

My interpretation: When they say "will lead to human extinction", they are trying to vocalize their existential terror that an AGI would render them and their fellow rationalist cultists permanently irrelevant - by being obviously superior to them, by the only metric that really matters to them.

Re: Why are there so many rationalist cults?

#413

I get the impression that these people desperately want to study philosophy but for some reason can't be bothered to get formal training because it would be too humbling for them. I call it "small fishbowl syndrome," but maybe there's a better term for it.

The reason why people can't be bothered to get formal training is that modern philosophy doesn't seem that useful.

It was a while ago, but take the infamous story of the 2006 rape case in Duke University. If you check out coverage of that case, you get the impression every member of faculty that joined in the hysteria was from some humanities department, including philosophy. And quite a few of them refused to change their mind even as the prosecuting attorney was being charged with misconduct. Compare that to Socrates' behavior during the trial of the admirals in 406 BC.

Meanwhile, whatever meager resistence was faced by that group seems to have come from economists, natural scientist or legal scholars.

I wouldn't blame people for refusing to study in a humanities department where they can't tell right from wrong.

Re: Why are there so many rationalist cults?

#414
post #193

Earlier quoted context omitted.

Humans are not chickens, but we sure do seem to love having a pecking order.

I think it is more simple in that we love tribalism. A long time ago being part of a tribe had such huge benefits over going it alone that it was always worth any tradeoffs. We have a much better ability to go it alone now but we still love to belong to a group. Too often we pick a group based on a single shared belief and don't recognize all the baggage that comes along. Life is also too complicated today. It is dif…

maybe the real innie/outie is the in-group/out-group. no spoilers, i haven't finished that show yet

Re: Why are there so many rationalist cults?

#415
Is it really that surprising that a group of humans who think they have some special understanding of reality compared to others tend to separate and isolate themselves until they fall into an unguided self-reinforcing cycle?

I'd have thought that would be obvious since it's the history of many religions (which seem to just be cults that survived the bottleneck effect to grow until they reached a sustainable population).

In other words, humans are wired for tribalism, so don't be surprised when they start forming tribes...

Re: Why are there so many rationalist cults?

#416

> Many of them also expect that, without heroic effort, AGI development will lead to human extinction. Odd to me. Not biological warfare? Global warming? All-out nuclear war? I guess The Terminator was a formative experience for them. (For me perhaps it was The Andromeda Strain .)

Check out "the precipice" by Tony Ord. Biological warfare and global warming are unlikely to lead to total human extinction (though both present large risks of massive harm).

Part of the argument is that we've had nuclear weapons for a long time but no apocalypse so the annual risk can't be larger than 1%, whereas we've never created AI so it might be substantially larger. Not a rock solid argument obviously, but we're dealing with a lot of unknowns.

A better argument is that most of those other risks are not neglected, plenty of smart people working against nuclear war. Whereas (up until a few years ago) very few people considered AI a real threat, so the marginal benefit of a new person working on it should be bigger.

Re: Why are there so many rationalist cults?

#417
> One way that thinking for yourself goes wrong is that you realize your society is wrong about something, don’t realize that you can’t outperform it, and wind up even wronger.

I've been there myself.

> And without the steadying influence of some kind of external goal you either achieve or don’t achieve, your beliefs can get arbitrarily disconnected from reality — which is very dangerous if you’re going to act on them.

I think this and the entire previous two paragraphs preceding it are excellent arguments for philosophical pragmatism and empiricism. It's strange to me that the community would not have already converged on that after all their obsessions with decision theory.

> The Zizians and researchers at Leverage Research both felt like heroes, like some of the most important people who had ever lived. Of course, these groups couldn’t conjure up a literal Dark Lord to fight. But they could imbue everything with a profound sense of meaning. All the minor details of their lives felt like they had the fate of humanity or all sentient life as the stakes. Even the guilt and martyrdom could be perversely appealing: you could know that you’re the kind of person who would sacrifice everything for your beliefs.

This helps me understand what people mean by "meaning". A sense that their life and actions actually matter. I've always struggled to understand this issue but this helps make it concrete, the kind of thing people must be looking for.

> One of my interviewees speculated that rationalists aren’t actually any more dysfunctional than anywhere else; we’re just more interestingly dysfunctional.

"We're"? The author is a rationalist too? That would definitely explain why this article is so damned long. Why are rationalists not able to write less? It sounds like a joke but this is seriously a thing. [EDIT: Various people further down in the comments are saying it's amphetamines and yes, I should have known that from my own experience. That's exactly what it is.]

> Consider talking about “ethical injunctions:” things you shouldn’t do even if you have a really good argument that you should do them. (Like murder.)

This kind of defeats the purpose, doesn't it? Also, this is nowhere justified in the article, just added on as the very last sentence.

Re: Why are there so many rationalist cults?

#418
post #195

Earlier quoted context omitted.

Her books were very popular with the gifted kids I hung out with in the late 80s. Cool kids would carry around hardback copies of Atlas Shrugged, impressive by the sheer physical size and art deco cover. How did that trend begin?

People reading the book and being into it and telling other people. It’s also a hard book to read so it may be smart kids trying to signal being smart.

Fountainhead is written at the 7th grade reading level. Its Lexile level is 780L. It's long and that's about it. By comparison, 1984 is 1090L.

Re: Why are there so many rationalist cults?

#419

Earlier quoted context omitted.

Perhaps part of being rational, as opposed to rationalist, is having a sense of when to override the conclusions of seemingly logical arguments.

In philosophy grad school, we described this as 'being reasonable' as opposed to 'being rational'. That said, big-R Rationalism (the Lesswrong/Yudkowsky/Ziz social phenomenon) has very little in common with what we've standardly called 'rationalism'; trained philosophers tend to wince a little bit when we come into contact with these groups (who are nevertheless chockablock with fascinating personalities and compelli…

> to me, it seems like this is a cargo-cultish transposition of the act of finding _fallacies in arguments_ into the domain of finding _faults in persons_.

Well put, thanks!

Re: Why are there so many rationalist cults?

#420
post #187

One of a few issues I have with groups like these, is that they often confidently and aggressively spew a set of beliefs that on their face logically follow from one another, until you realize they are built on a set of axioms that are either entirely untested or outright nonsense. This is common everywhere, but I feel especially pronounced in communities like this. It also involves quite a bit of navel gazing that m…

I don’t think it’s just (or even particularly) bad axioms, I think it’s that people tend to build up “logical” conclusions where they think each step is a watertight necessity that follows inevitably from its antecedents, but actually each step is a little bit leaky, leading to runaway growth in false confidence. Not that non-rationalists are any better at reasoning, but non-rationalists do at least benefit from some…

> I don’t think it’s just (or even particularly) bad axioms, I think it’s that people tend to build up “logical” conclusions where they think each step is a watertight necessity that follows inevitably from its antecedents, but actually each step is a little bit leaky, leading to runaway growth in false confidence.

This is what you get when you naively re-invent philosophy from the ground up while ignoring literally 2500 years of actual debugging of such arguments by the smartest people who ever lived.

You can't diverge from and improve on what everyone else did AND be almost entirely ignorant of it, let alone have no training whatsoever in it. This extreme arrogance I would say is the root of the problem.

Post reply on HN