Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are there so many rationalist cults?

asteriskmag.com

Post reply on HN
Print view View original

561–570 of 947 posts

Re: Why are there so many rationalist cults?

#561

Post by throw4847285 »

1attice wrote: post #393
analog31 wrote: post #262

Earlier quoted context omitted.

Perhaps part of being rational, as opposed to rationalist, is having a sense of when to override the conclusions of seemingly logical arguments.

In philosophy grad school, we described this as 'being reasonable' as opposed to 'being rational'. That said, big-R Rationalism (the Lesswrong/Yudkowsky/Ziz social phenomenon) has very little in common with what we've standardly called 'rationalism'; trained philosophers tend to wince a little bit when we come into contact with these groups (who are nevertheless chockablock with fascinating personalities and compelli…

I actually think that the fact that rationalists use the term "steel manning" betrays a lack of charity.

If the only thing you owe your interlocutor is to use your "prodigious intellect" to restate their own argument in the way that sounds the most convincing to you, maybe you are in fact a terrible listener.

Re: Why are there so many rationalist cults?

#562

Post by sfink »

ar-nelson wrote: post #200
JohnMakin wrote: post #85

One of a few issues I have with groups like these, is that they often confidently and aggressively spew a set of beliefs that on their face logically follow from one another, until you realize they are built on a set of axioms that are either entirely untested or outright nonsense. This is common everywhere, but I feel especially pronounced in communities like this. It also involves quite a bit of navel gazing that m…

I find Yudowsky-style rationalists morbidly fascinating in the same way as Scientologists and other cults. Probably because they seem to genuinely believe they're living in a sci-fi story. I read a lot of their stuff, probably too much, even though I find it mostly ridiculous. The biggest nonsense axiom I see in the AI-cult rationalist world is recursive self-improvement. It's the classic reason superintelligence tak…

> What if "increasing intelligence", which is a very vague goal, has diminishing returns, making recursive self-improvement incredibly slow?

This is sort of what I subscribe to as the main limiting factor, though I'd describe it differently. It's sort of like Amdahl's Law (and I imagine there's some sort of Named law that captures it, I just don't know the name): the magic AI wand may be very good at improving some part of AGI capability, but the more you improve that part, the more the other parts come to dominate. Metaphorically, even if the juice is worth the squeeze initially, pretty soon you'll only be left with a dried-out fruit clutched in your voraciously energy-consuming fist.

I'm actually skeptical that there's much juice in the first place; I'm sure today's AIs could generate lots of harebrained schemes for improvement very quickly, but exploring those possibilities is mind-numbingly expensive. Not to mention that the evaluation functions are unreliable, unknown, and non-monotonic.

Then again, even the current AIs have convinced a large number of humans to put a lot of effort into improving them, and I do believe that there are a lot of improvements that humans are capable of making to AI. So the human-AI system does appear to have some juice left. Where we'll be when that fruit is squeezed down to a damp husk, I have no idea.

Re: Why are there so many rationalist cults?

#564

Post by mitthrowaway2 »

staticman2 wrote: post #548
mitthrowaway2 wrote: post #247

Earlier quoted context omitted.

These aren't mutually exclusive. Even in The Terminator , Skynet's method of choice is nuclear war. Yudkowsky frequency expressses concern that a malevolent AI might synthesize a bioweapon. I personally worry that destroying the ozone layer might be an easy opening volley. Either way, I don't want a really smart computer spending its time figuring out plans to end the human species, because I think there are too many…

Terminator descends from a tradition of science fiction cold war parables. Even in Terminator 2 there's a line suggesting the movie isn't really about robots: John:We're not gonna make it, are we? People, I mean. Terminator: It's in your nature to destroy yourselves. Seems odd to worry about computers shooting the ozone when there's plenty of real existential threats loaded in missles aimed at you right now.

I'm not in any way discounting the danger represented by those missiles. In fact I think AI only makes it more likely that they might someday be launched. But I will say that in my experience the error-condition that causes a system to fail is usually the one that didn't seem likely to happen, because the more obvious failure modes were taken seriously from the beginning. Is it so unusual to be able to consider more than one risk at a time?

Re: Why are there so many rationalist cults?

#565

Post by lyu07282 »

fuzztester wrote: post #550

Has anyone here ever been a part of a cult? If so, got anything to share - anecdotes, learnings, cautions, etc.? I am never planning to be part of one; just interested to know, partly because I have lived adjacent to what might be one, at times.

There was this interview with Diane Benscoter who talked about her experience and reasons for joining a cult that I found very insightful: https://www.youtube.com/watch?v=6Ibk5vJ-4-o

The main point is that it isn't so much the cult (leader) so much as the victims being in a vulnerable mental state getting exploited.

Re: Why are there so many rationalist cults?

#566

Post by lmm »

freejazz wrote: post #435
username332211 wrote: post #413

Earlier quoted context omitted.

The reason why people can't be bothered to get formal training is that modern philosophy doesn't seem that useful. It was a while ago, but take the infamous story of the 2006 rape case in Duke University. If you check out coverage of that case, you get the impression every member of faculty that joined in the hysteria was from some humanities department, including philosophy. And quite a few of them refused to change…

>The reason why people can't be bothered to get formal training is that modern philosophy doesn't seem that useful. But rationalism is?

Well, maybe. It seems at least adjacent to the stuff that's been making a lot of people rich lately.

Re: Why are there so many rationalist cults?

#567

Post by sien »

dragonwriter wrote: post #545
sien wrote: post #537

Earlier quoted context omitted.

Trying to do a bit of formal philosophy at University is really worth doing. You realise that it's very hard to do well and it's intellectual quicksand. Reading philosophers and great writers as you suggest is better than joining a cult. It's just that you also want to write about what you're thinking in response to reading such people and ideally have what you write critiqued by smart people. Perhaps an AI could do…

> It's just that you also want to write about what you're thinking in response to reading such people and ideally have what you write critiqued by smart people. Perhaps an AI could do some of that these days. An AI can neither write about what you are thinking in your place nor substitute for a particularly smart critic, but might still be useful for rubber-ducking philosophical writing if used well.

Errrf. That was poor writing on my part.

I meant use the AI to critique what you have written in response to reading the suggested authors.

Yes, a particularly smart critic would be better. But an LLM is easily available.

Re: Why are there so many rationalist cults?

#568

Post by skybrian »

JKCalhoun wrote: post #235

> Many of them also expect that, without heroic effort, AGI development will lead to human extinction. Odd to me. Not biological warfare? Global warming? All-out nuclear war? I guess The Terminator was a formative experience for them. (For me perhaps it was The Andromeda Strain .)

Yes, sufficiently high intelligence is sometimes assumed to allow for rapid advances in many scientific areas. So, it could be biological warfare because AGI. Or nanotech, drone warfare, or something stranger.

I'm a little skeptical (there may be bottlenecks that can't be solved by thinking harder), but I don't see how it can be ruled out.

Re: Why are there so many rationalist cults?

#569

Post by sunshowers »

copularent wrote: post #472
sunshowers wrote: post #135

Earlier quoted context omitted.

I don't disagree, but to steelman the case for (neo)rationalism: one of its fundamental contributions is that Bayes' theorem is extraordinarily important as a guide to reality, perhaps at the same level as the second law of thermodynamics; and that it is dramatically undervalued by larger society. I think that is all basically correct. (I call it neorationalism because it is philosophically unrelated to the more trad…

As if these neorationalist are building a model and markov chain monte carlo sampling their life decisions. That is the bullshit part.

Agreed, yeah.

Re: Why are there so many rationalist cults?

#570

Post by gedy »

collingreen wrote: post #166
gedy wrote: post #53

Earlier quoted context omitted.

Paraphrasing someone I don't recall - when people believe in nothing, they'll believe anything.

And therefore you should believe in me and my low low 10% tithe! That's the only way to not get tricked into believing something wrong so don't delay!

That's not an endorsement of a particular religion.
Post reply on HN