Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are there so many rationalist cults?

asteriskmag.com

Post reply on HN
Print view View original

491–500 of 947 posts

Re: Why are there so many rationalist cults?

#491

Post by at-fates-hands »

ctoth wrote: post #112
JohnMakin wrote: post #85

One of a few issues I have with groups like these, is that they often confidently and aggressively spew a set of beliefs that on their face logically follow from one another, until you realize they are built on a set of axioms that are either entirely untested or outright nonsense. This is common everywhere, but I feel especially pronounced in communities like this. It also involves quite a bit of navel gazing that m…

> I immediately become suspicious of anyone who is very certain of something, especially if they derived it on their own. Are you certain about this?

Isaac Newton would like to have a word.

Re: Why are there so many rationalist cults?

#493

Post by elictronic »

at-fates-hands wrote: post #491
ctoth wrote: post #112

Earlier quoted context omitted.

> I immediately become suspicious of anyone who is very certain of something, especially if they derived it on their own. Are you certain about this?

Isaac Newton would like to have a word.

I am not a big fan of alchemy, thank you though.

Re: Why are there so many rationalist cults?

#494

Post by tim333 »

tempfile wrote: post #293
tim333 wrote: post #250

Earlier quoted context omitted.

I've pondered recursive self-improvement. I'm fairly sure it will be a thing - we're at a point already where people could try telling Claude or some such to have a go, even if not quite at a point it would work. But I imagine take off would be very gradual. It would be constrained by available computing resources and probably only comparably good to current human researchers and so still take ages to get anywhere.

I honestly am not trying to be rude when I say this, but this is exactly the sort of speculation I find problematic and that I think most people in this thread are complaining about. Being able to tell Claude to have a go has no relation at all to whether it may ever succeed, and you don't actually address any of the legitimate concerns the comment you're replying to points out. There really isn't anything in this co…

I don't think it's vibes rather than my thinking about the problem.

If you look at the "legitimate concerns" none are really deal breakers:

>What if "increasing intelligence", which is a very vague goal, has diminishing returns, making recursive self-improvement incredibly slow?

I'm will to believe it will be slow though maybe it won't

>LLMs already seem to have hit a wall of diminishing returns

Who cares - there will be other algorithms

>What if there are several paths to different kinds of intelligence with their own local maxima

well maybe, maybe not

>Once AI realizes it can edit itself to be more intelligent, it can also edit its own goals. Why wouldn't it wirehead itself?

well - you can make another one if the first does that

Those are all potential difficulties with self improvement, not reasons it will never happen. I'm happy to say it's not happening right now but do you have any solid arguments that it won't happen in the next century?

To me the arguments against sound like people in the 1800s discussing powered flight and saying it'll never happen because steam engine development has slowed.

Re: Why are there so many rationalist cults?

#495

Post by [deleted] »

MajimasEyepatch wrote: post #271
jl6 wrote: post #187

Earlier quoted context omitted.

I don’t think it’s just (or even particularly) bad axioms, I think it’s that people tend to build up “logical” conclusions where they think each step is a watertight necessity that follows inevitably from its antecedents, but actually each step is a little bit leaky, leading to runaway growth in false confidence. Not that non-rationalists are any better at reasoning, but non-rationalists do at least benefit from some…

I feel this way about some of the more extreme effective altruists. There is no room for uncertainty or recognition of the way that errors compound. - "We should focus our charitable endeavors on the problems that are most impactful, like eradicating preventable diseases in poor countries." Cool, I'm on board. - "I should do the job that makes the absolute most amount of money possible, like starting a crypto exchang…

[deleted]

Re: Why are there so many rationalist cults?

#496

Post by copularent »

empath75 wrote: post #149
dfabulich wrote: post #82

The whole game of Rationalism is that you should ignore gut intuitions and cultural norms that you can't justify with rational arguments. Well, it turns out that intuition and long-lived cultural norms often have rational justifications, but individuals may not know what they are, and norms/intuitions provide useful antibodies against narcissist would-be cult leaders. Can you find the "rational" justification not to…

> The whole game of Rationalism is that you should ignore gut intuitions and cultural norms that you can't justify with rational arguments. The game as it is _actually_ played is that you use rationalist arguments to justify your pre-existing gut intuitions and personal biases.

Exactly. Humans are rationalizers. Operate on pre-existing gut intuitions and biases then invent after the fact rational sounding justifications.

I guess Pareto wasn't on the reading list for these intellectual frauds.

Those are actually the priors being updated lol.

Re: Why are there so many rationalist cults?

#497

Post by taurath »

Jtsummers wrote: post #387

> Many of them also expect that, without heroic effort, AGI development will lead to human extinction. > These beliefs can make it difficult to care about much of anything else: what good is it to be a nurse or a notary or a novelist, if humanity is about to go extinct? Replace AGI causing extinction with the Rapture and you get a lot of US Christian fundamentalists. They often reject addressing problems in the envir…

Raised to huddle close and expect the imminent utter demise of the earth and being dragged to the depths of hell if I so much as said a bad word I heard on TV, I have to keep an extremely tight handle on my anxiety in this day and age.

It’s not from a rational basis, but from being bombarded with fear from every rectangle in my house, and the houses of my entire community

Re: Why are there so many rationalist cults?

#498

Post by teh_klev »

mm263 wrote: post #307
skrebbel wrote: post #246

This article is beautifully written, and it's full of proper original research. I'm sad that most comments so far are knee-jerk "lol rationalists" type responses. I haven't seen any comment yet that isn't already addressed in much more colour and nuance in the article itself.

[flagged]

I have a link for you:

https://news.ycombinator.com/newsguidelines.html

Scroll to the bottom of the page.

Re: Why are there so many rationalist cults?

#499

Post by joe_the_user »

Jtsummers wrote: post #387

> Many of them also expect that, without heroic effort, AGI development will lead to human extinction. > These beliefs can make it difficult to care about much of anything else: what good is it to be a nurse or a notary or a novelist, if humanity is about to go extinct? Replace AGI causing extinction with the Rapture and you get a lot of US Christian fundamentalists. They often reject addressing problems in the envir…

A lot of people also believe that global warming will cause terrible problems. I think that's a plausible belief but if you combine people believing one or another of these things, you've a lot of the US.

Which is to say that I don't think just dooming is going on. Especially, the belief in AGI doom has a lot of plausible arguments in its favor. I happen not to believe in it but as a belief system it is more similar to a belief in global warming than to a belief in the raptures.

Re: Why are there so many rationalist cults?

#500

Post by toasterlovin »

ctkhn wrote: post #473
dan_quixote wrote: post #339

Earlier quoted context omitted.

As a former mechanical engineer, I visualize this phenomenon like a "tolerance stackup". Effectively meaning that for each part you add to the chain, you accumulate error. If you're not damn careful, your assembly of parts (or conclusions) will fail to measure up to expectations.

Basically the same as how dead reckoning your location works worse the longer you've been traveling?

Dead reckoning is a great analogy for coming to conclusions based on reason alone. Always useful to check in with reality.
Post reply on HN