Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are there so many rationalist cults?

asteriskmag.com

Post reply on HN
Print view View original

941–947 of 947 posts

Re: Why are there so many rationalist cults?

#941

Post by throwanem »

radekn wrote: post #940
ImaCake wrote: post #490

Earlier quoted context omitted.

Not sure why you got down voted for this. The opening paragraph of the article reads as suspicious to the observant outsider: >The rationalist community was drawn together by AI researcher Eliezer Yudkowsky’s blog post series The Sequences, a set of essays about how to think more rationally. Anyone who had just read a lot about Scientology would read that and have alarm bells ringing.

It's been 17 years since the series was written and rationalists haven't become a cult with Yudkowsky as leaders, so it's safe to say those were false alarms

Not with Yudkowsky as leader, that much is true.

Re: Why are there so many rationalist cults?

#942

Post by eschaton »

elliotto wrote: post #692
eschaton wrote: post #687

Earlier quoted context omitted.

Holding one’s own against Stephen Wolfram isn’t exactly the endorsement it might seem.

I'm interested in this perspective, I haven't come across much criticism of Wolfram but I haven't really looked for it much either. Is it because of his theory of everything ruliad stuff? I really enjoy his blog posts and his work on automata seems to be well respected. I've felt he presents a solid epistemology.

He’s pretty widely regarded as a kook by real scientists. He did some good math and physics back in the day and then some good work on cellular automata, and assumed that meant he was smarter than everybody else and that all of science was in fact a subset of his special focus, which it’s pretty trivially not.

Furthermore he has a very long history of grossly misrepresenting his accomplishments, whether by claiming authorship of his employees’ accomplishments or denying he was ever exposed to work that he built upon.

For example, I believe it’s been proven now that he had substantial exposure to MACSYMA before he started to lead his employees in the development of Mathematica—but that didn’t stop him from claiming it was entirely original and even threatening others with patent and other IP infringement claims for trying to do things that were similar.

(Which is the other problem with him: He doesn’t just take credit for the work he pays others to do, he’s an “IP protection” maximalist.)

Re: Why are there so many rationalist cults?

#943

Post by DirkH »

sjducb wrote: post #823
stickfigure wrote: post #674

Earlier quoted context omitted.

You're falling into some sort of fallacy; maybe a better rationalist than I could name it. The "they" you are describing is a large body of disparate people spread around the world. We're reading an article that focuses on a few dysfunctional subgroups. They are interesting because they are so dysfunctional and rare. Or put it this way: Name one -ism that _doesn't_ have sub/splinter groups that kill people. Even Paci…

This sounds like the No True Scotsman fallacy. We know all true scotsmen are good upstanding citizens. If you find a Scotsman who is a criminal, then obviously he is not a true Scotsman. If you find a rationalist who believes something mad then obviously he is not a true rationalist. There are now so many logical fallacies that you can point to any argument and say it’s a logical fallacy.

Rationality also last I checked doesn't really build ought statements insofar as morality is concerned. You can use it for good or evil and still be rational.

There is no contradiction in being very evil and very rational. It is easy to think of brilliant scientists who were evil beyond evil throughout history as an analogous example.

Re: Why are there so many rationalist cults?

#944

Post by DirkH »

lyu07282 wrote: post #586
skrebbel wrote: post #246

This article is beautifully written, and it's full of proper original research. I'm sad that most comments so far are knee-jerk "lol rationalists" type responses. I haven't seen any comment yet that isn't already addressed in much more colour and nuance in the article itself.

> I haven't seen any comment yet that isn't already addressed in much more colour and nuance in the article itself. I once called rationalists infantile, impotent liberal escapism, perhaps that's the novel take you are looking for. Essentially my view is that the fundamental problem with rationalists and the effective altruist movement is that they are talking about profound social and political issues, with any and…

This is basically exactly the same kind of criticism the far-left throws at effective altruism (EA), except they call EA moronic hyper-capitalism that is too fixated on quantifying cause and effect with modern economic theories that it ignores the systemic injustices inherent within the systems it works within. As a consequence it is doomed from the onset and completely ineffective and is only counterproductive and any other conclusion is dumb.

Re: Why are there so many rationalist cults?

#945

Post by DirkH »

JohnMakin wrote: post #85

One of a few issues I have with groups like these, is that they often confidently and aggressively spew a set of beliefs that on their face logically follow from one another, until you realize they are built on a set of axioms that are either entirely untested or outright nonsense. This is common everywhere, but I feel especially pronounced in communities like this. It also involves quite a bit of navel gazing that m…

Rationalists and effective altruism have reasoning under uncertainty as cornerstones of their respective movements.

There is an entire philosophical theory called deep cluelessness which is far more nuanced than just "unsure" which was built on by an Oxford EA philosopher.

I personally know multiple people in the movement who say they are deeply clueless on matters where they can altogether affect where hundreds of thousands of dollars under their care is directed.

And guess what, it does give them pause and they don't just follow some weird entirely untested or nonsense set of axioms. They consider second order effects, backfire risk and even hedging interventions in case their worldview is incorrect. All this careful reasoning I just never ever see in any any other social movement that has money it can direct where there is clear uncertainty with how best to allocate it.

Re: Why are there so many rationalist cults?

#946

Post by DirkH »

_Algernon_ wrote: post #728
quickthrowman wrote: post #489

Earlier quoted context omitted.

I only know Eliezer Yudkowsky from his Harry Potter fanfiction, most notably Harry Potter and the Methods of Rationality . Is he known publicly for some other reason?

He writes scare-mongering books about AI doomerism such as If Anyone Builds It, Everyone Dies . I short, another variant of commercializing the human fear response.

Name one other industry that has adherents that say its products will kill us all as a means to boost sales. Assuming you're actually claiming people are actually saying AI might kill us all to boost AI market share. Like this is absurd. Apply it to any other industry:

Oil companies: "Climate change will soon end the human race and we are pushing scary narrative here so that people will buy more oil"

Biotech: "Our lethal compounds can be synthesized to a degree to end the human race. Clearly me scaring you like this will boost sales"

Arms dealers: "We are selling bombs that will guaruntee end the human race with 90% certainty. Wanna buy?"

It makes no sense. Hundreds of top AI searchers with zero financial stake in the private sector have raised the alarm about AI posing a risk on the same order of magnitude as nukes, climate change and bioterrorism.

Saying this is all just commercializing the human fear response is the same sort of lazy dismissive reasoning I can imagine was used when the first warning signs from epidemiologists came out regarding covid-19. "You're just trying to get funding for your anti-covid-19 research or some other political reason & and you're just using covid-19 fear to get what you want, how dare you!"

How much risk their is I can agree is up for debate, but that the motivations of people bringing up thee is a risk is just to commercialize things is so blatantly false it is a comical view to hold at this point. One of the fucking founders of AI Geoffrey Hinton spent the entire time during his Nobel prize award speech warning AI has a significant chance of killing us all if trends continue.

Like what, you think his old ass really cares about boosting book sales or trending on Twitter for saying something contrarian or something? No, he sincerely believes what he is saying.

If none of this convinces you something is up, what will? What do you need to see for it to be reasonable to go from completely dismiss to consider it as possibly serious?

Re: Why are there so many rationalist cults?

#947

Post by DirkH »

rsynnott wrote: post #763
joe_the_user wrote: post #499

Earlier quoted context omitted.

A lot of people also believe that global warming will cause terrible problems. I think that's a plausible belief but if you combine people believing one or another of these things, you've a lot of the US. Which is to say that I don't think just dooming is going on. Especially, the belief in AGI doom has a lot of plausible arguments in its favor. I happen not to believe in it but as a belief system it is more similar…

> A lot of people also believe that global warming will cause terrible problems. I think that's a plausible belief but if you combine people believing one or another of these things, you've a lot of the US. They're really quite different; precisely nobody believes that global warming will cause the effective end of the world by 2027. A significant chunk of AI doomers do believe that, and even those who don't specific…

I've yet to meet anyone who thinks it a certainty in the community. Even my short-timeline friends who don't contribute to retirement savings and otherwise have clearly seen their mental health deteriorate over the stress still don't think it a certainty at all. Just something they reason as likely enough that it is reasonable to stress out over and change life plans over.

I don't think it was dumb of scientists during the nuclear arms race in the 50s to think there was a significant enough chance humans were going to go extinct soon (years, not decades) because of the developments they were seeing and as a result getting very anxious and doing things like stopping their retirement savings plans.

The fears from that era were 100% well-founded within the group or scientists who were most on the know about developments. Same goes for AI development.

Talking to people I have met who used to work at OpenAI as AI researchers and left because of AI safety concerns paints a very clear "these are reasonable-headed people with well-informed beliefs" when they say things like there is a small but significant chance we lose control by 2027. Because humans are ganna human and place shareholder value and beating China over safety.

Post reply on HN