Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are there so many rationalist cults?

asteriskmag.com

Post reply on HN
Print view View original

631–640 of 947 posts

Re: Why are there so many rationalist cults?

#631

Post by ethbr1 »

wombatpm wrote: post #522
godelski wrote: post #450

Earlier quoted context omitted.

I think you have this oversimplified. Stringing together inferences can take us in either direction. It really depends on how things are being done and this isn't always so obvious or simple. But just to show both directions I'll give two simple examples (real world holds many more complexities) It is all about what is being modeled and how the inferences string together. If these are being multiplied, then yes, this…

Can’t you improve thing if you can calibrate with a known good vampire? You’d think NIST or the CDC would have one locked in a basement somewhere.

The thought of a BIPM Reference Vampire made me chuckle.

Re: Why are there so many rationalist cults?

#632

Post by doubleunplussed »

ar-nelson wrote: post #619
socalgal2 wrote: post #610

Earlier quoted context omitted.

> it assumes that soon LLMs will gain the capability of assisting humans No, it does not. It assumes there will be progress in AI. It does not assume that progress will be in LLMs It doesn't require AI to be better than humans for AI to take over because unlike a human an AI can be cloned. You have have 2 AIs, then 4, then 8.... then millions. All able to do the same things as humans (the assumption of AGI). Build ca…

> It does not assume that progress will be in LLMs If that's the case then there's not as much reason to assume that this progress will occur now, and not years from now; LLMs are the only major recent development that gives the AI 2027 scenario a reason to exist. > You have have 2 AIs, then 4, then 8.... then millions The most powerful AI we have now is strictly hardware-dependent, which is why only a few big corpor…

> The most powerful AI we have now is strictly hardware-dependent

Of course that's the case and it always will be - the cutting edge is the cutting edge.

But the best AI you can run on your own computer is way better than the state of the art just a few years ago - progress is being made at all levels of hardware requirements, and hardware is progressing as well. We now have dedicated hardware in some of our own devices for doing AI inference - the hardware-specificity of AI doesn't mean we won't continue to improve and commoditise said hardware.

> The part that seems indefensible is the unexamined assumptions about LLMs' ability (or AI's ability more broadly) to jump to optimal human ability [...]

I don't think this is at all unexamined. But I think it's risky to not consider the strong possibility when we have an existence proof in ourselves of that level of intelligence, and an algorithm to get there, and no particular reason to believe we're optimal since that algorithm - evolution - did not optimise us for intelligence alone.

Re: Why are there so many rationalist cults?

#633

Post by emmelaich »

pavlov wrote: post #398

A very interesting read. My idea of these self-proclaimed rationalists was fifteen years out of date. I thought they’re people who write wordy fan fiction, but turns out they’ve reached the point of having subgroups that kill people and exorcise demons. This must be how people who had read one Hubbard pulp novel in the 1950s felt decades later when they find out he’s running a full-blown religion now. The article see…

I was reminded of Hubbard too. In particular the "[belief that one] should always escalate when threatened" strongly echoes Hubbard's advice to always attack attack. Never defend.

The whole thing reminds me of EST and a thousand other cults / self-improvement / self-actualisation groups that seem endemic to California ever since the 60s or before.

Re: Why are there so many rationalist cults?

#634

Post by ethbr1 »

toasterlovin wrote: post #500
ctkhn wrote: post #473

Earlier quoted context omitted.

Basically the same as how dead reckoning your location works worse the longer you've been traveling?

Dead reckoning is a great analogy for coming to conclusions based on reason alone. Always useful to check in with reality.

And always worth keeping an eye on the maximum possible divergence from reality you're currently at, based on how far you've reasoned from truth, and how less-than-sure each step was.

Maybe you're right. But there's a non-zero chance you're also max wrong. (Which itself can be bounded, if you don't wander too far)

Re: Why are there so many rationalist cults?

#635

Post by throw4847285 »

Dylan16807 wrote: post #596
throw4847285 wrote: post #561

Earlier quoted context omitted.

I actually think that the fact that rationalists use the term "steel manning" betrays a lack of charity. If the only thing you owe your interlocutor is to use your "prodigious intellect" to restate their own argument in the way that sounds the most convincing to you, maybe you are in fact a terrible listener.

Listening to other viewpoints is hard. Restating is a good tool to improve listening and understanding. I don't agree with this criticism at all, since that "prodigious intellect" bit isn't inherent to the term.

I was being snarky, but I think steelmanning does have one major flaw.

By restating the argument in terms that are most convincing to you, you may already be warping the conclusions of your interlocutor to fit what you want them to be saying. Charity is, "I will assume this person is intelligent and overlook any mistakes in order to try and understand what they are actually communicating." Steelmanning is "I can make their case for them, better than they could."

Of course this is downstream of the core issue, and the reason why steelmanning was invented in the first place. Namely, charity breaks down on the internet. Steelmanning is the more individualistic version of charity. It is the responsibility of people as individuals, not a norm that can be enforced by an institution or community.

Re: Why are there so many rationalist cults?

#636

Post by doubleunplussed »

Muromec wrote: post #342
psunavy03 wrote: post #335

A problem with this whole mindset is that humans, all of us, are only quasi-rational beings. We all use System 1 ("The Elephant") and System 2 ("The Rider") thinking instinctively. So if you end up in deep denial about your own capacity for irrationality, I guess it stands to reason you could end up getting led down some deep dark rabbit holes.

Wasn't the "fast&slow" thingy debunked as another piece of popscience?

Many specific studies on the matter don't replicate, I think the book preceded the replication crisis so this is to be expected, but I don't think that negates the core idea that our brain does some things on autopilot whereas other things take conscious thought which is slower. This is a useful framework to think about cognition, though any specific claims need evidence obviously.

TBH I've learned that even the best pop sci books making (IMHO) correct points tend to have poor citations - to studies that don't replicate or don't quite say what they're being cited to say - so when I see this, it's just not very much evidence one way or the other. The bar is super low.

Re: Why are there so many rationalist cults?

#637

Post by booleandilemma »

meowface wrote: post #503
quickthrowman wrote: post #489

Earlier quoted context omitted.

I only know Eliezer Yudkowsky from his Harry Potter fanfiction, most notably Harry Potter and the Methods of Rationality . Is he known publicly for some other reason?

He's considered the father of rationalism and the father of AI doomerism. He wrote this famous article in Time magazine a few years ago: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no... His book If Anyone Builds It, Everyone Dies comes out in a month: https://www.amazon.com/Anyone-Builds-Everyone-Dies-Superhuma... You can find more info here: https://en.wikipedia.org/wiki/Eliezer_Yudkowsky

You didn't mention LessWrong, the rationalist website he founded and the reason he got famous in the first place.

Is he trying to distance himself from it now?

https://en.wikipedia.org/wiki/LessWrong

Re: Why are there so many rationalist cults?

#638

Post by Nursie »

human_person wrote: post #389
MajimasEyepatch wrote: post #271

Earlier quoted context omitted.

I feel this way about some of the more extreme effective altruists. There is no room for uncertainty or recognition of the way that errors compound. - "We should focus our charitable endeavors on the problems that are most impactful, like eradicating preventable diseases in poor countries." Cool, I'm on board. - "I should do the job that makes the absolute most amount of money possible, like starting a crypto exchang…

"I should do the job that makes the absolute most amount of money possible, like starting a crypto exchange, so that I can use my vast wealth in the most effective way." Has always really bothered me because it assumes that there are no negative impacts of the work you did to get the money. If you do a million dollars worth of damage to the world and earn 100k (or a billion dollars worth of damage to earn a million d…

And not just that, but the very fact that someone considers it valid to try to accumulate billions of dollars so they can have an outsized influence on the direction of society, seems somewhat questionable.

Even with 'good' intentions, there is the implied statement that your ideas are better than everyone else's and so should be pushed like that. The whole thing is a self-satisfied ego-trip.

Re: Why are there so many rationalist cults?

#639

Post by doubleunplussed »

thedudeabides5 wrote: post #312
thedudeabides5 wrote: post #304

Purity Spirals + Cheap Talk = irrational rationalists

Eliezer Yudkowsky, shows little interest in running one. He has consistently been distant from and uninvolved in rationalist community-building efforts, from Benton House (the first rationalist group house) to today’s Lightcone Infrastructure (which hosts LessWrong, an online forum, and Lighthaven, a conference center). He surrounds himself with people who disagree with him, discourages social isolation. Ummm, EY lit…

Blocking people on Twitter doesn't necessarily imply intolerance of people who disagree with you. People often block for different reasons than disagreement.

Re: Why are there so many rationalist cults?

#640

Post by joenot443 »

quickthrowman wrote: post #489
gwbas1c wrote: post #310

Many years ago I met Eliezer Yudkowsky. He handed me a pamphlet extolling the virtues of rationality. The whole thing came across as a joke, as a parody of evangelizing. We both laughed. I glanced at it once or twice and shoved it into a bookshelf. I wish I kept it, because I never thought so much would happen around him.

I only know Eliezer Yudkowsky from his Harry Potter fanfiction, most notably Harry Potter and the Methods of Rationality . Is he known publicly for some other reason?

Do you spend much time in communities which discuss AI stuff? I feel as if he's mentioned nearly daily, positively or not, in a lot of the spaces I frequent.

I'm surprised you're unfamiliar otherwise, I figured he was a pretty well known commentator.

Post reply on HN