Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are there so many rationalist cults?

asteriskmag.com

Post reply on HN
Print view View original

791–800 of 947 posts

Re: Why are there so many rationalist cults?

#791

Post by throwanem »

aa-jv wrote: post #776
throwanem wrote: post #757

Earlier quoted context omitted.

Even a marginal familiarity with the history of Scientology is an excellent curative for the idea that you can think yourself into superpowers, or that you should ever trust anyone who promises to teach you how. The consequences of ignorance on this score are all drearily predictable to anyone with a modicum of both good sense and world knowledge, which is why they've come as such a surprise to Yudkowsky.

You can say all of this of drug-oriented seekers of superpowers, too. Trust the SSRI cult much? It just seems to be a human condition that whenever anyone tries to find a way to improve themselves and others, there will always be other human beings who attempt to prevent that from occurring. I don't think this is a cult thing - I think its a culture thing. Humans have an innate desire to oppress others in their envir…

The phrase you failed to find is "crab-bucket thinking," but the one you really should have paid attention to this morning is "take with food."

Re: Why are there so many rationalist cults?

#792

Post by NoGravitas »

vehemenz wrote: post #344

I get the impression that these people desperately want to study philosophy but for some reason can't be bothered to get formal training because it would be too humbling for them. I call it "small fishbowl syndrome," but maybe there's a better term for it.

I think a larger part of it is the assumption that an education in humanities is useless - that if you have an education (even self-education) in STEM, and are "smart", you will automatically do better than the three thousand year conversation that comprises the humanities.

Re: Why are there so many rationalist cults?

#793

Post by lostmsu »

chatmasta wrote: post #745
lmm wrote: post #566

Earlier quoted context omitted.

Well, maybe. It seems at least adjacent to the stuff that's been making a lot of people rich lately.

It’s also adjacent to sociopathy, which is more likely the driving factor behind that wealth generation.

Did not know "sociopathy" is a precursor to wealth generation. I guess I'm in luck! Any day now!

Re: Why are there so many rationalist cults?

#794

Post by bglazer »

doubleunplussed wrote: post #665
bglazer wrote: post #661

Earlier quoted context omitted.

Yudkowsky seems to believe in fast take off, so much so that he suggested bombing data centers. To more directly address your point, I think it’s almost certain that increasing intelligence has diminishing returns and the recursive self improvement loop will be slow. The reason for this is that collecting data is absolutely necessary and many natural processes are both slow and chaotic, meaning that learning from obs…

How do you reconcile e.g. AlphaGo with the idea that data is a bottleneck? At some point learning can occur with "self-play", and I believe this is already happening with LLMs to some extent. Then you're not limited by imitating human-made data. If learning something like software development or mathematical proofs, it is easier to verify whether a solution is correct than to come up with the solution in the first pl…

AlphaGo showed that RL+search+self play works really well if you have an easy to verify reward and millions of iterations. Math partially falls into this category via automated proof checkers like Lean. So, that’s where I would put the highest likelihood of things getting weird really quickly. It’s worth noting that this hasn’t happened yet, and I’m not sure why. It seems like this recipe should already be yielding results in terms of new mathematics, but it isn’t yet.

That said, nearly every other task in the world is not easily verified, including things we really care about. How do you know if an AI is superhuman at designing fusion reactors? The most important step there is building a fusion reactor.

I think a better reference point than AlphaGo is AlphaFold. Deepmind found some really clever algorithmic improvements, but they didn’t know whether they actually worked until the CASP competition. CASP evaluated their model on new Xray crystal structures of proteins. Needless to say getting Xray protein structures is a difficult and complex process. Also, they trained AlphaFold on thousands of existing structures that were accumulated over decades and required millenia of graduate-student-hours hours to find. It’s worth noting that we have very good theories for all the basic physics underlying protein folding but none of the physics based methods work. We had to rely on painstakingly collected data to learn the emergent phenomena that govern folding. I suspect that this will be the case for many other tasks.

Re: Why are there so many rationalist cults?

#795

Post by ruszki »

stickfigure wrote: post #674
pavlov wrote: post #398

A very interesting read. My idea of these self-proclaimed rationalists was fifteen years out of date. I thought they’re people who write wordy fan fiction, but turns out they’ve reached the point of having subgroups that kill people and exorcise demons. This must be how people who had read one Hubbard pulp novel in the 1950s felt decades later when they find out he’s running a full-blown religion now. The article see…

You're falling into some sort of fallacy; maybe a better rationalist than I could name it. The "they" you are describing is a large body of disparate people spread around the world. We're reading an article that focuses on a few dysfunctional subgroups. They are interesting because they are so dysfunctional and rare. Or put it this way: Name one -ism that _doesn't_ have sub/splinter groups that kill people. Even Paci…

The level of dysfunction which is described in the article is really rare. But dysfunction, the kind of which we talk about, is not really that rare, I would even say that quite common, in self proclaimed rationalist groups. They don’t kill people - at least directly - but they definitely not what they claim to be: rational. They use rational tools, more than others, but they are not more rational than others, they simply use these tools to prove their irrationality.

I touch rationalists only with a pole recently, because they are not smarter than others, but they just think that, and on the surface level they seem so. They praise Julia Galef, then ignore everything what she said. Even Galef invited people who were full blown racists, just it seemed that they were all right because they knew whom they talked with, and they couldn’t bullshit. They tried to argue why their racism is rational, but you couldn’t tell from the interviews. They flat out lies all the time on every other platforms. So at the end she just gave platform for covered racism.

Re: Why are there so many rationalist cults?

#796

Post by lostmsu »

sien wrote: post #648
kayodelycaon wrote: post #582

Earlier quoted context omitted.

I took a few philosophy classes. I found it incredibly valuable in identifying assumptions and testing them. Being Christian, it helped me understand what I believe and why. It made faith a deliberate, reasoned choice. And, of course, there are many rational reasons for people to have very different opinions when it comes to religion and deities. Being bipolar might give me an interesting perspective. Everything I’ve…

Rationalists have not read or understood David Hume. You cannot work out what out to be from what is. To want to be alive is irrational. Nietzsche and the Existentialists understood that. Arguably religions too.

> To want to be alive is irrational.

This is some philosophy bullshit. Taking "rational" to be ~ "logical choice" the truthness of this statement depends on the assumed axioms, and given you didn't list them this statement is clearly false under rather simple "sum of all life is the value" system until that system is proven self-contradictory. Which I doubt you or the famous mouths you mentioned did at any point, because it probably is not.

Re: Why are there so many rationalist cults?

#797

Post by lostmsu »

wredcoll wrote: post #527
meowface wrote: post #503

Earlier quoted context omitted.

He's considered the father of rationalism and the father of AI doomerism. He wrote this famous article in Time magazine a few years ago: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no... His book If Anyone Builds It, Everyone Dies comes out in a month: https://www.amazon.com/Anyone-Builds-Everyone-Dies-Superhuma... You can find more info here: https://en.wikipedia.org/wiki/Eliezer_Yudkowsky

> He's considered the father of rationalism [citation needed] Even for this weird cult that is trying to appropriate the word, would they really consider him the father of redefining the word?

Not of the philosophy (I'd attribute that to Popper), but of the modern movement.

Re: Why are there so many rationalist cults?

#798

Post by lostmsu »

Jtsummers wrote: post #387

> Many of them also expect that, without heroic effort, AGI development will lead to human extinction. > These beliefs can make it difficult to care about much of anything else: what good is it to be a nurse or a notary or a novelist, if humanity is about to go extinct? Replace AGI causing extinction with the Rapture and you get a lot of US Christian fundamentalists. They often reject addressing problems in the envir…

And if you replace water with vodka you get a bunch of alcoholics.

Re: Why are there so many rationalist cults?

#799

Post by quickthrowman »

joenot443 wrote: post #640
quickthrowman wrote: post #489

Earlier quoted context omitted.

I only know Eliezer Yudkowsky from his Harry Potter fanfiction, most notably Harry Potter and the Methods of Rationality . Is he known publicly for some other reason?

Do you spend much time in communities which discuss AI stuff? I feel as if he's mentioned nearly daily, positively or not, in a lot of the spaces I frequent. I'm surprised you're unfamiliar otherwise, I figured he was a pretty well known commentator.

I’m just joking around, I’m well aware of who he is.

I never miss an opportunity to highlight his magnum opus, it delights me that so many people take him seriously when he’s at best a second-rate fanfiction author who thinks robots from the future are going to torture copies of him forever or some such nonsense.

Re: Why are there so many rationalist cults?

#800

Post by naasking »

spott wrote: post #199
naasking wrote: post #114

Earlier quoted context omitted.

> We're normally data-limited. This is a common sentiment but is probably not entirely true. A great example is cosmology. Yes, more data would make some work easier, but astrophysicists and cosmologists have shown that you can gather and combine existing data and look at it in novel new ways to produce unexpected results, like place bounds that can include/exclude various theories. I think a philosophy that encourag…

This depends on what you are trying to figure out. If you are talking about cosmology? Yea, you can look at existing data in new ways, cause you probably have enough data to do that safely. If you are looking at human psychology? Looking at existing data in new ways is essentially p-hacking. And you probably won’t ever have enough data to define a “universal theory of the human mind”.

> If you are looking at human psychology? Looking at existing data in new ways is essentially p-hacking.

I agree that the type of analysis is important, as is the type and quality of the data you're analyzing. You can p-hack in cosmology too, but it's not a quality argument there either.

> And you probably won’t ever have enough data to define a “universal theory of the human mind”.

I think you're underestimating human ability to generalize principles from even small amounts of data [1]. Regardless, my point was more specifically that we could use existing data to generate constraints to exclude certain theories of mind, which has definitely happened.

[1] https://en.wikipedia.org/wiki/Predictive_coding

Post reply on HN