Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are there so many rationalist cults?

asteriskmag.com

Post reply on HN
Print view View original

661–670 of 947 posts

Re: Why are there so many rationalist cults?

#661

Post by bglazer »

doubleunplussed wrote: post #627
ar-nelson wrote: post #200

Earlier quoted context omitted.

I find Yudowsky-style rationalists morbidly fascinating in the same way as Scientologists and other cults. Probably because they seem to genuinely believe they're living in a sci-fi story. I read a lot of their stuff, probably too much, even though I find it mostly ridiculous. The biggest nonsense axiom I see in the AI-cult rationalist world is recursive self-improvement. It's the classic reason superintelligence tak…

I'm surprised not see see much pushback on your point here, so I'll provide my own. We have an existence proof for intelligence that can improve AI: humans can do this right now. Do you think AI can't reach human-level intelligence? We have an existence proof of human-level intelligence: humans. If you think AI will reach human-level intelligence then recursive self-improvement naturally follows. How could it not? Do…

Yudkowsky seems to believe in fast take off, so much so that he suggested bombing data centers. To more directly address your point, I think it’s almost certain that increasing intelligence has diminishing returns and the recursive self improvement loop will be slow. The reason for this is that collecting data is absolutely necessary and many natural processes are both slow and chaotic, meaning that learning from observation and manipulation of them will take years at least. Also lots of resources.

Regarding LLM’s I think METR is a decent metric. However you have to consider the cost of achieving each additional hour or day of task horizon. I’m open to correction here, but I would bet that the cost curves are more exponential than the improvement curves. That would be fundamentally unsustainable and point to a limitation of LLM training/architecture for reasoning and world modeling.

Basically I think the focus on recursive self improvement is not really important in the real world. The actual question is how long and how expensive the learning process is. I think the answer is that it will be long and expensive, just like our current world. No doubt having many more intelligent agents will help speed up parts of the loop but there are physical constraints you can’t get past no matter how smart you are.

Re: Why are there so many rationalist cults?

#662

Post by ipnon »

pizzadog wrote: post #445

I have a lot of experience with rationalists. What I will say is: 1) If you have a criticism about them or their stupid name or how "'all I know is that I know nothing' how smug of them to say they're truly wise," rest assured they have been self flagellating over these criticisms 100x longer than you've been aware of their group. That doesn't mean they succeeded at addressing the criticisms, of course, but I can tel…

A lot of terrible people are self-aware, well-read and ultimately concerned with doing good. All of the catastrophes of the 20th century were led by men that fit this description: Stalin, Mao, Hitler. Perhaps this is a bit hyperbolic, but the troubling belief that the Rationalists have in common with these evil men is the ironclad conviction that self-awareness, being well-read, and being concerned with good, somehow makes it impossible for one to do immoral and unethical things.

I think we don't believe in hubris in America anymore. And the most dangerous belief of the Rationalists is that the more complex and verbose your beliefs become, the more protected you become from taking actions that exceed your capability for success and benefit. In practice it is often the meek and humble who do the most good in this world, but this is not celebrated in Silicon Valley.

Re: Why are there so many rationalist cults?

#663

Post by toasterlovin »

DoctorOetker wrote: post #659
toasterlovin wrote: post #655

Earlier quoted context omitted.

> The key takeaway from the article is that if you have a group leader who cuts you off from other people, that's a red flag Even better: a social group with a lot of invented lingo is a red flag that you can see before you get isolated from your loved ones.

By this token, most scientists would be considered cultists: normal people don't have "specific tensile strength" or "Jacobian" or "Hermitian operator" etc in their vocabulary. "Must be some cult"? Edit: it seems most people don't understand what I'm pointing out. Having terminology is not the red flag. Having intricate terminology without a domain is the red flag. In science or mathematics, there are enormous amount…

> social group

Re: Why are there so many rationalist cults?

#664

Post by DoctorOetker »

toasterlovin wrote: post #663
DoctorOetker wrote: post #659

Earlier quoted context omitted.

By this token, most scientists would be considered cultists: normal people don't have "specific tensile strength" or "Jacobian" or "Hermitian operator" etc in their vocabulary. "Must be some cult"? Edit: it seems most people don't understand what I'm pointing out. Having terminology is not the red flag. Having intricate terminology without a domain is the red flag. In science or mathematics, there are enormous amount…

> social group

fine, the jargon of a "social group" of science is a red flag?

sure, theres lots of nasty side effects of how academia is run, rewarded, etc..

but thats not because of precision of language employed.

do you want scientists recycling the same words and overloading ever more meanings onto ever more ambiguous words?

Re: Why are there so many rationalist cults?

#665

Post by doubleunplussed »

bglazer wrote: post #661
doubleunplussed wrote: post #627

Earlier quoted context omitted.

I'm surprised not see see much pushback on your point here, so I'll provide my own. We have an existence proof for intelligence that can improve AI: humans can do this right now. Do you think AI can't reach human-level intelligence? We have an existence proof of human-level intelligence: humans. If you think AI will reach human-level intelligence then recursive self-improvement naturally follows. How could it not? Do…

Yudkowsky seems to believe in fast take off, so much so that he suggested bombing data centers. To more directly address your point, I think it’s almost certain that increasing intelligence has diminishing returns and the recursive self improvement loop will be slow. The reason for this is that collecting data is absolutely necessary and many natural processes are both slow and chaotic, meaning that learning from obs…

How do you reconcile e.g. AlphaGo with the idea that data is a bottleneck?

At some point learning can occur with "self-play", and I believe this is already happening with LLMs to some extent. Then you're not limited by imitating human-made data.

If learning something like software development or mathematical proofs, it is easier to verify whether a solution is correct than to come up with the solution in the first place, many domains are like this. Anything like that is amenable to learning on synthetic data or self-play like AlphaGo did.

I can understand that people who think of LLMs as human-imitation machines, limited to training on human-made data, would think they'd be capped at human-level intelligence. However I don't think that's the case, and we have at least one example of superhuman AI in one domain (Go) showing this.

Regarding cost, I'd have to look into it, but I'm under the impression costs have been up and down over time as models have grown but there have also been efficiency improvements.

I think I'd hazard a guess that end-user costs have not grown exponentially like time horizon capabilities, even though investment in training probably has. Though that's tricky to reason about because training costs are amortised and it's not obvious whether end user costs are at a loss or what profit margin for any given model.

On the fast-slow takeoff - Yud does seem to beleive in a fast takeoff yes, but it's also one of the the oldest disagreements in rationality circles, on which he disagreed with his main co-blogger on the orignal rationalist blog, Overcoming Bias, some discussion of this and more recent disagreements here [1].

[1] https://www.astralcodexten.com/p/yudkowsky-contra-christiano...

Re: Why are there so many rationalist cults?

#667

Post by [deleted] »

to11mtm wrote: post #404
xg15 wrote: post #290

Earlier quoted context omitted.

The "longtermism" idea never made sense to me: So we should sacrifice the present to save the future. Alright. But then those future descendants would also have to sacrifice their present to save their future, etc. So by that logic, there could never be a time that was not full of misery. So then why do all of that stuff?

Well, there's a balance to be had. Do the most good you can while still being able to survive the rat race. However, people are bad at that. I'll give an interesting example. Hybrid Cars. Modern proper HEVs[0] usually benefit to their owners, both by virtue of better fuel economy as well as in most cases being overall more reliable than a normal car. And, they are better on CO2 emissions and lower our oil consumption…

[deleted]

Re: Why are there so many rationalist cults?

#668

Post by toasterlovin »

ethbr1 wrote: post #634
toasterlovin wrote: post #500

Earlier quoted context omitted.

Dead reckoning is a great analogy for coming to conclusions based on reason alone. Always useful to check in with reality.

And always worth keeping an eye on the maximum possible divergence from reality you're currently at, based on how far you've reasoned from truth, and how less-than-sure each step was. Maybe you're right. But there's a non-zero chance you're also max wrong. (Which itself can be bounded, if you don't wander too far)

My preferred argument against the AI doom hypothesis is exactly this: it has 8 or so independent prerequisites with unknown probabilities. Since you multiply the probabilities of each prerequisite to get the overall probability, you end up with a relatively low overall probability even when the probability of each prerequisite is relatively high, and if just a few of the prerequisites have small probabilities, the overall probability basically can’t be anything other than very small.

Given this structure to the problem, if you find yourself espousing a p(doom) of 80%, you’re probably not thinking about the issue properly. If in 10 years some of those prerequisites have turned out to be true, then you can start getting worried and be justified about it. But from where we are now there’s just no way.

Re: Why are there so many rationalist cults?

#669

Post by joe_the_user »

godelski wrote: post #536
joe_the_user wrote: post #507

Earlier quoted context omitted.

IME most people aren't very good at building axioms. It seems you implying that some people are good building good axiom systems for the real world. I disagree. There are a few situations in the world where you have generalities so close to complete that you can use simple logic on them. But for the messy parts of the real world, there simply is not set of logical claims which can provide anything like certainty no m…

I don't even know what you're arguing. > you implying that some people are good building good axiom systems How do you go from "most people aren't very good" to "this implies some people are really good"? First, that is just a really weird interpretation of how people speak (btw, "you're" not "you" ;) because this is nicer and going to be received better than "making axioms is hard and people are shit at it." Second,…

I think I misunderstood that you talking of axiomatization of mathematical or related systems.

The original discussion are about the formulation of "axioms" about the real world ("the bus always X minutes late" or more elaborate stuff). I suppose I should have considered with your username, you would have consider the statement in terms of the formulation of mathematical axioms.

But still, I misunderstood you and you misunderstood me.

Re: Why are there so many rationalist cults?

#670

Post by toasterlovin »

DoctorOetker wrote: post #664
toasterlovin wrote: post #663

Earlier quoted context omitted.

> social group

fine, the jargon of a "social group" of science is a red flag? sure, theres lots of nasty side effects of how academia is run, rewarded, etc.. but thats not because of precision of language employed. do you want scientists recycling the same words and overloading ever more meanings onto ever more ambiguous words?

I don’t think we disagree. I’m not taking issue with scientists having jargon, which I agree is good and necessary (though I think the less analytical academic disciplines, not being rooted in fact, have come to bear many similarities to state-backed religions; and I think they use jargon accordingly). I’m pointing out that I specifically intended to exclude professionals by scoping my statement to “social groups”. Primarily I had in mind religion, politics, certain social media sites, and whatever you want to call movements like capital R Rationality (I have personally duck typed it as a religion).
Post reply on HN