Why are there so many rationalist cults?

Humans have a lot more going on than just an intelligence brain. The two big ones are: bodies, with which to richly interact with reality, and emotions/desire, which drive our choices. The one that I don't think gets enough attention in this discussion is the body. The body is critical to our ability to interact with the environment, and therefore learn about it. How does an AI do this without a body? We don't have any kind of machine that comes close to the level of control, feedback, and adaptability that a human body offers. That seems very far away. I don't think that an AI can just "improve itself" without being able to interact with the world in many ways and experiment. How does it find new ideas? How does it test its ideas? How does it test its abilities? It needs an extremely rich interface with the physical world, that external feedback is necessary for improvement. That requirement would put the prospect of a recursive self-improving AI much further into the future than many rationalists believe.

And of course, the "singularity" scenario does not only make "recursive self-improvement" the only assumption, it assumes exponential recursive self-improvement all the way to superintelligence. This is highly speculative. It's just as possible that the curve is more logarithmic, sinusoid, or linear. The reason to believe that fully exponential self-improvement is the likely scenario, based on curve of some metric now that hasn't existed for very long, does not seem solid enough to justify a strong belief. It is just as easy to imagine that intelligence gains get harder and harder as intelligence increases. We see many things that are exponential for a time, and then they aren't anymore, and basing big decisions on "this curve will be exponential all the way" because we're seeing exponential progress now, at the very early stages, does not seem sound.

Humans have human-level intelligence, but we are very far away from understanding our own brain such that we can modify it to increase our capacity for intelligence (to any degree significant enough to be comparable to recursive self-improvement). We have to improve the intelligence of humanity the hard way: spend time in the world, see what works, the smart humans make more smart humans (as do the dumb humans, which often slows the progress of the smart humans). The time spent in the world, observing and interacting with it, is crucial to this process. I don't doubt that machines could do this process faster than humans, but I don't think it's at all clear that they could do so, say, 10,000x faster. A design needs time in the world to see how it fares in order to gauge its success. You don't get to escape this until you have a perfect simulation of reality, which if it is possible at all is likely not possible until the AI is already superintelligent.

Presumably a superintelligent AI has a complete understanding of biology - how does it do that without spending time observing the results of biological experiments and iterating on them? Extrapolate that to the many other complex phenomena that exist in the physical world. This is one of the reasons that our understanding of computers has increased so much faster than our understanding of many physical sciences: to understand a complex system that we didn't create and don't have a perfect model of, we must do lots of physical experiments, and those experiments take time.

The crucial assumption that the AI singularity assumption relies on is that once intelligence hits a certain threshold, it can gaze at itself and self-improve to the top very quickly. I think this is fundamentally flawed, as we exist in a physical reality that underlies everything and defines what intelligence is. Interaction and experimentation with reality is necessary for the feedback loop of increasing intelligence, and I think this both severely limits how short that feedback loop can be, and makes the bar for an entity that can recursively self-improve itself much higher, as it needs a physical embodiment far more complex and autonomous than any robot we've managed to make.

I agree, and that's almost exactly why I replied to your statement that anyone who saw it differently than you did was "just following a religion" (to slightly paraphrase). They aren't, they just have a different perspective on the situation and have just made different calculations regarding the risk/reward ratio.

> Ignoring or disregarding Maslow's Hierarchy of Needs is a sure way to be surprised and rejected by the people.

>I'm looking medium to long term, the same scale that I think needs to be considered for the planet.

I don't think they ARE ignoring Maslow's hierarchy. It seems to me, that they just see the environmental destruction as being a more immediate concern than you do. You seem to have a "we'll fix it when it's more convenient" stance. That doesn't work for the folks who believe we'll all be starving within a decade or less, or who believe that it will NEVER be more convenient. To them this is near the top of the hierarchy.

At the end of the day, I'm very much on your side of the argument. I think we do have some time to sort it out, and I suspect that we will eventually make significant progress towards those goals (despite modern Republicans Ostrich based approach to risk). However, I understand why other people disagree, and I respect that. There's even some science that agrees with the "sky is falling" crowd. It's certainly not a totally irrational stance.

https://www.theguardian.com/world/2021/oct/26/the-great-cove...

"The psychology of masks: why have so many people stopped covering their faces? (This article is more than 3 years old)

In England, masks are expected and recommended in crowded and enclosed spaces – but not legally required. Many have abandoned them altogether. What would convince everyone to put them back on?"

That story opens with a vox pop from Dave The Normal Guy:

Dave stopped wearing his face mask “the second I didn’t have to. I grudgingly wore it, because it was the right thing to do and because it was mandatory,” says the teacher from East Sussex. “But I felt, and still do, that the reason we were told to wear masks was to make scared people feel less scared.” He didn’t feel awkward abandoning his mask, he says, as “hardly anybody bothers”, but he will put one on when visiting the vet, pharmacist or doctor, because he knows they want him to. “I feel it’s the respectful thing to do, but it’s a bit of theatre.”

Dave was 100% correct.

... all of them? Or rather, the body as a whole. Pointing at one set of faculty at one university 20 years ago and using that as the sole point to say "modern philosophy doesn't seem that useful", is just not useful.