Superintelligence: The Idea That Eats Smart People (2016)

I didn't miss the point. I showed other more usable (in my opinion) point of views. I think that definition is so narrow that it starts being absurd.

> It's absolutely nothing to do with what's good to do, only brute facts of power.

This sentence reads like: "if we narrow our view this much, this makes sense". I agree that it makes sense under the condition that we narrow the view of issue. It's valid in this small context (a film about controlling one thing).

> And indeed, toddlers and psychopaths have a scarily good understanding of what power is.

I disagree completely with this sentence. They are good at controlling in certain situations. They don't understand it. If you want to understand it, there is a lot of information about controlling, whole fields of knowledge that people spend many years on studying. As for psychopaths, they are very predictable and controllable when you understand control theory and how psychopaths operate. There are courses on this single topic by people who need to do it to prevent tragedies (police negotiators) and they are not that complicated.

> The pressing ethical questions in machine learning are not about machines becoming self-aware and taking over the world, but about how people can exploit other people, or through carelessness introduce immoral behavior into automated systems.

I think the main point still stands. (And there have been some pretty prescient depictions, e.g. Marvin the paranoid android was a pretty spot-on prediction of Bing. Or perhaps the fact that Marvin was in the training set was what led to Bing?)

Why is this a "fun fact", what is this reddit-tier comment contributing?

Paying someone to physically force the cat in the carrier is missing the point. Compared to the cat, Hawking is a super intelligence. But all that intelligence isn't able to get him to speak cat and give the cat a reason to get inside the carrier of its own volition.

> In particular, the concept of the Roko's Basilisk is some rehash of the Pascal's Wager.

Oh boy couldn't agree more. The whole Basilisk drama really caused me to rethink the idea that a group of Rationalists were in practice, were in fact particularly rational. CF The Atheists vs someone that happens to be atheist. To their credit, they also acknowledged this somewhat and hence the CFAR crowd.

The word "rehash" also resonates. Like Google has a tendency to use PHD's to reinvent everything over and make up new words and terms for the same concepts.

Having spent a little too much time in the early 2010's at Bay Area LW meetups, the more fantastical LARP of fanfic, and religious mythology often felt a bit at odds. There was the aspect of a charismatic, autodidact leader obsessed with a certain J.K. Rowling IP and the kinky stuff..Don't get me wrong, I still have fond memories of this time overall :)

From my perspective, a core issue seemed to be no-one seemed to particularly motivated in defining what it meant to be rational, aside from some loose segmentation around instrumental vs epistemic rationality. (ie practice vs theory). And because of this it almost had a faith vibe to the scene. Like "trust me bro" this is "super high brow nerd stuff" and on your third helping of "The Sequences" all these formulas and shiny new words will all make sense what and it will be clear why we’re doing these meetups. (It totally wasn't anything to do with mental masturbation and high-iq crowd bonding and feeling good ;)

When I was into Christian apologetics (C.S. Lewis etc) as in my first year of CS & philosophy of science in college, there was a similar thing.. after a year of seeking out the scientific, and logical explanations for all the religious dogma I was indoctrinated into growing up. In the end, it pretty much reduced to "just have faith". This was after exhausting the "well you're not an expert on Christian theology, so you cant have a solid argument around the nature and existence of God because you need to study more" counter. This despite reading and studying the Bible at length.

For example, right now. Is it rational to be typing this up on HN, when I have other more important goals to do? OTOH reminiscing on the past and connecting with a single serving online friend OP, gives a bit of a dopamine hit. And maybe sharing resonates with others and increases happiness in the world? (or not if anyone still LW reads this and maybe feels a different type of way)

So that’s community right and good (in the sense it's aligning with goals)? But then its driven by emotions, so that kinda is not supposed to be rational. Is it rational to observe one's own mental states and take action? Turtles all the way down!