Bayes's Theorem: What's the Big Deal?

Did you account for the possibility you're dreaming, or in a simulation? Or in a simulation of a simulation and so on? Yes, that's the curse of real numbers. In practice you set a limit of precision required, and a threshold of evidence you need to tell you that the precision should be more precise. You can get quite far with Newtonian physics without considering the corrections of general relativity, you can get quite far in engineering by assuming linearity by only expanding the Taylor transform of a non-linear expression by a bit instead of infinitely, and you can do a lot by assuming probabilistic independence of your card counting from what's going on on Pluto right now.

"1. Bond posits Yudkowski thinks that racism is mostly due to genetic difference, and not about the deliberate, mostly political disenfranchisement of minorities."

You don't need to take the author's word for it. Look at the article itself.

The article is not about the question of whether race affects intelligence. It's saying that that question is much less important than the fact that individuals have different IQ levels regardless of race. At least in the context of this article, which is asking whether "God is fair".

To make it crystal clear - Yudkowsky is saying:

1. Forget race. There are clear, obvious, and gigantic genetic differences in individuals which cause differences in intelligence.

2. This is "unjust".

3. We should be upset about this, and try to fix it.

That's all he's saying in that article - it is not at all racist.

If you think otherwise, please - quote the relevant part of the article and explain yourself.

I've been saying t forever. Thanks for putting it so succinctly. LessWrong is a cult of people who want to be smart and they've essentially found a community in which certain assumptions and hypothetical scenarios combined with mathematical concepts make them think they've found the answer to everything in the Universe.

They're no better than any other cult in my book. The problem is that it's only going to get worse with the advances in AI that are going on. Yudkowski has managed to convince some wealthy people to fund his so called research and we have OpenAI operating in the same waters which somehow gives LW people more legitimacy.

So I'm a LessWronger and know a bit about the "movement", and think you are misunderstanding what "LessWrongers think". Obviously not all LessWrongers think the same thing at all, but I'm talking about the average position of the people who believe AI safety should be worked towards.

I'd love to explain the basic position, and tell me where you disagree with it. This is the basic position:

1. Intelligence can be created, because there is nothing "special"/"magical" about humans, and our intelligence was eventually created.

2. At some point, humanity will create an "artificial general intelligence". (Since we'll just keep improving science and technology, and there's no fundamental reason why this won't eventually allow us to create an intelligence.

3. "Artificial general intelligence" basically means a machine that is capable of achieving its goals, where the goals and methods it uses to achieve them are general. I.e. not "is able to play chess really well", but rather "is able to e.g. cure cancer".

4. For various reasons, once we have an artificial intelligence, it will likely become much smarter than us. (There are many reasons and debates about this, but let's just assume that since it's a computer, we can run it much faster than a human. If you dispute this point, we can talk about it more).

5. Something being much more intelligent than us means that, in effect, it has almost absolute power over what happens in the world (like we are basically all-powerful from the vantage point of monkeys, and their fate is totally in our hands).

6. (This is, I believe, the main point): Something being "intelligent" in the sense we're talking about doesn't say anything about what its goals are, or about how its mind works. We're used to everything that's intelligent being a human being, therefore the way our mind works is basically the same across every human. With an artificial intelligence, it will work completely differently from our mind. So if we "tell it" something like "cure cancer", it won't have our intuition and background knowledge to understand that we mean "but don't turn half the world into a giant computer in order to cure it".

7. Combine the two points above, and you get the large idea - whatever the goals of the AI will be, it will achieve them. Its goals won't, by default, be ones that are good for humanity, if only because we have no idea how to program our "value system" into a computer.

8. Therefore, we need to start working on making sure that when AI does come, it's safe. Even if we create an AI, the "extra" problem of making it safe is both hard, and we have absolutely no idea how to do it right now. We have no idea how long AI will take, or how long figuring out safety will take, but since this is a humanity-threatening problem, we should devote at least some resources to working on it right now.

That's it, that's the basic idea. I'd love to hear which part you disagree with. I totally understand that not everyone will agree on some of the final details like, e.g., how many resources we should effectively devote right now (you might even claim it's 0 because anything we do now won't be useful).

But I think the overall reasoning is sound, and would love to hear an intelligent disagreement.

Then you should take the Bayesian side, because Bayesians look at the data first, and they take their data as given rather than taking a null hypothesis as given. They don't just blindly go off and run a test (which assumes a particular prior implicitly that may be wildly inappropriate) and see what it says about the likelihood of their already observed data being generated by the test's assumed data generator.