Live data from Hacker News

Guess I'm a rationalist now

scottaaronson.blog

431–440 of 1001 posts

Re: Guess I'm a rationalist now

#431
post #355

Earlier quoted context omitted.

I read the whole tree of responses under this comment and I could only convince myself that when people have no arguments they try to make you look bad. Most of criticisms are just "But they think they are better than us !" and the rest is "But sometimes they are wrong !" I don't know about the community and couldn't care less but their writings have brought me some almost life saving fresh air in how to think about…

Here's a theory of what's happening, both with you here in this comment section and with the rationalists in general. Humans are generally better at perceiving threats than they are at putting those threats into words. When something seems "dangerous" abstractly, they will come up with words for why---but those words don't necessarily reflect the actual threat, because the threat might be hard to describe. Neverthele…

> Ironically paperclip-maximizing-robots are exactly the thing that the rationalists are so worried about. They are a group of people who missed, and then disavowed, and now advocate disavowing, this "personal" morality, and unsurprisingly they view the world in a lens that doesn't include personality morality, which means mostly being worried about problems of the same sort. But it provokes a strong negative reaction from everyone who thinks about the world in terms of that personal duty to safety.

I head not read any rationalist writing in a long time (and I didn't know about Scott's proximity), but the whole time I was reqding the article I was thinking the same thing you just wrote... "why are they afraid of AI, i.e. the ultimate rationalist taking over the world", maybe something deep inside of them has the same reaction to their own theories as you so eloquently put above.

Re: Guess I'm a rationalist now

#432
post #375

Earlier quoted context omitted.

Not meaning to be too direct, but you are misinterpreting a lot about rationalists. In my view, rationalists are often "Bayesian" in that they are constantly looking for updates to their model. Consider that the default approach for most humans is to believe a variety of things and to feel indignant if someone holds differing views (the adage never discuss religion or politics ). If one adopts the perspective that th…

not to be too cynical here, but I would say that the most-apt description of the rationalists is that they are people who would say they are constantly looking for updates to their models. But that they are not necessarily doing it appreciably more than anyone else is. They will do it freely on unimportant things---they tend to be smart people who view the world intellectually and so they are free to toss or keep fac…

You're generalizing that all self-proclaimed rationalists are hypocrites and heavily biased? I mean, regardless of whether or not that is true, what is the point of making such a broad generalization? Strange!

Re: Guess I'm a rationalist now

#433

Earlier quoted context omitted.

Scott Aaronson is probably the most publicly-vulnerable academic I've ever found, at least outside of authors who write memoirs about childhood trauma. I think a lot of other prominent rationalists also put a lot of vulnerability out there.

He didn't take the rationalist label until today. Him doing so might help their image.

Right, but him doing so is the very context of this discussion, which is why I mentioned him in particular. Scott Alexander is more well-known as a rationalist and also (IMO) displays a lot of vulnerability in his writing.

Re: Guess I'm a rationalist now

#434
post #137

Earlier quoted context omitted.

I think you’re mostly right. But also that it isn’t what the Yudkowsky is (was?) trying to do with it. I think he’s trying to distill useful tools which increase baseline rationality. Religions have this. It’s what the original philosophers are missing. (At least as taught, happy to hear counter examples)

I think I'd rather subscribe to an actual religion, than listen to these weird rationalist types of people who seem to have solved the problem that is "everything". At least there is some interesting history to learn about with religion

I would too if I could but organized religions make me uncomfortable even though I admire parts of them. Similar to my admiration you don’t need to like the rationality types or believe in their program to find one or more of their tools useful.

I’ll also respond to the silent downvoters apparent disagreement. CFAR holds workshops and a summer camp for teaching rationality tools. In HPMoR Harry discusses the way he thinks and why. I read it as more of a way to discuss EY’s views in fiction as much as fiction itself.

Re: Guess I'm a rationalist now

#435

Earlier quoted context omitted.

The overlap between the Effective Altruism community and the Rationalist community is extremely high. They’re largely the same people. Effective Altruism gained a lot of early attention on LessWrong, and the pessimistic focus on AI existential risk largely stems from an EA desire to avoid “temporal-discounting” bias. The reasoning is something like: if you accept that future people count just as much as current peopl…

> Therefore, nothing can produce greater positive expected value than preventing existential risks—so working to reduce these risks becomes the highest priority. Incidentally, the flaw in this theory is in thinking you understand what all the existential risks are . Suppose you clock "malicious AI" as a huge risk and then hamper AI, but it turns out the bigger risk is not doing space exploration, which AI would have…

> Incidentally, the flaw in this theory is in thinking you understand what all the existential risks are.

Rationalist community understands that very well. They even know how to put bounds on the unknowns and their own lack of information.

> The thing evolution teaches us is that diversity is a group survival trait. Anybody insisting "nobody anywhere should do X" is more likely to cause an ELE than prevent one.

Right. Good thing they'd agree with you 100% on this.

Re: Guess I'm a rationalist now

#436
post #375

Earlier quoted context omitted.

not to be too cynical here, but I would say that the most-apt description of the rationalists is that they are people who would say they are constantly looking for updates to their models. But that they are not necessarily doing it appreciably more than anyone else is. They will do it freely on unimportant things---they tend to be smart people who view the world intellectually and so they are free to toss or keep fac…

You're generalizing that all self-proclaimed rationalists are hypocrites and heavily biased? I mean, regardless of whether or not that is true, what is the point of making such a broad generalization? Strange!

um.... because I think it's true and relevant? I'm describing a pattern I have observed over many years. It is of course my opinion (and are not a universal statement, just what I believe to be a common phenomenon).

Re: Guess I'm a rationalist now

#437

Earlier quoted context omitted.

They remind me of the "Effective Altruism" crowd who get completely wound up in these hypothetical logical thought exercises and end up coming to insane conclusions that they feel trapped in because they got there using pure logic. Not realizing that their initial conditions were highly artificial so any conclusion they reach is only of academic value. There is a term for this. "Getting stuck up your own butt." It wo…

The overlap between the Effective Altruism community and the Rationalist community is extremely high. They’re largely the same people. Effective Altruism gained a lot of early attention on LessWrong, and the pessimistic focus on AI existential risk largely stems from an EA desire to avoid “temporal-discounting” bias. The reasoning is something like: if you accept that future people count just as much as current peopl…

In what sense are people in those communities "quite smart"? Stupid is as stupid does. There are plenty of people who get good grades and score highly on standardized tests, but are in fact nothing but pontificating blowhards and useless wankers.

Re: Guess I'm a rationalist now

#438

Earlier quoted context omitted.

> They remind me of the "Effective Altruism" crowd who get completely wound up in these hypothetical logical thought exercises and end up coming to insane conclusions that they feel trapped in because they got there using pure logic. Not realizing that their initial conditions were highly artificial so any conclusion they reach is only of academic value. Do you have examples of that? I have a different perception, mo…

As Adam Becker shows in his book, EAs started out being reasonable "give to charity as much as you can, and research which charities do the most good" but have gotten into absurdities like "it is more important to fund rockets than help starving people or prevent malaria because maybe an asteroid will hit the Earth, killing everyone, starving or not".

It's also not a very big leap from "My purpose is to do whatever is the greatest good" to "It doesn't matter if I hurt people as long as the overall net result is good (by some arbitrary standard)"

Re: Guess I'm a rationalist now

#439

Earlier quoted context omitted.

> Therefore, nothing can produce greater positive expected value than preventing existential risks—so working to reduce these risks becomes the highest priority. Incidentally, the flaw in this theory is in thinking you understand what all the existential risks are . Suppose you clock "malicious AI" as a huge risk and then hamper AI, but it turns out the bigger risk is not doing space exploration, which AI would have…

> Incidentally, the flaw in this theory is in thinking you understand what all the existential risks are. Rationalist community understands that very well. They even know how to put bounds on the unknowns and their own lack of information. > The thing evolution teaches us is that diversity is a group survival trait. Anybody insisting "nobody anywhere should do X" is more likely to cause an ELE than prevent one. Right…

The nature of unknowns is that you don't know them.

What's the probability of AI singularity? It has never happened before so you have no priors and any number you assign will be pure speculation.

Re: Guess I'm a rationalist now

#440
post #355

Earlier quoted context omitted.

I read the whole tree of responses under this comment and I could only convince myself that when people have no arguments they try to make you look bad. Most of criticisms are just "But they think they are better than us !" and the rest is "But sometimes they are wrong !" I don't know about the community and couldn't care less but their writings have brought me some almost life saving fresh air in how to think about…

Here's a theory of what's happening, both with you here in this comment section and with the rationalists in general. Humans are generally better at perceiving threats than they are at putting those threats into words. When something seems "dangerous" abstractly, they will come up with words for why---but those words don't necessarily reflect the actual threat, because the threat might be hard to describe. Neverthele…

> The explanation is something like "we extrapolate from the way that rationalists think and realize that their philosophy leads to dangerous conclusions."

I really like the depth of analysis in your comment, but I think there's one important element missing, which is that this is not an individual decision but a group heuristic to which individuals are then sensitized. Individuals don't typically go so far as to (consciously or unconsciously) extrapolate others' logic forward to decide that it's dangerous. Instead, people get creeped out when other people don't adhere to social patterns and principles that are normalized as safe in their culture, because the consequences are unknown and therefore potentially dangerous; or when they do adhere to patterns that are culturally believed to be dangerous. This can be used successfully to identify things that are really dangerous, but also has a high false positive rate (people with disabilities, gender identities, or physical characteristics that are not common or accepted within the beholder's culture can all trigger this, despite not posing any immediate/inherent threat) as well as a high false negative rate (many serial killers are noted to have been very charismatic, because they put effort into studying how to behave to not trigger this instinct). When we speak of something being normalized, we're talking about it becoming accepted by the mainstream so that it no longer triggers the ‘creepy’ response in the majority of individuals. As far as I can tell, the social conservative basically believes that the set of normalized things has been carefully evolved over many generations, and therefore should be maintained (or at least modified only very cautiously) even if we don't understand why they are as they are, while the social liberal believes that we the current generation are capable of making informed judgements about which things are and aren't harmful to a degree that we can (and therefore should) continuously iterate on that set to approach an ideal goal state in which it contains only things that are factually known to be harmful.

As an interesting aside, the ‘creepy’ emotion, (at least IIRC in women) is triggered not by obviously dangerous situations but by ambiguously dangerous situations, i.e. ones that don't obviously match the pattern of known safe or unsafe situations.

> Sometimes people don't or can't practice this protection for various reasons, and that's fine; it's a local problem that can be solved locally. But it's very insidious to turn around and justify not practicing it: "actually it's better not to behave morally; it's better to allocate resources to people far away; it's better to dedicate ourselves to fighting nebulous threats like AI safety or other X-risks instead of our neighbors".

The problem with the ‘us before them’ approach is that if two neighbourhoods each prioritize their local neighbourhood over the remote neighbourhood and compete (or go to war) to better their own neighbourhood at the cost of the other, generally both neighbourhoods are left worse off than they started, at least in the short term: both groups trying to make locally optimal choices leads (without further constraints) to globally highly suboptimal outcomes. In recognition of this a lot of people, not just capital-R Rationalists, now believe that at least in the abstract we should really be trying to optimize for global outcomes.

Whether anybody realistically has the computational ability to do so effectively is a different question, of course. Certainly I personally think the future-discounting ‘bias’ is a heuristic used to acknowledge the inherent uncertainty of any future outcome we might be trying to assign moral weight to, and should be accorded some respect. Perhaps you can make the same argument for the locality bias, but I guess that Rationalists (generally) either believe that you can't, or at least have a moral duty to optimize for the largest scope your computational power allows.

Post reply on HN