Intelligence Is Not the Main Bottleneck

I suggest people greatly underestimate how localized expertise is. How very rapidly someone's expertise deteriorates as you move away from some focus, from the usually-very-narrow region where that expertise is externally nurtured.

A snark characterization of that region being, the issues where someone badly fears getting something wrong, and by that being embarrassed in front of the people whose opinion they much care about. One can have what superficially look like intellectual communities (picture DC thinktanks), which largely lack this property.

You can have someone well respected in their "real" subfield of biology, while being basically nutter in another subfield, which they care and think about, but aren't professionally embedded.

Ask a quantum chemist a protein chemistry question, and don't be surprised by a grad student answer. Ask "what causes the seasons?", and perhaps high-school or middle-school. Why would it be otherwise?

Even within focus, a bunch of research community gossip is discount functions. "Don't believe his observations using - he sees what he wants to see". "She always looking for things to be ."

For a recent impactful example, someone can be a successful CEO, notable engineering manager, and unusually good at thinking from first principles... while simultaneously, on other topics, being so batshit-crazy nutter that closely-held important views are 5+ orders of magnitude disconnected from reality. Because on those topics, their external community is nutter. And instead of finding this juxtaposition unsurprising, commenters twist themselves up trying to level that curve.

It's the shitty textbook publisher fallacy of "We'll find ourselves an outstanding professor of chemistry, and he'll write us an outstanding chemistry textbook". That's just not how this works.

> monoculture

Which brings us to groupthink. Expertise is a community-associated property.

In foreign affairs for example, the investment community is good at noticing and correctly evaluating some aspects of reality but not others, and similarly for diplomatic community, development, business, political, etc. Each has a partial view with systemic flaws.

I suggest groupthink is a critically important aspect of most human behavior. And yet somehow, we rarely manage to discuss this, and are repeatedly surprised by it. I'm unclear on why.

For decades I thought it bizarre that the New York Times didn't even manage to say the word. Until Trump and MAGA. Yay, a breakthrough? No, at least not the last time I checked several years ago. It remained associated with only Trump and MAGA. Some things are easier to see across the battlefield, than in the mirror, or literally everywhere?

True, but that doesn't mean intelligence is the bottleneck of good governance. Good governance also requires simple, plain brute force. You have to herd millions of cats, all of which have their own intelligence, and the richest of which are basically giant tigers. At our current level of individual intelligence, we have plenty of simple solutions to many of the problems that plague us, but the reality of the boiling ocean of society is that it tends to crush them faster than they can be built up.

When the problem is the coordination of intelligent agents with disparate interests, increasing everyone's intelligence isn't necessarily going to lead to a solution. Maybe if everyone operates in good faith with a compromising spirit. But if our best solution creates losers, they will fight to the bitter end, in which case we don't need intelligence as much as we need a bigger boot. That's governance.

They are effectively arguing that AI will be God-like.

I don't mean here God in the religiously charged sense but in the philosophical sense.

A god is simply a person or thing that grants salvation.

Whether that's looking to a rain god to to solve a drought, a luck god to to solve a series of bad events, etc.

It doesn't matter how the god does it but the adherents trust that the god has the powers to do it.

In this case the idea is that the AI is going to be all-knowing and all-wise, and thus can lead mankind towards every manner of salvation.

It has a god-like flavor to it.

https://www.youtube.com/watch?v=XuoqKYxDHVc&t=1880s

Minton Beddoes: What people who lose their jobs, who are displaced, who may go off and garden, but how are they going to live? And what kind of I mean, it sounds to me...

Musk: universal high income

Minton Beddoes: but how are you going to get there? Let's talk about the politics of this cuz right now we if we look at the political pressures in most countries people are very very worried about this very against it. I mean it seems to me that there's a risk that we are heading more for revolution than for an easy route to this nirvana you're playing out.

Musk: I mean it will be a bumpy road. I'm not suggesting it will be sort of that everything will be smooth sailing here. Um, so I mean there are many jobs that used to exist that don't exist because of computers. I mean the the word computer used to refer to a person doing computations like that used to be a job description of a person. And you can see pictures online of um of buildings that... drifts away

I think this is true. But it doesn’t get rid of the possibility of hyperpersuasive bots, just, points to a reason they might not be as consequential as expected. I think bots that are very good at writing rhetoric might happen (why not? They are already good at math and chess), and they would be devastating to the spaces where people go online and argue with anonymous strangers. But I don’t know if that’ll matter. Could be bad for bloggers. Complex for policy people.

IMO if we’re really honest with ourselves, I think we get more of our opinions by stewing in memes and other social processes. Offline or online. The sort of hyperpersuasive ML that we really should have been worrying about was the Twitter/reddit/Tick-Tock engagement algorithms.