Effective Altruism’s obsession with AI safety helps bury bad behavior

It's just super obvious by this point that none of the AI safety LARP is really about safety it's more about justifying why a technology that to hire the team to work on it requires that work to be published (Apple tried to do it without publishing and struggled to get the talent) but by publishing you lose part of your advantage so the only way to lock that advantage up is to make it so use of the models has to go through you and also trying to push for reasons for others to not have access to the tech required to train their own model.

If you disagree then why is OpenAI's safety committee talking about how you should need a government contract to buy an A100.

The first few chapters have a song scene parodying ghostbusters and a joke about the protagonist biting a teacher in a tantrum as a young kid. There's plenty of elements of crackfic parody in it.

Well, we're past the asteroids, at least. [1]

The AIpocalypse seems incredibly unlikely. I'm a lot more worried about the nukes, and even if we end up with robotic overlords, I'd bet they'll be a whole lot better at administration than meat -admins have proven.

I'm with you on engineered superviruses. Feasible, likely, and incredibly high impact.

What I kind of keep coming back to is the risk profile of all this stuff. That magic vector of likelihood * impact. Global warming is happening now. And it's real bad - worse than I think we give it credit for.

What I worry about is that we're the proverbial frog in the pot. Things get just slightly hotter each year, so we'll miss it when we actually boil.

1 https://www.google.com/search?q=double+asteroid+redirection+...

> Sooner than most and the media.

This is a very low bar to jump over!

Anyway I read through that article. There's a lot of guff about Newcomb's paradox and Eliezer, and a single paragraph referencing COVID, with two links supporting your statement. I've clicked through those links and have come out unimpressed.

The first link is used to supported the claim that "The rationalist community was well ahead of the public": https://slatestarcodex.com/2020/04/14/a-failure-but-not-of-p... .

Scott rightly highlights how the media and institutions got things wrong, but conversely gives only a few examples of "generic smart people on Twitter" getting some things right in the early days of the outbreak. He confesses that he himself did not predict the seriousness of COVID.

The second link is used to support the claim "the wider tech community, were using masks and stocking up on essential goods, even as others were saying to worry about the flu instead": https://putanumonit.com/2020/02/27/seeing-the-smoke/ .

The linked article was written at the end of February, when panic-buying had already firmly set in across the US.

There is nothing here about the reasoning or predictive abilities of the rationalist or EA community specifically. Nor is there any compelling comparison of its response compared to the wider public.

So there are many failure modes possible here.

Yudkowsky's hard take off mentions "solving protein folding", nanomachines, persuading everyone to let them connect to the internet, infinite adaptability, etc. I think those are unrealistic, but obviously they still have a non-zero chance.

More probably bad possibilities are AI-Hitler (monomaniacal populist absolute leader) or AI-Stalin (manipulative, smart, absolutely paranoid, rising through the ranks) ... so something that's human-like enough to be able to connect with humans, to manipulate them, but at the same time less affected by the psychological shortcomings. (Ie. such an AI could spend enough time to cross-interrogate every underling, constantly watch them, etc.)

And yes, a very efficient immortal dictator is very bad news, but still bound by human-like limits.

And the big infinite dollar question is could this hypothetical AI improve on itself by transcending human limits? Let's say by directly writing programs that it has conscious control over? Can it truly "watch" a 1000 video streams in real-time?

Can it increase the number of its input-and-output channels while maintaining its human-like efficiency?

Because it's very different to run a fast neural network that spits out a myriad labels for every frame of a video stream (YOLO does this already, but it's not 20W!) and to integrate those labels into actions based on a constantly evolving strategy.

Sure maybe the hypothetical AI will simply run a lot of AlphaZero-like hybrid tree-search estimator-evaluator things ...

Anyway, what I'm trying to say is that our 20W efficiency comes with getting tried very fast, and using "fast mode" thinking for everything. (Slow mode is the exception, and using it is so rare that we basically pop open a glass of champagne every time.)