Effective Altruism’s obsession with AI safety helps bury bad behavior

I’d argue that power is in relation to reach and the capacity to influence. In this regard, ML has quite a bit of power these days, from deciding whether you are allowed to take a loan, to driving engagement in social media, to predicting and influencing what you will watch, to the way our language is even used.

There are also quite a few implicit ways that ML models have influenced society, eg GPGPUs with emphasis on ML, and ML accelerators everywhere. The effectiveness of certain algorithms made us use them more often which influenced NVDA and Google towards developing hardware to accelerate those common cases thereby creating a feedback loop where our algorithms are chosen based on works well on our GPUs (cf transformers).

As such systems become more prevalent and influential, ensuring safety and explainability will help us prevent pitfalls that put humans at risk.

This hits the nail on the head for me.

I mostly enjoyed the books as a kid, but didn’t really maintain being a Harry Potter “fan” into adulthood. Out of the dozen or so “hardcore” millennial nerds that I know really well, there’s exactly one person that I think would attempt to read something like that, and based on the reviews I’ve read, I would bet that she would struggle to finish it, if at all.

There is a somewhat interestingly vocal group of internet adults that insist that the default mode for nerds in their 30s is to be Harry Potter super fans, some of whom lapse. Like we were all raised Catholic and some stray.

I was a big fan of Animorphs at that age too, but you won’t see me going around reading or writing Hork-Bajir/Andalite fanfic. Because I enjoyed it as a kid and then went on to read other books.

It matters a lot. Whilst I was eating dinner I've now read the article you cited, the paper that it cites (Kump, Arthur & Pavlov, 2015). I also cross checked the argument and numbers against several other papers. Ward's argument is extremely slippery. This is sadly what I'm coming to expect from academics with unverifiable models. You have to read everything they write adversarially.

"In [Kump2015]'s models, if the deepwater H2S concentrations were to increase beyond a critical threshold during such an interval of oceanic anoxia, then the chemocline separating the H2S-rich deepwater from oxygenated surface water could have floated up to the top abruptly. The horrific result would be great bubbles of toxic H2S gas erupting into the atmosphere."

Observe that his argument starts with a "critical threshold" for H2S levels, not CO2, and that he doesn't tell us what this critical level is. The obvious questions are thus: what is this level, is it realistic for our present day oceans to reach this critical level and if so, how? To get to those levels of H2S he hypothesises that:

"if ancient volcanism raised CO2 and lowered the amount of oxygen in the atmosphere, and global warming made it more difficult for the remaining oxygen to penetrate the oceans, conditions would have become amenable for the deep-sea anaerobic bacteria to generate massive upwellings of H2S."

In other words in this theory the supervolcanoes have to come first, triggering a sharp drop in oxygen levels in the atmosphere, which in turn then causes the chemocline to move, which then causes the H2S upswellings. It's all caused by a massive loss of oxygen, not increase in CO2 levels, which is simply another result of the volcanisms. Kump2015 also makes it clear that their hypothesis requires a truly massive drop in oxygen levels to occur. We'll look at how much in a moment.

But just a few paragraphs later Ward has forgotten all about the volcanoes and oxygen levels. Suddenly it's all about absolute CO2 levels - not even rates! Cause, effect and unrelated side effects have become entirely muddled, probably because he knows nobody would care about his article unless he ties it to global warming armageddon somehow and because hey, this entire field is nothing but assumptions, suppositions, and playing with numbers that can never be verified anyway so why not?

His argument is brittle in other ways. He asserts without any backing argument that whilst supervolcanoes explain all the other non-asteroid mass extinction events it doesn't for the Permian because, apparently, supervolcanoes are really great for plants on land which can "probably" survive the warming. And there was me thinking that large scale volcanic activity is supposed to be very bad for plants because it dims the atmosphere:

https://www.vulkane.net/en/volcanism/supervolcano.html#:~:te....

"Plant growth is restricted and mass extinction can be caused."

Back to the H2S claims. Turning our attention to the Kump2015 paper we find immediately that it's got an annoying structure in which they work backwards from their desired scenario to calculate the level of H2S that could trigger it:

"Thus, if the H2S of the deep sea increased during an anoxic interval beyond a critical value (1 mmol/kg), upwelling regions of the world ocean would become sulfidic, even with the modern [oxygen level at atmospheric pressure] .... A slightly more sulfidic ocean with H2S = 3 mmol/kg ... could sustain ... 2000 times the present-day flux and a critical value for the atmosphere (see following)."

But these critical values are never placed clearly in context. Is 3 millimols a lot or not much? Note that the 3 mmol/kg value is the absolute "best" case for their scenario; it can only be that low in (they estimate) 0.1% of the world's oceans because normal ocean requires levels 20x higher. They do admit that:

"The [H2S] condition is extreme, and thus likely to have been rarely achieved in Earth history. Is there any evidence that such conditions have occurred in the geologic past?"

How extreme is it? The value of the normal ocean would be important to have here but they don't give it to us. The paper "Hydrogen Sulfide in the Black Sea" by Volkov & Neretin does give values though, and bear in mind this is by far the most anoxic basin in the world: at a depth of 1km the value is ~314 micromols/kg. So these H2S levels that are claimed to trigger this process are literally many, many orders of magnitude higher than even the Black Sea.

The levels of oxygen drop needed is something they also don't seem to directly share, but in one section they're kicking around a figure like half of all today's oxygen having vanished, or for a different event, 99% of all oxygen having gone.

Even putting aside that these papers are just piles of completely unverifiable suppositions stacked like a jenga tower, there is absolutely nothing even remotely close to realistic about this scenario happening to us. It requires oxygen levels to drop so much that if it were to ever become an actual threat we'd all have died of oxygen starvation long before.

Consciousness is irrelevant, as it's clearly not necessary for AGI nor even human intelligence; otherwise you wouldn't say "it's not even possible to determine with any scientific certainty that another human being is conscious."

On what grounds do you believe that there are capabilities of the brain that cannot be replicated by a computer? When I look at the 1.5 kgs of matter in a typical human brain, I don't see anything that jumps out and says "my operation is not computable!"

At least with fusion, the high temperatures and pressures are a clear barrier. Our brains don't need to be held at 3.8 trillion psi and 15 million Kelvin in order to enjoy poetry.

I can certainly argue on the other points as well (breaking free / deciding threat / acquiring means) but you need to pick your goalposts one at a time. I think the first step would be asking yourself: if you were a super-genius but being held by guards in solitary confinement on a remote island with only a supercomputer connected to the internet, how would you earn money?

Those conditions clearly would not have stopped even an ordinary Satoshi Nakamoto from gaining control of sufficient resources to hire private military contractors to arrange his escape. I'm not sure what a superhuman would do, but that's a human baseline.