Ask HN: Have top AI research institutions just given up on the idea of safety?

safety is like climate change mitigation -- it's an extra expense with little obvious immediate financial return, and if your competitors don't care, then you caring just holds you back while they can forge ahead

until the day there's a catastrophe and the cost of repairing it far far exceeds the cost of safety, but by then it's someone else's problem and you have your golden parachute and your investors have cashed out

Challenge in court is fine, even healthy.

Threatening to burn everything down and refuse to comply might well work; simply daring Trump to a game of Russian Roulette about this popping the bubble that's only just managing to keep the US economy out of recession, on the basis that he TACOs a lot, I can see it working in a way it wouldn't if he were a sane leader making the same actual demands just for sane reasons.

Move the company to a different jurisdiction? That would have worked if AI was a few hundred people and a handful of servers, as per classic examples of:

  At the height of its power, Kodak employed more than 140,000 people and was worth $28 billion. They even invented the first digital camera. But today Kodak is bankrupt, and the new face of digital photography has become Instagram. When Instagram was sold to Facebook for a billion dollars in 2012, it employed only 13 people. Where did all those jobs disappear? And what happened to the wealth that all those middle class jobs created?

But (I think) now that AI needs new data centres so fast and on such a scale that they're being held back by grid connection and similar planning permission limits, this isn't a viable response.

They can be burned down, but I think they can't realistically be moved at this point. That said, I guess it depends on how much Anthropic relies on their own data centres vs. using 3rd parties, given Amazon's announced AWS sovereign cloud in Europe?

When I was working in defense technology I had two questions for engineers when we talked about Safety:

1) Can the operator assess the risk of using this technology? 2) If something goes wrong during operations, can the operator mitigate the risk?

The degree to which either of those statements is true is a measure of how safe that technology is. Technology that is simple to understand and executes deterministically every single time and where it is obvious if it is malfunctioning and the operator has enough time to either correct it or stop it, is generally perceived as safe. Technology that hides what it is really doing, confuses the operator about what the effects of operating it might be, and either executes faster than the operator can respond or specifically prevents the operator from responding, is more likely to trigger negative safety outcomes.

The problem the AI industry faces is that tricking the operator into thinking the technology is doing something it is not is explicitly part of their business model. Read any of the mentioned authors (Dekker is probably the best starting point) and it will become obvious why AI Safety is impossible when AI is dependent on pretending to "think" and "reason". In order to be safe they would have to abandon that. If they abandon that, they will be unable to raise the capital they need to keep the bubble from bursting. The technology will survive, maybe with another AI winter, but many of the businesses will not.

So they will abandon the lip service about Safety instead, but then that was never real Safety to begin with. Real Safety is not about zero risk. It is just as impossible to have zero risk as it is to have 100% uptime. Real Safety is about how the technology is designed to manage risk as part of an overall system.