Underwriting Superintelligence

Thanks for the thoughtful response! Some replies:

1. Someone is always carrying the risk; the question is, who it should be? We suggest private markets should price and carry the first $10B+ before the government backstop. That incentivizes them to price and manage it.

2. Insurance has plenty of ways to manage moral hazard (e.g. copays). Pricing any new risk is hard, but at least with AI you can run simulations, red-team, review existing data, etc.

3. We agree on existential losses, but catastrophic events can be priced and covered. Insurers enforcing compliance with audits/standards would help them reduce catastrophes, in turn reducing the risk of many existential risks.

You seem to be part of a trend where most humans are defined as unintelligent - there are remarkably few people out there capable of formulating novel algorithms or physics hypothesises. Novels there are a few more if we admit unreadable slop produced by people who really should choose careers other than writing. It speaks to the progress that machines have made that traditional tests of intelligence, like holding a conversation or doing well on an undergraduate-level university test, apparently no longer measure anything of importance related to intelligence.

If we admit that even relatively stupid humans show some levels of intelligence, as far as I can tell we've already achieved artificial intelligence.

> Is there any indication whatsoever that there's even a glimpse of artificial intelligence out there?

no

> But we don’t want medical device manufacturers or nuclear power plant operators to move fast and break things. AI will quickly get baked into critical infrastructure and could enable dangerous misuse.

nobody will put a language model in a pacemaker or a nuclear reactor, because the people who would be in a position to do such things are actual doctors or engineers aware both of their responsibilities and of the long jail term that awaits them if they neglect them

this inevitabilism, to borrow a word from another submission earlier today, about "AI" ending up in critical infrastructure and the important thing being to figure out how to do it right is really quite repugnant

sure, yes, i know about the shitty kinda-explainable statistical models that already control my insurance premiums or likelihood of getting policed or whatever

but why is it a foregone conclusion that people are going to (implicitly rightly so given the framing lets it pass unquestioned!) put llms into things that materially affect my life on the level of it ending due to a stopped heart or a lethal dose of radiation

This article is a bizarre mix of center-right economic ideas and completely unfounded assumptions about the nature of AI technology, to the point where I'm genuinely not sure if this is intended as parody or not.

> We’re navigating a tightrope as Superintelligence nears.

There is no evidence we're anywhere near "superintelligence" or AGI. There is no evidence any AI tools are intelligent in any sense, yet alone "superintelligence". The only reference for this, given much later, is to https://ai-2027.com/ which is no more than fan fiction. You might as well have cited Terminator or The Matrix as evidence.

The only people actually claiming any advancement towards "superintelligence" or "AGI" directly financially gain from people thinking that it's right around the corner.

> If the West slows down unilaterally, China could dominate the 21st century.

Is this casual sinophobia intended to appeal to a particular audience? I can't see what purpose this statement, and others like it, serves other than to try to frame this as "it's us or them".

> Faster than regulation: major pieces of regulation, created by bureaucrats without technical expertise, move at glacial pace.

This is a very common right-wing viewpoint. That regulation, government oversight, and "red tape" is unacceptable to business. Forgetting that building codes, public safety regulations, and workers rights all stem directly from government regulation. This article goes out of it's way to frame it as obvious, like a simple fact unworthy of introspection.

> Enterprises must adopt AI agents to maintain competitiveness domestically and internationally.

There is no evidence this is the case, and no citation is even attempted.