Underwriting Superintelligence

There are certainly pretty gaping holes in its logic but it’s more than a fanfic. I’m a bit confused about the incentive of its authors to add their names to it, since it seems if they’re wrong they lose credibility and if they’re right I’m not sure they’ll be able to cash in on the upside.

This world where wars and stuff happen? You can just use actual force, you know.

>You're dreaming if you think you could actually get all the players to co-operate on this.

Let me play it out more explicitly since you lack the eyes to see. Say China implements such a rule. Top researchers in the US start mysteriously turning up in China and getting turned in for the cash fine. They aren't allowed to leave China until they empty their bank accounts, because they have broken a very serious Chinese law.

Now the US has to seriously ask itself whether it's actually willing to unleash World War 3 for the sake of a couple thousand eggheads. Not gonna happen.

Then someone takes a poll and finds this actually has mass support _in_ the US, because the average person is terrified of where AI is going. If you can't beat em, join em. The US implements its own law similar to the one in China so that this madness can stop. All of the top researchers bitch and moan and move to Portugal.

Top researchers in Portugal start mysteriously turning up in the US... So on and so forth. Rinse and request until everyone is on board with the new Nash equilibrium.

>[I]t doesn't take a huge lab to do it. You can do it at home. It might take longer but there's an 1.4kg blob in everyone's head as proof of concept.

You are thinking so unbelievably small. So, so unbelievably small.

You decide you want to push the state of AI forward and create an enormous new model. Okay, you're going to need to buy or source a bunch of GPUs. You try to order a bunch of GPUs from Nvidia. Nvidia flags you as a high risk client, turns on the tracking firmware, gathers info on your training runs, hires some people in your local area to peek through your curtains to confirm, and turns you in themselves. Nvidia walks away getting roughly half, the other half went to its contractors. You are out of the game.

Alright, so maybe you need to buy a bunch of GPUs on the secondhand market to get started. And maybe you need to source them one or two at a time from several dozen, or hundreds of different players to get away with it. A few of these players take an interest in watching you, notice you're buying GPUs from several secondhand sources, realize this would be extremely weird if you weren't planning on doing something illegal with them, and put together a small posse to turn you in. Your life savings are emptied out and your condo repossessed. You are out of the game.

> It might take longer

Thousands of years longer by my reckoning, at minimum. It's very hard to design an AGI entirely inside your own head when the outside world is so inherently adversarial.