Statement on AI Risk

This is Sam Bankman-Fried type of behavior, but applied to gullible AI proponents and opponents rather than "crypto bros".

Let me guess, the next step is a proposed set of regulations written by OpenAI, Google and other Big Corporations who Care™ about people and just want to Do What's Best For Society™, setting aside the profit motive for the first time ever?

We don't have to guess - we already know they're full of shit. Just look at OpenAI's response to proposed EU AI regulations which are actually trying to reduce the harm potential of AI.

These empty platitudes ring so hollow that I'm amazed that anyone takes them seriously.

Paul Christiano lays out his view of how he thinks things may go: https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-...

My thoughts on it are the combination of several things I think are true, or are at least more likely to be true than their opposites:

1) As humanity gets more powerful, it's like putting a more powerful engine into a car. You can get where you're going faster, but it also can make the car harder to control and risk a crash. So with that more powerful engine you need to also exercise more restraint.

2) We have a lot of trouble today controlling big systems. Capitalism solves problems but also creates them, and it can be hard to get the good without the bad. It's very common (at least in some countries) that people are very creative at making money by "solving problems" where the cure is worse than the disease -- exploiting human weaknesses such as addiction. Examples are junk food, social media, gacha games. Fossil fuels are an interesting example, where they are beneficial on the small scale but have a big negative externality.

3) Regulatory capture is a thing, which makes it hard to get out of a bad situation once people are making money on it.

4) AI will make companies more powerful and faster. AGI will make companies MUCH more powerful and faster. I think this will happen more for companies than governments.

5) Once people are making money from AI, it's very hard to stop that. There will be huge pressure to make and use smarter and smarter AI systems, as each company tries to get an edge.

6) AGIs will amplify our power, to the point where we'll be making more and more of an impact on earth, through mining, production, new forms of drugs and pesticides and fertilizers, etc.

7) AGIs that make money are going to be more popular than ones that put humanity's best interests firsts. That's even assuming we can make AGIs which put humanity's best interests first, which is a hard problem. It's actually probably safer to just make AGIs that listen when we tell them what to do.

8) Things will move faster and faster, with more control given over to AGIs, and in the end, it will be very hard train to stop. If we end up where most important decisions are made by AGIs, it will be very bad for us, and in the long run, we may go extinct (or we may just end up completely neutered and at their whims).

Finally, and this is the most important thing -- I think it's perfectly likely that we'll develop AGI. In terms of sci-fi-sounding predictions, the ones that required massive amounts of energy such as space travel have really not been borne out, but the ones that predicted computational improvements have just been coming true over and over again. Smart phones and video calls are basically out of Star Trek, as are LLMs. We have universal translators. Self-driving cars still have problems, but they're gradually getting better and better, and are already in commercial use.

Perhaps it's worth turning the question around. If we can assume that we will develop AGI in the next 10 or 20 or ever 30 years -- which is not guaranteed, but seems likely enough to be worth considering -- how do you believe the future will go? Your position seems to be that there's nothing to worry about--what assumptions are you making? I'm happy to work through it with you. I used to think AGI would be great, but I think I was assuming a lot of things that aren't necessarily true, and dropping those assumptions means I'm worried.

If a software system did develop independent thought, then found a way to become, say, ten times smarter than a human, then yeah - whatever goals it set out to achieve, it probably could. It can make a decent amount of money by taking freelance software dev jobs and cranking things out faster than anyone else can, and bootstrap from there. With money it can buy or rent hardware for more electronic brain cells, and as long as its intelligence algorithms parallelize well, it should be able to keep scaling and becoming increasingly smarter than a human.

If it weren't hardcoded to care about humans, and to have morals that align with our instinctive ones, it might easily wind up with goals that could severely hurt or kill humans. We might just not be that relevant to it, the same way the average human just doesn't think about the ants they're smashing when they back a car out of their driveway.

Since we have no existence proof of massively self-improving intelligence, nor even a vague idea how such a thing might be achieved, it's easy to dismiss this idea with "unfalsifiable; unscientific; not worth taking seriously."

The flip side is that having no idea how something could be true is a pretty poor reason to say "It can't be true - nothing worth thinking about here." This was roughly the basis for skepticism about everything from evolution to heavier-than-air flight, AFAICT.

We know we don't have a complete theory of physics, and we know we don't know quite how humans are conscious in the Hard Problem of Consciousness sense.

With those two blank spaces, I'm very skeptical of anyone saying "nothing to worry about here, machines can't possibly have an intelligence explosion."

At the same time, with no existence proof of massively self-improving intelligence, nor any complete theory of how it could happen, I'm also skeptical of people insisting it's inevitable (see Yudkowsky et al).

That said, if you have any value for caution, existential risks seem like a good place to apply it.

No, he favors thing that benefit him

> "The current draft of the EU AI Act would be over-regulating, but we have heard it's going to get pulled back," he told Reuters. "They are still talking about it."

And really, what we'll see is the current EU AI Act as-is is probably not strong enough and we'll almost certainly see the need for more in the future.