Rogue superintelligence: Inside the mind of OpenAI's chief scientist

Tried with GPT4, I got 12.

They had very significant effects on the nuclear policy. Oppenheimer's (of course a big movement of which he figureheaded) big idea was to prevent a nuclear arms race, which obviously didn't fully materialize, but what was the basis for e.g. test bans, disarmantments and limiting proliferation.

I agree that you shouldn't ask the inventor just because they invented the gadget. But at least in the Manhattan project the scientists were in a very strong position that if they refuse to co-operate, the bomb just won't happen (soon enough). And for that you'd want inventors versed in the wider implications.

One difference from that era is probably that interest in wider philosophy and politics was encouraged from academics. E.g. the "giants of modern physics" (Oppenheimer, Einstein, Bohr etc) took great interest and scholarship in philosophy and societal issues whereas nowadays, as you said, there's practically 0% of the inventor of the gadget to understand the issues. They should "shut up and calculate", and leave the philosophy to philosophers and the societal impact to economists.

A problem is that philosophers and economists don't really understand the technology and its ramifications, and are heavily influenced by the hype. And philosophers have very little power to influence anyway. There are valid reasons for such specialization, but it has drawbacks.

> (Sutskever) has an exemplar in mind for the safeguards he wants to design: a machine that looks upon people the way parents look on their children

The most troubling statement in the entire article, buried at the bottom, almost a footnote.

Imagine for a moment a superintelligent AGI. It has figured out solutions to climate change, cured cancer, solved nuclear proliferation and world hunger. It can automate away all menial tasks and discomfort and be a source of infinite creative power. It would unquestionably be the greatest technological advancement ever to happen to humanity.

But where does that leave us? What kind of relationship can we have with an ultimate parental figure that can solve all of our problems and always knows what's best for us? What is left of the human spirit when you take away responsibility, agency, and moral dilemma?

I for one believe humans were made to struggle and make imperfect decisions in an imperfect world, and that we would never submit to a benevolent AI superparent. And I hope not to be proven wrong.

What's clear here is that users of OpenAI's products will end up in a worse place as a result of these developments. Ilya is on record as being against open sourced models with the view that they are too "powerful" to release. There are also accounts that Dev Day became a driving force for ousting Altman and stopping signups. Dev Day was about putting tools in the hands of users, so it's clear that his motivation is to restrict access to this technology. I don't want amateur philosophy from an LLM, I want greater capabilities and reduced costs. My hope is that this motivates user-focused competitors now that they have a sizeable window to catch up. So from my view Ilya will set back the field in the short term, but will spur competition in the long term.

“It is the idea—” he starts, then stops. “It’s the point at which AI is so smart that if a person can do some task, then AI can do it too. At that point you can say you have AGI.”

—-

Ilya’s success has been predicated on very effectively leveraging more data and more compute and using both more efficiently. But his great insight about DL isn’t a great insight about AGI.

Fundamentally, he doesn’t define AGI correctly, and without a correct definition, his efforts to achieve it will be fruitless.

AGI is not about the degree of intelligence, but about a kind of intelligence. It is possible to have a dumb general intelligence (a dog) and a smart narrow intelligence (GPT).

When Ilya muses about GPT possibly being ephemerally conscious, he reveals a critically wrong assumption: that consciousness emerges from high intelligence and that high intelligence and general intelligence are the same thing. According to this false assumption, there is no difference of kind between general and narrow intelligence, but only a difference of degree between low and high. Moreover, consciousness is merely a mysterious artifact of little consequence beyond theoretical ethics.

AGI is a fundamentally different type of intelligence than anything that currently exists, unrelated and orthogonal to the degree of intelligence. AGI is fundamentally social, consisting of minds modeling minds — their own, and others. This modeling is called consciousness. Artificial phenomenological consciousness is the fundamental prerequisite for artificial (general) intelligence.

Ironically, alignment is only possible if empathy is built into our AGIs, and empathy (like intelligence) only resides in consciousness. I’ll be curious to see if the work Ilya is now doing on alignment leads him to that conclusion. We can’t possibly control something more intelligent than ourselves. But if the intelligence we create is fundamentally situated within a empathetic system (consciousness), then we at least stand a chance of being treated with compassion rather than contempt.