The future of AI according to thousands of forecasters

And you may want to check the weighted Metaculus Prediction if you haven't already.

Counterpoint 1:

> Talking about AI (and AGI) as if its some xenomorph lurking somewhere in silicon, waiting for its inevitable escape from its human prison.

> AI will not bootstrap itself with some emergent property if somebody spends gazillions of dollars and Watts to estimate petazillions of parameters.

Not right now, though recent breakthroughs look worryingly close. But the problem isn't AGI creating itself ex nihilo, but rather us creating it. And if you look at how the world reacted to LLMs, millions of minds and billions of dollars are being thrown towards making that happen - because for the first time in history, it looks like we have an actual "angle of attack", and neither curious minds nor greedy businesses will leave it on the table.

> Further progress is not going to come unless some very human brain and intelligence opens up completely new algorithmic vistas.

All we need to do is to get to roughly average human intelligence in higher cognitive functions, running in pure software. At that point, you can have an AI programmer and an AI ML researcher of similar occupational capability as their human counterparts. But this is software we're talking about - if you can have a pair of them, you can keep horizontally scaling until you have thousands or millions of them. And if you let them directly or indirectly work on their own code... they won't stay human-level for long. You get a feedback loop that's to our technological progress what our technological progress is to biological evolution.

Counterpoint 2:

Maybe it's clearer if instead of using the word "intelligence", we distill it into "optimization". After all, at high level, this is what intelligence is - a powerful optimization process. Now, we humans are no strangers to optimization processes that get out of control. Corporations, arguably, are one. Markets are definitely one, and the aggregate market economy is a powerful optimization process that's already out of control - it's self-directing and owns us. Culture arguably is too, but it's a weak one.

Consider that every time people say things like "if only we weren't so short-sighted and greedy, we would've solved poverty/war/climate change". If you take seriously the sociological and macroeconomic fact that humans follow incentives, then this really becomes "the system/economy/market makes it impossible for us to solve those problems", and... well, what is that system if not for a self-directed optimization process that arose, unintentionally, from our individual needs and desires? Well, semi-unintentionally - free market proponents use exactly this strong optimizer nature to argue why the market is fair and awesome and better than any kind of central planning.

Now, as powerful optimizer as our economic system is, it's still kind of dumb. It averages away individual human intelligence. But what if it didn't? What if it had an actual human-level mind of its own, or at least a human-level mind that's able to perceive and act on the market faster than any one human or group of humans could? Literally getting inside our collective OODA loop? This influence won't get cancelled out - at this point, we'd be screwed.

And that's just one possible scenario - AI tech giving a mind to the economy itself. But the broader point here is, we've already established we can accidentally create powerful generic optimizers that end up controlling us. The ones we deal with today are slow, because they run on humans and human interaction in physical space. But when we create an optimizer that runs on our digital infrastructure, that one will be faster.

I write technical finance/econ articles and the only thing I've found it useful for is getting a vague outline of typical topics that might be touched on in an entry-level article about something.

For example to GPT-4 I ask:

>is silver a good investment? what are the considerations to take into account?

Returns essentially an enumerated list of topics about supply and demand, volatility, storage considerations, and opportunity cost compared to other investments. These are very basic explanations of a sentence or two and usually correct, but too general to be of much use.

I then ask:

>how has the historical performance of silver compared to gold and the S&P 500 over the past 5 decades?

And it provides vague history of what happened to each investment in each decade, no quantifiable data. Okay, let's ask:

>Re-consider the above prompt using quantifiable data, how much would an investment of $1,000 per month adjusted for inflation into each of these investments have compared at the end of the 5 decades?

Output: I apologize for the confusion, but as an AI language model, I don't have access to real-time data or the ability to perform specific calculations.

> Not sure how I feel about a significant portion of the population already not meeting that threshold. The elderly? The disabled?

Anyone? How did the saying go? The minimum viable unit of reproduction for homo sapiens is a village.

None of us passes the bar, if the test excludes "supervision and maintenance" by other people - and not just our peers, but our parents, and their parents, and their parents, ... all the way back until we reach some self-sufficient-ish animal life form. That's, AFAIR, way below primates on evolutionary scale.

But that test is bad also for other reasons, including but not limited to:

- It is confusing intelligence with survival. Survival is not intelligence, it is a highly likely[0] consequence of it[1].

- It underplays the non-random aspect of it. Phrased like GP phrased it, a rock can pass this test. The power of intelligence isn't in passively enduring novel circumstances - it's in actively navigating the world to get more of what you want. Including changing the world itself.

--

[0] - A process optimizing for just about any goal will find its own survival to be beneficial towards achieving that goal.

[1] - If you extend it from human-like intelligence to general optimization, then all life is an example of this: survival is a consequence of natural selection - the most basic form of optimization, that arises when you get a self-replicating something that replicates with some variability into similar self-replicating somethings, and when that variability affects survival.

One argument against ChatGPT's AI-ness was that it's wildly unstable: a small change in the input may become a big change in the output. When that's fixed, GPT will still need memory to learn things, as AI that can't learn isn't really AI. However the memory can be added easily: it's just a hidden step after every prompt when GPT is asked to update it's memory state (a blob of text, its scratchpad). If GPT is stable, it will update the memory more or less properly. With these two changes, and they are fairly simple, GPT5 will earn its AI badge. P.S. I find it interesting that eye and AI sound similarly, so perhaps the symbol of the future AI deity will be an eye.