What Is AGI-Hard

Try Noita. No idea how people discovered all the ways to complete all the quests in the game. See e.g. https://noita.wiki.gg/wiki/Sun_Quest_Guide

Same goes for Spelunky, or Caves of Qud, or nearly any modern roguelike/rogue-lite game.

The first example the author gives, adjusting language to the level of a student, does not seem unrealistic at all, provided there are examples of this in the training data.

It's an interesting question. Solving extremely difficult math problems, e.g. the Riemann hypothesis or P vs NP, would be very impressive. Or coming up with algorithms given many examples of desired results. There should be many benefits to having a much better short-term memory and instant access to the accumulated human knowledge. We've already seen AI do surprising things through just pattern recognition, such as predicting a patient's race from medical images when human experts cannot do it. AI might be able to understand an extremely complex system, such as the world economy, and tell us what to change to make it work better.

It might not be what you asked for but in practice "any sufficiently advanced technology is indistinguishable from magic" rule will come into play even with just quantitative advances. And we might get practical quantum computers at some point...

Yes, the problem is to replicate hundred of millions years of acquisition of heuristics that allows cognition on «natural» environment. Those heuristics defines all bricks our cognition is based upon, think of basics things like objects permanence, we just cannot reason without such ability.

To me, as long as no AI manage to replicate those bricks, at least some of them, it won't be able to interact with us in ways that helps us efficiently with the outer, real, world.

> how accurate would experts have been at predicting the current capabilities of AI if you asked them 10 years ago?

Maybe less inaccurate than you may think. Especially if you include "experts" that focus specifically at estimating future developments, and ignore "experts" that have focused all their effort into some very specific algorithmic detail.

It seems to me that a key factor in longer-term estimations is simply compute performance. If development of AI seems to be lagging somewhat compared to some predictions, it seems to me that this lag is similar to the slowdown in Moore's law. Also, for robotics, wearables and vehicles, power consumption of electronics combined with the fact that batteries as still heavy and expensive is an important limiting factor.

Now the METHODS we use to reach predictions of people like Kurzweil may be different from what we imagined. Specifically, it seems that many futurists were thinking that AGI would be reached through Turing Machines and rationalist algorithms. Instead it turns out that most progress is made through building generic learning architectures (like Transformers) and apply them at huge scale with vast amounts of data.

Similarily, a generation ago, we may have imagined we could have a single-core processor performing 1petaflop of computation in the 2020's. Instead, we have the 4090 now, that can do that many computations, but they are limited to "tensor" operations.

Now, assuming we're not making some drastic breakthrough that enables a "rationalist" type of algorithm to form AGI, we should still reach human brain-level of raw compute power sometime between 2030-2050 (which fits reasonably well with many predictions), and I would be surprised if we don't have full AGI within such a timeframe. I would give about 1:1 odds that it happens before 2040.

Still, ideas based on future computers being generic Turing Machines, just faster mean some predictions become fundamentally hard. For instance, Uploading our brains to a computer becomes a lot harder of the computer is not designed simply as a faster Turing Machine, but rather has hardware that is very specialized for some running some simple low level computation massively parallel. It may be close to impossible to run a human brain inside a GPU (at least efficiently), regardless of how many CUDA cores it has.

That means that "uploading" may require us to make a near-exact replica of an actual human brain, with it's spiking style neurons, in silicon.

Also, related predictions by people like Kurzweil may seem far fetched, such as his preditions about nanotech. On the other hand, we HAVE actually started large scale treatment of a large percentage of the world's population using nanotech medicine (mRNA vaccines). Maybe the revolution he predicted is actually about to happen. But my expectation is that it's going to be relatively slow for a few more decades.

It may become radically better if techniques like AlphaFold develop at Moore's law pace in terms of price an capability, but I suspect generic nanotech is a harder problem than AGI simply from a computational perspective.