ARC Prize – a $1M+ competition towards open AGI progress

So? I'm still seeing the same object. Large models aren't trained on 10k different images of a single cat.

I don't think this is obviously correct.

Three things:

1) Many actions we think of as "intelligence" are just short-cuts based on heuristics.

2) While there's probably an argument that problem solving is selected for it's not clear to me how far this goes at all. There's little evidence that smarter people end up in more powerful positions for example. Seems like there is perhaps there is a cut-off beyond which intelligence is just a side effect of the problem solving ability that is useful.

3) Perhaps humans individually aren't (very?) intelligent and it is only a society of humans that are.

(also perhaps human GI? Nothing artificial about it.)

> no idea how large the search space is for finding AGI, so applying something like Bayes theorem (which is basically my argument) tells you more about my priors than reality.

There are plenty of imaginable forms of intelligence that are often ignored during these conversations. One in common use is "an intelligent footballer" which applies to sport for someone who can read a game well. There are other, non-human examples too (Dolphins, crows, parrots etc).

And then in the world of speculative fiction there's a range of different types of intelligence. Vernor Vinge wrote about intelligences which had motivations that people couldn't comprehend (and Vinge is generally credited with the concept of the singularity). More recently Peter Watt's Blindside contemplates the separation of intelligence and sentience.

Basically I don't think your expression of Bayes' theorem had nearly enough possibilities in it.

LLMs do have a broad range of abilities, so not narrow AI, but clearly it's not general intelligence (or at least not human level), else they would not be failing or struggling on things that to us are easy - general means universal (not confined to specific types of problem), not just multi-capability.

The lack of reasoning ability, especially since it is architecturally based, seems more than a matter of patching up corner cases that aren't handled well. This shoring up of areas of weakness by increasing model size, adding targeted synthetic data and post-training is mostly just addressing static inference, much like adding more and more rules to CYC.

To make an LLM capable of reasoning it needs to go beyond a fixed N-layers of compute and support open-ended exploration, and probably replace gradient descent with a learning mechanism that can also be used at inference time. In a recent interview John Schulman (one of the OpenAI co-founders) indicated that they hoped that RL training on reasoning would improve it, but that is still going to be architecturally limited. You can learn a repertoire of reasoning templates than can be applied in gestalt fashion, but that's not the same as being able to synthesize a solution to a novel problem on the fly.

LLMs are certainly amazing, and as you say 10-years ago we would have regarded them as AI, but of course the same was true of expert systems and other techniques - we call things we don't know how to do "AI" then relabel them once we move past them to new challenges. Just as we no longer regard expert systems as AI, I doubt in 20 years we'll regard LLMs (which in some regards are also very close to expert systems) as AI, certainly not AGI. AGI will be the technology than can replace humans in many jobs, and when we get there LLMs will in hindsight look very limited.

Sure, I did that plenty too, but that doesn't refute the point that memorization is core to understanding mathematics, it's just a specific kind of memorization that results maximal flexibility for minimal state retention. All you're claiming is that you memorized some core axioms/primitives and the procedures that operate on them, and then memorized how higher-level concepts are defined in terms of that core. I go into more detail of the specifics here:

https://news.ycombinator.com/item?id=40669585

I agree that this is a better way to memorize mathematics, eg. it's more parsimonious than memorizing lots of shortcuts. We call this type of memorizing "understanding" because it's arguably the most parsimonious approach, requiring the least memory, and machine learning has persuasively argued IMO that compression is understanding [1].

[1] https://philpapers.org/rec/WILUAC-2

Evolution hasn't had enough time to adapt us to our new fangled lifestyle of last few hundred years, or few thousand for that matter, and anyways in the modern world people are not generally competing on things affecting survival, but rather on cultural factors that affect number of children we have.

Humans and most (all?) intelligent animals are generalists, which is why we need a big brain and intelligence - to rapidly adapt to a wide variety of ever changing circumstances. Non-generalists such as herbivores, crocodiles don't need intelligence and therefore don't have it.

The main thing that we need to survive & thrive as generalists - and what evolution has evidentially selected for - is ability to predict so that we can plan ahead and utilize past experience. Where will the food be, where will the water be in a drought, etc. I think active reasoning (not just LLM-like prediction/recall) would also play a large role in survival, and presumably parts of our brain have evolved specifically to support that, even if the CEO probably got his job based more on height/looks and golf handicap.