Anyone have a list of benchmarks that do not release the actual test set?
Anyone else share the suspicion that ML rapidly approaching 100% on benchmarks is sometimes due to releasing the test set?
ARC Prize – a $1M+ competition towards open AGI progress
241–250 of 351 posts
Re: ARC Prize – a $1M+ competition towards open AGI progress
#242I'm Simon Strandgaard and I participated in ARCathon 2022 (solved 3 tasks) and ARCathon 2023 (solved 8 tasks). I'm collecting data for how humans are solving ARC tasks, and so far collected 4100 interaction histories ( https://github.com/neoneye/ARC-Interactive-History-Dataset ). Besides ARC-AGI, there are other ARC like datasets, these can be tried in my editor ( https://neoneye.github.io/arc/ ). I have made some vi…
The UX of your solution entry is _way_ better than the ARC site itself.
Being able to hold the mouse button down is certainly much nicer. Not being able to see the examples while you are solving makes it harder than it should be though.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#243I found them all extremely easy for a while, but then I couldn't figure out the rules of this one at all: e6de6e8f https://i.imgur.com/ExMFGqU.png
yeah it's off somehow. rule 1: start at the green dot? rule 2: glue the left outer piece to the bottom rule 3: overlap every now and then :D rule 4: invert some of the pieces every now and then
[deleted]
Re: ARC Prize – a $1M+ competition towards open AGI progress
#244Earlier quoted context omitted.
AGI should obviously be able to do them. But AI being able to do those 100 percent wouldn't be evidence of AGI however. It is a very narrow domain.
Why not? If the only thing that can solve problem X is AGI (e.g. humans), and something else comes along that solves it, then rationally that should be evidence that the something else is AGI right? Unless you have strong prior beliefs (like "computers can't be AGI") or something else that's problem specific ("these problems can be solved by these techniques which don't count as AGI"). So I guess that's my real quest…
Humans can do infinitely many things because we have general intelligence.
Testing whether an AI can play chess or solve Chollet's ARC problems, or some other set of narrow skills, doesn't prove generality. If you want to test for generality, then you either have to:
1) Have a huge and very broad test suite, covering as many diverse human-level skills as possible.
and/or,
2) Reductively understand what human intelligence is, and what combination of capabilities it provides, then test for all of those capabilities both individually and in combination.
As Chollet notes, a crucial part of any AGI test is solving novel problems that are not just templated versions (or shallow combinatins) of things the wanna-be AGI has been trained on, so for both of above tests this is key.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#245Earlier quoted context omitted.
> How many homework questions did your entire calc 1 class have? I'm guessing less than 100 and (hopefully) you successfully learned differential calculus. Not just that: people learn mathematics mainly by _thinking over and solving problems_, not by memorising solutions to problems. During my mathematics education I had to practice solving a lot of problems dissimilar what I had seen before. Even in the theory part,…
> People think that knowledge lies in the texts themselves; it does not, it lies in what these texts relate to and the processes that they are part of, a lot of which are out in the real world and in our interactions And almost all of it is just more text, or described in more text. You're very much right about this. And that's exactly why LLMs work as well as they do - they're trained on enough text of all kinds and…
It is not "just more text". That is an extremely reductive approach on human cognition and experience that does favour to nothing. Describing things in text collapses too many dimensions. Human cognition is multimodal. Humans are not computational machines, we are attuned and in constant allostatic relationship with the changing world around us.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#246Earlier quoted context omitted.
To parent: the spatial reasoning and blind person were great counterexamples. It still might be OK despite the blind exceptions if it showed general reasoning. To OP: I like your project goal. I think you should look at prior, reasoning engines that tried to build common sense. Cyc and OpenMind are examples. You also might find use for the list of AGI goals in Section 2 of this paper: https://arxiv.org/pdf/2308.04445…
CYC was an expert system, which is arguably what LLMs are. A similar vintage GOFAI project that might do better on these, with a suitable visual front end, is SOAR - a general purpose problem solver.
LLM’s aren't expert systems. A hallmark of expert systems is they encoded human-readable, human-checked knowledge with explainable reasoning. It was usually done as if-then rules. Others with logic programming. Forward and backward chaining for rules. Usually had specialist knowledge for one, use case.
LLM’s are unsupervised, use probabilities with unpredictable results, and don’t explain every step of their thinking. They’re the opposite.
You might argue Cyc was. It was also more complex than any expert system I had ever seen. We just called stuff like that a reasoning engine or just Cyc to avoid confusion.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#247I found them all extremely easy for a while, but then I couldn't figure out the rules of this one at all: e6de6e8f https://i.imgur.com/ExMFGqU.png
It seems there is an error in the 3rd example. The rule is, take each figure from left to right and stack each under the previous one. For L and J shapes the top cell is stripped. The L shape dictates that the next shape will be shifted one cell to the right, the J shape tells the next figure to shift to the left.
If all examples are right, then the rule is more complicated than that, involving rotating L clockwise, J counterclockwise. Authors claim that it should be solvable by children, then the rule must be simple.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#248Earlier quoted context omitted.
CYC was an expert system, which is arguably what LLMs are. A similar vintage GOFAI project that might do better on these, with a suitable visual front end, is SOAR - a general purpose problem solver.
LLM’s aren't expert systems. A hallmark of expert systems is they encoded human-readable, human-checked knowledge with explainable reasoning. It was usually done as if-then rules. Others with logic programming. Forward and backward chaining for rules. Usually had specialist knowledge for one, use case. LLM’s are unsupervised, use probabilities with unpredictable results, and don’t explain every step of their thinking…
An expert system is just a system based on repeated application of declarative rules. CYC was certainly an expert system - the ultimate scaling experiment of expert systems. I believe CYC also had a variety of inference/reasoning engines in addition to it's set of rules.
The rules (some prefer to call it a world model) in an LLM are deduced, via gradient descent, from the training samples, but are still there. The transformations effected by each layer of a transformer are exactly those it has learnt - the rules it is applying.
As with CYC people seem to be hoping that some external scaffolding (better inference engine(s)) will rescue LLMs from just being a set of rules to something more general and capable, but I tend to agree with Chollet that this active inference (reasoning) is actually the hard part.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#249Earlier quoted context omitted.
The UX of your solution entry is _way_ better than the ARC site itself.
Being able to hold the mouse button down is certainly much nicer. Not being able to see the examples while you are solving makes it harder than it should be though.
I have create an issue with your suggestion.
https://github.com/neoneye/ARC-Interactive/issues/67
Seeing the examples while having the editor visible. That's a good idea. I haven't explored this direction, since I had my phone (with tiny screen estate) in mind.
Drafts for a such a UI are much welcome. However I'm probably too lazy to code it though.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#250Earlier quoted context omitted.
Literally every single thing you reason about is something happening in space-time.
Where exactly in space-time are complex numbers? Could you point me to 2+i for example? How about some aliens in a SF book. When we reason about them, where are they exactly? Literally on the pages of the book? How about a context-free grammar?
Complex numbers are just 2d numbers and they are mapped on a 2d plane, yeah. They are just a 2d vector. Callling them imaginary numbers is silly in the first place, 2+i is just the vector (2,1), all we mean here are the two numbers are orthogonal, i.e they are destinguished by some independent factor. The imaginary component is no more imaginary than the real component.
I mean what problems does physics solve not with just complex number but with even more complex vectors? Problems of...space-time.
Aliens in a SF book. What do you imagine? I see some kind of physical entity having geometric compomnents in some kind of space.
Context free grammers are represented by...trees where one side of a spatial relationship maps to one idea and the other to another. What is context? Things surrounding something, where something is.
Come up with any idea, it can be represented in space and time.