Earlier quoted context omitted.
ARC 3 is still spatially 2D, but it adds a time dimension, and it's interactive.
If you aren't joking, that will filter most humans.
They said at least two people out of 400 solved each problem so they're pretty hard.
Arc-AGI-2 and ARC Prize 2025
31–40 of 103 posts
Re: Arc-AGI-2 and ARC Prize 2025
#32Did we run out of textual tasks that are easy for humans but hard for AI, or why are the examples all graphics?
Re: Arc-AGI-2 and ARC Prize 2025
#33> and was the only benchmark to pinpoint the exact moment in late 2024 when AI moved beyond pure memorization This is self-referential, the benchmark pinpointed the time when AI went from memorization to problem solving, because the benchmark requires problem solving to complete. How do we know it requires problem solving skills? Because memorization-only LLMs can't do it but humans can. I think ARC are producing som…
The reason these tasks require fluid intelligence is because they were designed this way -- with task uniqueness/novelty as the primary goal. ARC 1 was released long before in-context learning was identified in LLMs (and designed before Transformer-based LLMs existed), so the fact that LLMs can't do ARC was never a design consideration. It just turned out this way, which confirmed our initial assumption.
>> The reason these tasks require fluid intelligence is because they were designed this way -- with task uniqueness/novelty as the primary goal.
That's in no way different than claiming that LLMs understand language, or reason, etc, because they were designed that way.
Neural nets of all sorts have been beating benchmarks since forever, e.g. there's a ton of language understanding benchmarks pretty much all saturated by now (GLUE, SUPERGLUE ULTRASUPERAWESOMEGLUE ... OK I made that last one up) but passing them means nothing about the ability of neural net-based systems to understand language, regardless of how much their authors designed them to test language understanding.
Failing a benchmark also doesn't mean anything. A few years ago, at the first Kaggle competition, the entries were ad-hoc and amateurish. The first time a well-resourced team tried ARC (OpenAI) they ran roughshod over it and now you have to make a new one.
At some point you have to face the music: ARC is just another benchmark, destined to be beat in good time whenever anyone makes a concentrated effort at it and still prove nothing about intelligence, natural or artificial.
Re: Arc-AGI-2 and ARC Prize 2025
#34Did we run out of textual tasks that are easy for humans but hard for AI, or why are the examples all graphics?
You can easily convert these tasks to token strings. The reason why ARC does not use language as part of its format is that it seeks to minimize the amount of prior knowledge needed to approach the tasks, so as to focus on fluid intelligence as opposed to acquired knowledge.
All ARC tasks are built entirely on top of "Core Knowledge" priors, the kind of elementary knowledge that a small child has already mastered and that is possessed universally by all humans.
Re: Arc-AGI-2 and ARC Prize 2025
#35Earlier quoted context omitted.
Sorry, I probably phrased the question poorly. My question is more along the lines of "when you already scored e.g. OpenAI's o3 on ARC AGI 2 how did you guarantee OpenAI can't just look at its server logs to see question set 4"?
Ah yes, two things 1. We had a no-data retention agreement with them. We were assured by the highest level of their company + security division that the box our test was run on would be wiped after testing 2. We only tested o3 against the semi-private set. We didn't test it with the private eval.
>> We were assured by the highest level of their company + security division that the box our test was run on would be wiped after testing
Yuri Geller assured us he was bending the spoons with his mind. Somehow it was only when the Amazing Randi was present that Yuri Geller couldn't bend the spoons with his mind.
Re: Arc-AGI-2 and ARC Prize 2025
#36Earlier quoted context omitted.
The reason these tasks require fluid intelligence is because they were designed this way -- with task uniqueness/novelty as the primary goal. ARC 1 was released long before in-context learning was identified in LLMs (and designed before Transformer-based LLMs existed), so the fact that LLMs can't do ARC was never a design consideration. It just turned out this way, which confirmed our initial assumption.
>> The reason these tasks require fluid intelligence is because they were designed this way -- with task uniqueness/novelty as the primary goal. That's in no way different than claiming that LLMs understand language, or reason, etc, because they were designed that way. Neural nets of all sorts have been beating benchmarks since forever, e.g. there's a ton of language understanding benchmarks pretty much all saturated…
The first time a top lab spent millions trying to beat ARC was actually in 2021, and the effort failed.
By the time OpenAI attempted ARC in 2024, a colossal amount of resources had already been expended trying to beat the benchmark. The OpenAI run itself costs several millions in inference compute alone.
ARC was the only benchmark that highlighted o3 as having qualitatively different abilities compared to all models that came before. o3 is a case of a good approach meeting an appropriate benchmark, rather than an effort to beat ARC specifically.
Re: Arc-AGI-2 and ARC Prize 2025
#37Earlier quoted context omitted.
The reason these tasks require fluid intelligence is because they were designed this way -- with task uniqueness/novelty as the primary goal. ARC 1 was released long before in-context learning was identified in LLMs (and designed before Transformer-based LLMs existed), so the fact that LLMs can't do ARC was never a design consideration. It just turned out this way, which confirmed our initial assumption.
>> The reason these tasks require fluid intelligence is because they were designed this way -- with task uniqueness/novelty as the primary goal. That's in no way different than claiming that LLMs understand language, or reason, etc, because they were designed that way. Neural nets of all sorts have been beating benchmarks since forever, e.g. there's a ton of language understanding benchmarks pretty much all saturated…
I mostly agree with what your are saying but…
> passing them means nothing about the ability of neural net-based systems to understand language, regardless of how much their authors designed them to test language understanding.
Does this implicitly suggest that it is impossible to quantitatively assess a system’s ability to understand language? (Using the term “system” in the broadest possible sense)
Not agreeing or disagreeing or asking with skepticism. Genuinely asking what your position is here, since it seems like your comment eventually leads to the conclusion that it is unknowable whether a system external to yourself understands language, or, if it is possible, then only in a purely qualitative way, or perhaps purely in a Stewart-style-pornographic-threshold-test - you’ll know it when you see it.
I don’t have any problem if that’s your position- it might even be mine! I’m more or less of the mindset that debating whether artificial systems can have certain labels attached to them revolving around words like “understanding,” “cognition,” “sentience” etc is generally unhelpful, and it’s much more interesting to just talk about what the actual practical capabilities and functionalities of such systems are on the one hand in a very concrete, observable, hopefully quantitative sense, and how it feels to interact with them in a purely qualitative sense on the other hand. Benchmarks can be useful in the former but not the latter.
Just curious where you fall. How would you recommend we approach the desire to understand whether such systems can “understand language” or “solve problems” etc etc… or are these questions useless in your view? Or only useful in as much as they (the benchmarks/tests etc) drive the development of new methodologies/innovations/measurable capabilities, but not in assigning qualitative properties to said systems?
Re: Arc-AGI-2 and ARC Prize 2025
#38Maybe this is a really stupid question but I've been curious... are LLMs based on... "Neuronormativity"? Like, what neurology is an LLM based on? Would we get any benefit from looking at neurodiverse processing styles?
It’s kind of a silly question in that the neural architecture of neural nets is really only loosely inspired by neurology, and that basic vague neurology is shared by neurotypical people and neurodivergent people and animals and even bugs.
Re: Arc-AGI-2 and ARC Prize 2025
#39I spent half an hour playing with these now at https://arcprize.org/play and it's fun, but I must say that they are not "easy". So far I eventually solved all of the ones I've gone through, but several took me significantly more than the 2 tries allotted.
I wonder if this can be shown to be a valid IQ test, and if so, what IQ would a person need to solve e.g. 90% of them in 1 or 2 tries.
Re: Arc-AGI-2 and ARC Prize 2025
#40Probably openai will be >60% in three months if not immediately with these $1000/question level compute (which is the way tbh we should throw compute whenever possible that's the main advantage of silicon intelligence)