Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC-AGI-3

arcprize.org

Post reply on HN
Print view View original

331–340 of 394 posts

Re: ARC-AGI-3

#331

Post by daveguy »

BeetleB wrote: post #39

> As long as there is a gap between AI and human learning, we do not have AGI. Back in the 90's, Scientific American had an article on AI - I believe this was around the time Deep Blue beat Kasparov at chess. One AI researcher's quote stood out to me: "It's silly to say airplanes don't fly because they don't flap their wings the way birds do." He was saying this with regards to the Turing test, but I think the sentim…

>> As long as there is a gap between AI and human learning, we do not have AGI.

>> "It's silly to say airplanes don't fly because they don't flap their wings the way birds do."

> Just because a human can do X and the LLM can't doesn't negate the LLM's "intelligence", any more than an LLM doing a task better than a human negates the human's intelligence.

You misinterpret what is meant by "a gap between AI and human learning". The point isn't that they aren't similar enough or that they aren't as intelligent. The statement is specifically about "learning". Humans learn continuously and can devise new strategies for problem solving. Current AI, especially LLMs are just snapshots of a single strategy. LLMs do not learn at all -- they specifically have "knowledge cutoffs" even with all the tools available to them in a harness we still have to wait for new frontier models or new fine tuning for them to solve significantly new problems. A human does this continually -- learn regardless of intelligence.

Re: ARC-AGI-3

#332

Post by daveguy »

adamgordonbell wrote: post #299
suddenlybananas wrote: post #266

Earlier quoted context omitted.

LLMs haven't passed us in language, a child can learn language with so so much less data than an LLM can

isn't that more like rate of learning? Agreed LLM consume a lot of data. But your average LLM understands more languages then anyone alive. So super human understanding of various text based languages.

Rate of learning and general applicability of what is learned is essentially the point of ARC-AGI.

That's why all the AIs score abysmally until humans step in to guide them (fine tuning, harnesses, etc).

Re: ARC-AGI-3

#333

Post by codeinred »

Tiberium wrote: post #35

https://x.com/scaling01 has called out a lot of issues with ARC-AGI-3, some of them (directly copied from tweets, with minimal editing): - Human baseline is "defined as the second-best first-run human by action count". Your "regular people" are people who signed up for puzzle solving and you don't compare the score against a human average but against the second best human solution - The scoring doesn't tell you how m…

We're at the point where LLMs and coding agents are supposed to do higher-level work. It makes sense to benchmark them against top human performance, rather than average human performance, because at specialized tasks, average human performance isn't enough.

The issues you described seem like they're actually strengths of the benchmark.

Re: ARC-AGI-3

#334

Post by daveguy »

vessenes wrote: post #297
MadxX79 wrote: post #294

Same question I have for all these benchmarks: What's going to stop e.g. OpenAI from hiring a bunch of teenagers to play these games non-stop for a month and annotate the game with their logic for deriving the rules, generate a data set based on those playthroughs and fine tuning the next version of chatgpt on all those playthroughs?

Wrong question. I suggest: 1) Do models generalize? 2) If they do, and they generalize from this, is that a win? Chollet was one of the first “they do not generalize” evangelists. I’d be curious to hear what he thinks now, because a) most disagree with him, and b) this test seems designed to get models that can generalize better at visual long context problem solving and agency, exactly where the bleeding edge is rig…

Can AI models generalize+ at any long context problem solving and agency regardless of modality? I think the answer is no, and this is why they are not yet AGI.

+ generalize being the key word.

Re: ARC-AGI-3

#335

Post by notnullorvoid »

maldev wrote: post #314
Marha01 wrote: post #248

Earlier quoted context omitted.

People blind from birth literally lack the neural circuits to comprehend visual data. Are they not intelligent?

I think I can confidently say they are not visually intelligent at all. If you were phrasing things to quantify intelligence, you would have a visual intelligence pillar. And they would not pass that pillar. It doesn't make them dysfunctional or stupid, but visual intelligence is a key part of human intelligence.

Visual intelligence is a near meaningless term as it's almost entirely dependant on spatial intelligence. The visually impaired do have high spatial intelligence, I wouldn't be surprised if their spatial intelligence is actually higher on average than those without visual impairment.

Re: ARC-AGI-3

#336

Post by visarga »

ARC is trying to isolate a unitary intelligence signal, so it strips away coordination, specialization, and division of labor. But that also means it removes one of the dominant mechanisms by which intelligence actually scales in the real world. Their view on intelligence implicitly treats redundancy as necessary - one agent must do them all - and treats efficiency as something achieved internally rather than through restructuring the system. At the very least they should create environments that can help an agent compound intelligence, to self amplify, support itself, that is not happening in ARC.

Anyone wondered if ARC is a measure of intelligence or just a collection of hand picked tasks? was there a proof they encode anything meaningful about intelligence in such short tasks in miniature environments? One shot intelligence?

Re: ARC-AGI-3

#337

Post by notnullorvoid »

Marha01 wrote: post #221
fchollet wrote: post #65

Earlier quoted context omitted.

Francois here. The scoring metric design choices are detailed in the technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf - the metric is meant to discount brute-force attempts and to reward solving harder levels instead of the tutorial levels. The formula is inspired by the SPL metric from robotics navigation, it's pretty standard, not a brand new thing. We tested ~500 humans over 90 minute se…

Don't you see the massive problem with requiring visual input? Are blind people not intelligent because they cannot solve ARC-AGI-3 without a "harness"? A theoretical text-only superintelligent LLM could prove the Riemann hypothesis but fail ARC-AGI-3 and won't even be AGI according to this benchmark...

Think of it as spatial input, not visual. Blind people do have spatial inputs, and high spatial intelligence.

Re: ARC-AGI-3

#338

Post by winwang »

fc417fc802 wrote: post #296
LordDragonfang wrote: post #270

Earlier quoted context omitted.

> Yet I regularly catch them making errors that a human never would I have yet to see a "error" that modern frontier models make that I could not imagine a human making - average humans are way more error prone than the kind of person who posts here thinks, because the social sorting effects of intelligence are so strong you almost never actually interact with people more than a half standard deviation away. (The one…

> you almost never actually interact with people more than a half standard deviation away I wasn't talking about the average person there but rather those who could also craft the high undergrad to low grad level explanations I referred to. > This has not been a remotely credible claim for at least the past six months Well it's happened to me within the past six months (actually within the past month) so I don't know…

How much of this is expectations setting by the heights models reach? i.e. of we could assess a consistent floor of model performance in a vacuum, would we say it's better at "AGI" than the bottom 0.1% of humans?

Re: ARC-AGI-3

#339

Post by daveguy »

ACCount37 wrote: post #63
ZeWaka wrote: post #57

Earlier quoted context omitted.

Just finished it, 8/8. I mostly approached it by winging it and shuffling things around that looked good and like it was approaching the goal, since there's plenty of time to finish. I still don't quite understand the exact mirroring rules at play.

You control the mirroring by moving the axis, they're what reflects your shapes. So my first move was always to identify the symmetries in the target shape, and position the axis accordingly.

This is the correct strategy for this particular game (center the mirrors between the yellow squares, move the black squares). I didn't realize it until about round 6 or 7.

Re: ARC-AGI-3

#340

Post by daveguy »

vonneumannstan wrote: post #300

It's getting pretty old now when Francois Chollet puts out a new ARC challenge, claims definitively that no system is going to crack it without being full blown AGI, the benchmark gets saturated in a few months, he claims the systems definitely aren't AGI then puts out a new challenge that no non AGI system can clear and a few months later.... etc. etc.

Chollet literally never says that. Quite the opposite. He says that AIs are currently abysmally bad at the skills this benchmark tests. An AGI should be able to do this, but doing this doesn't mean it's AGI. He has been very clear about that. I suggest you go back and (re)read the intro ARC-AGI paper.

No system can crack these out of the box (like humans can) because we don't have AGI.

Post reply on HN