Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC-AGI-3

arcprize.org

Post reply on HN
Print view View original

361–370 of 394 posts

Re: ARC-AGI-3

#361

Post by daveguy »

fc417fc802 wrote: post #347
daveguy wrote: post #343

Earlier quoted context omitted.

It may have been tested on the full set, but the score you quote is for a single game environment. Not the full public set. That fact is verbatim in what you responded to and vbarrielle quoted. It scored 97% in one game , and 0% in another game. The full prelude to what vbarrielle quoted, the last sentence of which you left out, was: > We then tested the harnesses on the full public set (which researchers did not hav…

> intelligence for those specific games is baked into the harness This is your claim but the other commenter claims the harness consists only of generic tools. What's the reality? I also encountered confusion about this exact issue in another subthread. I had thought that generic tooling was allowed but others believed the benchmark to be limited to ingesting the raw text directly from the API without access to any a…

1) Pointing out what tools to use is part of the intelligence that LLMs aren't great at.

2) one of the tools is a path finding algorithm. A big improvement/crutch over a regular LLM that has no such capability.

You'd think if LLMs are intelligent they'd be able to determine that a path finding algorithm is necessary and have a sub agent code it up real quick. But apparently they just can't do that without humans stepping in to make it a standard tool for them.

Here's the paper on what they did for the Duke Harness:

https://blog.alexisfox.dev/arcagi3

Re: ARC-AGI-3

#362

Post by neop1x »

Stevvo wrote: post #8

Maybe I'm just not intelligent, but I gave it a couple of minutes and couldn't figure out WTF the game wants from you or how to win it.

Exactly my experience. It has nothing to do wirh some AGI testing. It is just some kind of useless weird game.

Re: ARC-AGI-3

#363

Post by ACCount37 »

culi wrote: post #91
Stevvo wrote: post #8

Maybe I'm just not intelligent, but I gave it a couple of minutes and couldn't figure out WTF the game wants from you or how to win it.

It's not an IQ test. Just a way to assess your ability to generalize rules. If you've played previous rounds you kinda get used to the "style" of these games and it gets easier

That's exactly what "an IQ test" is.

"Raven's progressive matrices" is "infer and generalize rules". Performance there also improves once "you kinda get used to the style", which is why training for IQ tests can improve human performance on IQ tests, including on unseen examples. This is well known and well documented.

Re: ARC-AGI-3

#364

Post by famouswaffles »

daveguy wrote: post #361
fc417fc802 wrote: post #347

Earlier quoted context omitted.

> intelligence for those specific games is baked into the harness This is your claim but the other commenter claims the harness consists only of generic tools. What's the reality? I also encountered confusion about this exact issue in another subthread. I had thought that generic tooling was allowed but others believed the benchmark to be limited to ingesting the raw text directly from the API without access to any a…

1) Pointing out what tools to use is part of the intelligence that LLMs aren't great at. 2) one of the tools is a path finding algorithm. A big improvement/crutch over a regular LLM that has no such capability. You'd think if LLMs are intelligent they'd be able to determine that a path finding algorithm is necessary and have a sub agent code it up real quick. But apparently they just can't do that without humans step…

>You'd think if LLMs are intelligent they'd be able to determine that a path finding algorithm is necessary and have a sub agent code it up real quick.

ARC 3 doesn't allow that so.

>Here's the paper on what they did for the Duke Harness: https://blog.alexisfox.dev/arcagi3

Yeah, and the tools are general, not 'baked into the harness by the humans who coded it for this specific challenge.'

Re: ARC-AGI-3

#365

Post by famouswaffles »

daveguy wrote: post #343
famouswaffles wrote: post #323

Earlier quoted context omitted.

The harness was designed with the preview, but no it was still tested on the full public set in that environment. You can run the benchmark in different 'environments' though it's unclear what the difference between them is. >We then tested the harnesses on the full public set (which researchers did not have access to at the time)

It may have been tested on the full set, but the score you quote is for a single game environment. Not the full public set. That fact is verbatim in what you responded to and vbarrielle quoted. It scored 97% in one game , and 0% in another game. The full prelude to what vbarrielle quoted, the last sentence of which you left out, was: > We then tested the harnesses on the full public set (which researchers did not hav…

>The point of ARC-AGI is to test the intelligence of AI systems in novel, but simple, environments.

The point is whatever Francois wants it to be.

>Having a human give it more powerful tools in a harness defeats the purpose.

Why does it defeat the purpose? Restricting the tools available is an arbitrary constraint. The Duke harness is a few basic tools. What's the problem ? In what universe would any AI Agent worth its salt not have access to read, grep and bash ? If his benchmark was as great and the difference as wide as he claimed, then it simply wouldn't matter if those tools were available. Francois removed access to tools because his benchmark falls apart with them. Simple as.

>You should go back and read the original ARC-AGI paper to see what this is about+.

>Are you upset about the benchmark because frontier LLM models do so poorly exhibiting the ability to generalize when the benchmarks are released?

I’m not upset about anything. I do not care about ARC, and I never have. I think it is a nothingburger of a benchmark: lots of grand claims about AGI, but very little predictive power or practical utility.

When models started climbing FrontierMath, that benchmark actually told us something useful: their mathematical capabilities were becoming materially stronger. And now state-of-the-art systems have helped with real research and even contributed to solving open problems. That is what a good benchmark is supposed to do.

ARC ? Has 0 utility on its own and manages to tell you nothing at the same time.

Unsaturated benchmarks matter because they help show where the state of the art actually is. The value is not “look, the score is low,” but whether the benchmark tells you something real and useful about capability. ARC has always struggled on that front, but 3 has taken that to a new level of useless.

Re: ARC-AGI-3

#366

Post by famouswaffles »

daveguy wrote: post #308
Imnimo wrote: post #211

Earlier quoted context omitted.

So, if you look at the way the scoring works, 100% is the max. For each task, you get full credit if you solve in a number of steps less than or equal to the baseline. If you solve it with more steps, you get points off. But each task is scored independently, and you can't "make up" for solving one slowly by solving another quickly. Like suppose there were only two tasks, each with a baseline score of solving in 100…

The purpose is to benchmark both generality and intelligence. "Making up for" a poor score on one test with an excellent score on another would be the opposite of generality. There's a ceiling based on how consistent the performance is across all tasks.

>"Making up for" a poor score on one test with an excellent score on another would be the opposite of generality.

Really ? This happens plenty with human testing. Humans aren't general ?

The score is convoluted and messy. If the same score can say materially different things about capability then that's a bad scoring methodology.

I can't believe I have to spell this out but it seems critical thinking goes out the window when we start talking about machine capabilities.

Re: ARC-AGI-3

#367

Post by kordlessagain »

fc417fc802 wrote: post #94
Tiberium wrote: post #35

https://x.com/scaling01 has called out a lot of issues with ARC-AGI-3, some of them (directly copied from tweets, with minimal editing): - Human baseline is "defined as the second-best first-run human by action count". Your "regular people" are people who signed up for puzzle solving and you don't compare the score against a human average but against the second best human solution - The scoring doesn't tell you how m…

Those are supposed to be issues? After reading your list my impression of ARC-AGI has gone up rather than down. All of those things seem like the right way to go about this.

What is "this", exactly?

Re: ARC-AGI-3

#368

Post by aogaili »

honestly the most interesting thing about ARC-AGI-3 isn't the 0.25% scores everyone is doomposting about. it's the Duke harness result.

if you give Opus just three generic tools (READ, GREP, BASH with Python) and literally zero game-specific help, it completes all three preview games in 1,069 actions. for comparison, humans do it in like ~900. that's actually insane. it writes its own BFS, builds a grid parser from scratch, and even solves a Lights Out puzzle with Gaussian elimination. all on its own.

i really think the benchmark is testing two different things and just smashing them together. can the model reason about novel interactive environments? yeah, clearly it can. can it do spatial reasoning over a 64x64 grid from raw JSON with zero tools? no. but then again, neither can a human if you ripped out their visual cortex lol.

humans come "pre-installed" with specialized subsystems for this exact stuff: a visual cortex for spatial perception, a hippocampus for persistent memory, etc. these aren't "tools" in Chollet's framing but they're basically identical to what the Duke harness provides. the model is just building its own version of those (Python for the cortex, grep for memory). it just needs the permission to build them.

the real gap the Duke team found isn't perception or memory anyway, it is hypothesis quality. some runs solve vc33 in 441 actions, others just plateau past 1,500. the variance is just down to whether the model commits early to the right explanation of how the game works. that's a way more interesting and targetable finding than just saying "frontier models score below 1%."

Chollet is probably right philsophically that AGI should handle any input format without help. but reporting 0.25% when the actual reasoning gap is in hypothesis formation (not spatial perception) makes the benchmark a way worse progress indicator than it could be imo.

Re: ARC-AGI-3

#369

Post by slfnflctd »

imiric wrote: post #328
slfnflctd wrote: post #316

Earlier quoted context omitted.

> The implications of a machine that can approximate or mimic human thinking are far beyond the implications of a machine that can approximate or mimic swimming It seems to me like too many people are missing this point. Modern philosophy tells us we can't even be certain whether other humans are conscious or not. The 'hard problem', p-zombies, etcetera. The fact that current LLMs can convince many actual humans that…

You lost me there. :) The question of whether the current generation of "AI" can think, whether it is conscious, let alone whether it can suffer(!), is not even worth discussing. It should be obvious to anyone who understands how these tools work that they don't in fact "think", for even the most liberal definition of that term. They're statistical models that can generate useful patterns when fed with vast amounts o…

> The question of whether the current generation of "AI" can think, whether it is conscious, let alone whether it can suffer(!), is not even worth discussing. It should be obvious to anyone who understands how these tools work that they don't in fact "think", for even the most liberal definition of that term.

While I agree with your second sentence here, the first one gives me pause. Why isn't it "worth discussing"? Do you refuse to engage in conversation with all mentally challenged people? Do you avoid all interactions with human children? There are many, many folks living their lives as fully as they can right now who are convinced these things are alive. There are ethical implications to that assumption regardless of whether the things are actually alive, especially when people respond to them as if they are.

We need to have better arguments and refine them for different audiences.

Are you aware of the concept of philosophical zombies? Some of the top minds on the planet are telling us they can't even determine if you or me are conscious and sentient, let alone if a machine is. On the other hand, some of those people's peers are arguing that weather patterns might be conscious (among even more extreme claims). From the standpoint of logic and reason being paramount, we cannot claim to know the answers to these questions. What we can do is discuss the ethical implications of various people coming to different conclusions about them.

Re: ARC-AGI-3

#370

Post by jpadkins »

dgfl wrote: post #200
Real_Egor wrote: post #191

I'll probably be the skeptic here, but: - Take a person who grew up playing video games. They'll pass these tests 100% without even breaking a sweat. - BUT, put a grandmother who has never used a computer in front of this game, and she'll most likely fail completely. Just like an LLM. As soon as models are "natively" trained on a massive dataset of these types of games, they'll easily adapt and start crushing these c…

Isn’t this what AGI is by design? People CAN learn to become good at videogames. Modern LLMs can’t, they have to be retrained from scratch (I consider pre-training to be a completely different process than learning). I also don’t necessarily agree that a grandma would fail. Give her enough motivation and a couple days and she’ll manage these. My main criticism would be that it doesn’t seem like this test allows onlin…

Kids develop video game skills, grandmothers do not. Hypothetically grandmothers develop baking skills, that kids do not (perfectly golden brown cookies). A human intelligence is generally capable of developing video game skills or baking skills, given enough motivation and experience to hone those skills. One test of AGI is if the same system can develop video game skills and baking skills, without having to rebuild the core models... this would demonstrate generalized intelligence.
Post reply on HN