Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC-AGI-3

arcprize.org

Post reply on HN
Print view View original

311–320 of 394 posts

Re: ARC-AGI-3

#311

Post by daveguy »

vova_hn2 wrote: post #283
pawelk411 wrote: post #249

Earlier quoted context omitted.

Yeah but thats literally above ASI, let alone AGI. Average human scores <1% on this bench, opus scores 97.1% when given an actual vision access, which means agi was long ago achieved

> opus scores 97.1% when given an actual vision access Do you have a source for this? I would be very curious to see how top models do with vision.

No, there is no source for this. Opus is scoring around 1% just like all the other frontier models. It would be fairly trivial to add a renderer intermediary. And if it improves to 97+%... Then you would get a huge cut of $2 million dollars. The assertion that Opus gets 97% if you just give it a gui is completely bogus.

Re: ARC-AGI-3

#312

Post by daveguy »

pawelk411 wrote: post #236
fchollet wrote: post #65

Earlier quoted context omitted.

Francois here. The scoring metric design choices are detailed in the technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf - the metric is meant to discount brute-force attempts and to reward solving harder levels instead of the tutorial levels. The formula is inspired by the SPL metric from robotics navigation, it's pretty standard, not a brand new thing. We tested ~500 humans over 90 minute se…

New benchmark idea: 20 questions of guess the number 1-10, with different answers. We run this on 10,000 humans, take best score. Then we take 50 ai attempts, but take the worst attempt so "worst case scenarior robustness or so". We also discard questions where human failed but ai passed because uhhh reasons... Then we also take the final relative score to the power of 100 so that the benchmark punishes bad answers o…

This is a gross misrepresentation of the scoring process.

Re: ARC-AGI-3

#313

Post by rolandhvar »

red75prime wrote: post #219
fc417fc802 wrote: post #94

Earlier quoted context omitted.

Those are supposed to be issues? After reading your list my impression of ARC-AGI has gone up rather than down. All of those things seem like the right way to go about this.

No, those aren't issues. But it's good to know the meaning of those numbers we get. For example, 25% is about the average human level (on this category of problems). 100% is either top human level or superhuman level or the information-theoretically optimal level.

Sure, but, aim for the stars and you hit the moon right? Like fundamentally who cares? For the purpose of an AGI benchmark I'd argue you'd rather err on the side of being more intelligent and counting that as less intelligent than vice versa.

Re: ARC-AGI-3

#314

Post by maldev »

Marha01 wrote: post #248
cubefox wrote: post #227

Earlier quoted context omitted.

Well, it would be AGI if you could connect a camera to it to solve it, similar to how blind people would be able to solve it if you restored their eyesight. But if the lack of vision is a fundamental limitation of their architecture, then it seems more fair not to call them AGI.

People blind from birth literally lack the neural circuits to comprehend visual data. Are they not intelligent?

I think I can confidently say they are not visually intelligent at all.

If you were phrasing things to quantify intelligence, you would have a visual intelligence pillar. And they would not pass that pillar. It doesn't make them dysfunctional or stupid, but visual intelligence is a key part of human intelligence.

Re: ARC-AGI-3

#315

Post by Marha01 »

degamad wrote: post #261
Marha01 wrote: post #220

Earlier quoted context omitted.

Does this mean blind people are not intelligent?

Blind people do function within the context of a human-centric world, though, so they would qualify as intelligent.

Yes, but they use various "harnesses" to do so (dog guides, text to speech software, assistance of other humans when needed..). Why can't AI?

Re: ARC-AGI-3

#316

Post by slfnflctd »

imiric wrote: post #129
daemonologist wrote: post #52

Earlier quoted context omitted.

Or the classic from Dijkstra ( https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867... ): > even Alan M. Turing allowed himself to be drawn into the discussion of the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim. (I am of the opinion that the thinking question is in fact a bit more relevant than the swimming one, but…

I've come across that quote several times, and reach the same conclusion as you. While I share Dijkstra's sentiment that "thinking machines" is largely a marketing term we've been chasing for decades, and this new cycle is no different, it's still worth discussing and... thinking about. The implications of a machine that can approximate or mimic human thinking are far beyond the implications of a machine that can app…

> The implications of a machine that can approximate or mimic human thinking are far beyond the implications of a machine that can approximate or mimic swimming

It seems to me like too many people are missing this point.

Modern philosophy tells us we can't even be certain whether other humans are conscious or not. The 'hard problem', p-zombies, etcetera.

The fact that current LLMs can convince many actual humans that they are conscious (whether they are or not is irrelevant, I lean toward not but whatever) has implications which aren't being discussed enough. If you teach a kid that they can treat this intelligent-seeming 'bot' like an object with no mind, is it not plausible that they might then go on to feel they can treat other kids who are obviously far less intelligent like objects as well? Seriously, we need to be talking more about this.

One of the most important questions about AI agents in my opinion should be, "can they suffer?", and if you can't answer that with a definitive "absolutely not" then we are suddenly in uncharted waters, ethically speaking. They can certainly act like they're suffering (edit: which, when witnessed by a credulous human audience, could cause them to suffer!). I think we should be treading much more carefully than many of us are.

Re: ARC-AGI-3

#319

Post by famouswaffles »

daveguy wrote: post #309
pawelk411 wrote: post #240

Earlier quoted context omitted.

Because it is. Opus 4.6 jumps from 0.0% to 97.1% when given visual input

Source? I haven't seen anything like that for ARC-AGI performance. Also, if it makes that big of a difference, then make a renderer for your agent that looks like the web page and have it solve them in the graphical interface and funnel the results to the API. I guarantee you won't get better performance, because the AGI is going to have to "understand" the raw data can be represented as a 2D matrix regardless of whe…

That score is in the arc technical paper [1]. It's the full benchmark score using this harness [2] (which is just open code with read, grep, bash tools).

This is already a solved benchmark. That's why scoring is so convoluted and a self proclaimed Agent benchmark won't allow basic agent tools. ARC has always been a bit of a nothing burger of a benchmark but this takes the cake.

[1] https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf

[2] https://blog.alexisfox.dev/arcagi3

Re: ARC-AGI-3

#320

Post by nearbuy »

fc417fc802 wrote: post #253
nearbuy wrote: post #228

Earlier quoted context omitted.

Even dumb humans are considered to have general intelligence. If the bar is having to outdo the median human, then 50% of humans don't have general intelligence.

Not true. We don't have a good definition for intelligence - it's very much an I'll know it when I see it sort of thing. Frontier models are reliably providing high undergraduate to low graduate level customized explanations of highly technical topics at this point. Yet I regularly catch them making errors that a human never would and which betray a fatal lack of any sort of mental model. What are we supposed to make…

> Not true.

It's certainly true. By definition. If the bar for general intelligence is being smarter than the median human, 50% of people won't reach the threshold for general intelligence. (And if the bar is beating the median in every cognitive test, then a much smaller fraction of people would qualify.)

People don't have a consistent definition of AGI, and the definitions have changed over the past couple years, but I think most people have settled on it meaning at least as smart as humans in every cognitive area. But that has to be compared to dumb people, not median. We don't want to say that regular people don't have general intelligence.

Post reply on HN