Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC-AGI-3

arcprize.org

Post reply on HN
Print view View original

241–250 of 394 posts

Re: ARC-AGI-3

#241

Post by keiferski »

nananana9 wrote: post #237
keiferski wrote: post #214

Earlier quoted context omitted.

The gap is important because of its special and devastating economic consequences. When the gap becomes truly zero, all human knowledge work is replaceable. From there, with robots, its a short step to all work is replaceable. I don’t know why statements like this are just taken as gospel fact. There are plenty of economic activities which do not disappear even if an AI can do them. Here’s one: I support certain arti…

Very few artists or aspiring artists make enough money from their art to make a living -- even now, when the average person has a job and at least some disposable money and can support artists. This % will not get higher if we get 1000x more artists, and 1000x less employed people working in the general economy. You can't get deep experience in any industry if there's a machine that can do the entry-level work for a…

You can't get deep experience in any industry if there's a machine that can do the entry-level work for a fraction of the cost you can.

As I said, not all types of jobs are set up this way. Pure knowledge ones, sure. But ones dependent on context are not going to have this elimination of entry-level work in the first place.

and we get 1000 robots for every person in the economy, they're in a better position than you to produce said valuable secret information.

Again, no, they aren't, because certain types of information are not merely a question of computational power.

There is this constant assumption that all knowledge is just a math problem to solve, ergo AI will eventually solve it. That isn't how information actually functions in the real world.

Re: ARC-AGI-3

#242

Post by keiferski »

yes_man wrote: post #239
keiferski wrote: post #235

Earlier quoted context omitted.

No, by continuity I mean literally moment to moment. Sorry if I didn’t clarify that. Even people with amnesia are still present moment to moment. As far as I know there are no things that we call conscious which have zero continuity. I think consciousness is not an abstract property in the world, therefore it’s tied to certain types of entities. Therefore an AI is not going to be “conscious” in the way an animal is,…

Doesn’t the LLM experience discrete continuity every time it infers the next token? > I think consciousness is not an abstract property in the world, therefore it’s tied to certain types of entities. Therefore an AI is not going to be “conscious” This pretty much sums up most arguments for why LLMs aren’t conscious: ”I think” followed by assertions. Only real argument is: science doesn’t quantify consciousness, we ca…

I don't think you really understood my point, because you didn't reply to it at all.

I am making a linguistic argument. AI may get as sophisticated as "traditional" consciousness. But this is only "real" consciousness if you are a functionalist and think the output is all that matters.

I disagree and think that "flying" is just a weak generic word that describes both planes and birds, and not some kind of ultimate Platonic Ideal in the world.

Ditto for AI consciousness: it may develop to be as complex as traditional animal consciousness, but I'm not a functionalist, and think it's merely a lack of our sophisticated language that makes us think it's the same thing. It's not. Planes PlaneFly through the air, while birds BirdFly.

Re: ARC-AGI-3

#243

Post by esailija »

Tiberium wrote: post #35

https://x.com/scaling01 has called out a lot of issues with ARC-AGI-3, some of them (directly copied from tweets, with minimal editing): - Human baseline is "defined as the second-best first-run human by action count". Your "regular people" are people who signed up for puzzle solving and you don't compare the score against a human average but against the second best human solution - The scoring doesn't tell you how m…

It actually makes sense. For any task it is completely trivial for anyone to become better than >80% humans and still easy to be better than >95%. The only problem is motivation not intelligence.

Re: ARC-AGI-3

#246

Post by baalimago »

You can tell it's an AI by it not becoming utterly by playing the "game". I could personally not stand any more than the first level.

Re: ARC-AGI-3

#247

Post by yes_man »

keiferski wrote: post #242
yes_man wrote: post #239

Earlier quoted context omitted.

Doesn’t the LLM experience discrete continuity every time it infers the next token? > I think consciousness is not an abstract property in the world, therefore it’s tied to certain types of entities. Therefore an AI is not going to be “conscious” This pretty much sums up most arguments for why LLMs aren’t conscious: ”I think” followed by assertions. Only real argument is: science doesn’t quantify consciousness, we ca…

I don't think you really understood my point, because you didn't reply to it at all. I am making a linguistic argument. AI may get as sophisticated as "traditional" consciousness. But this is only "real" consciousness if you are a functionalist and think the output is all that matters. I disagree and think that "flying" is just a weak generic word that describes both planes and birds, and not some kind of ultimate Pl…

I see it as LLMs, AI, whatever, can be intelligent enough to emulate consciousness, appear outside as if it were. But that is not proof it really has a qualia, an experience of existing.

All I am saying we should stop being so certain they are not conscious, since we lack a solid, quantifiable model for consciousness.

Re: ARC-AGI-3

#248

Post by Marha01 »

cubefox wrote: post #227
Marha01 wrote: post #221

Earlier quoted context omitted.

Don't you see the massive problem with requiring visual input? Are blind people not intelligent because they cannot solve ARC-AGI-3 without a "harness"? A theoretical text-only superintelligent LLM could prove the Riemann hypothesis but fail ARC-AGI-3 and won't even be AGI according to this benchmark...

Well, it would be AGI if you could connect a camera to it to solve it, similar to how blind people would be able to solve it if you restored their eyesight. But if the lack of vision is a fundamental limitation of their architecture, then it seems more fair not to call them AGI.

People blind from birth literally lack the neural circuits to comprehend visual data. Are they not intelligent?

Re: ARC-AGI-3

#249

Post by pawelk411 »

levocardia wrote: post #205
causal wrote: post #84

Earlier quoted context omitted.

Thanks, I mostly agree with your approach except for one thing: eyesight feels like a "harness" that humans get to use and LLMs do not. I'm guessing you did not pass the human testers JSON blobs to work with, and suspect they would also score 0% without the eyesight and visual cortex harness to their reasoning ability.

My sense is that a powerful enough AI would have the sense to think something like "ah, this sounds like a video game! Let me code up an interactive GUI, test it for myself, then use it to solve these puzzles..." and essentially self-harness (the way you would if you were reading a geometry problem, by drawing it out on paper).

Yeah but thats literally above ASI, let alone AGI. Average human scores <1% on this bench, opus scores 97.1% when given an actual vision access, which means agi was long ago achieved

Re: ARC-AGI-3

#250

Post by pawelk411 »

Veedrac wrote: post #176
fchollet wrote: post #65

Earlier quoted context omitted.

Francois here. The scoring metric design choices are detailed in the technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf - the metric is meant to discount brute-force attempts and to reward solving harder levels instead of the tutorial levels. The formula is inspired by the SPL metric from robotics navigation, it's pretty standard, not a brand new thing. We tested ~500 humans over 90 minute se…

There's a very simple solution to this problem here. Instead of wink-wink-nudge-nudge implying that 100% is 'human baseline', calculate the median human score from the data you already have and put it on that chart.

Its below 1% lmao
Post reply on HN