Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

111–120 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#111

Post by m3kw9 »

croemer wrote: post #93

The programming task they gave o3-mini high (creating Python server that allows chatting with OpenAI API and run some code in terminal) didn't seem very hard? Strange choice of example for something that's claimed to be a big step forwards. YT timestamped link: https://www.youtube.com/watch?v=SKBG1sqdyIU&t=768s (thanks for the fixed link @photonboom) Updated: I gave the task to Claude 3.5 Sonnet and it worked first s…

I would say they didn’t need to demo anything, because if you are gonna use the output code live on a demo it may make compile errors and then look stupid trying to fix it live

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#113

Post by 85392_school »

hammock wrote: post #96
cryptoegorophy wrote: post #76

Earlier quoted context omitted.

What’s interesting is it might be very close to human intelligence than some “alien” intelligence, because after all it is a LLM and trained on human made text, which kind of represents human intelligence.

In that vein, perhaps the delta between o3 @ 87.5% and Human @ 85% represents a deficit in the ability of text to communicate human reasoning. In other words, it's possible humans can reason better than o3, but cannot articulate that reasoning as well through text - only in our heads, or through some alternative medium.

I wonder how much of an effect amount of time to answer has on human performance.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#114

Post by lawlessone »

sigmoid10 wrote: post #64
zebomon wrote: post #19

My initial impression: it's very impressive and very exciting. My skeptical impression: it's complete hubris to conflate ARC or any benchmark with truly general intelligence. I know my skepticism here is identical to moving goalposts. More and more I am shifting my personal understanding of general intelligence as a phenomenon we will only ever be able to identify with the benefit of substantial retrospect. As it is…

These comments are getting ridiculous. I remember when this test was first discussed here on HN and everyone agreed that it clearly proves current AI models are not "intelligent" (whatever that means). And people tried to talk me down when I theorised this test will get nuked soon - like all the ones before. It's time people woke up and realised that the old age of AI is over. This new kind is here to stay and it wil…

Failing the test may prove the AI is not intelligent. Passing the test doesn't necessarily prove it is.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#115

Post by photonboom »

croemer wrote: post #93

The programming task they gave o3-mini high (creating Python server that allows chatting with OpenAI API and run some code in terminal) didn't seem very hard? Strange choice of example for something that's claimed to be a big step forwards. YT timestamped link: https://www.youtube.com/watch?v=SKBG1sqdyIU&t=768s (thanks for the fixed link @photonboom) Updated: I gave the task to Claude 3.5 Sonnet and it worked first s…

heres the right timestamp: https://www.youtube.com/watch?v=SKBG1sqdyIU&t=768s

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#116

Post by billyp-rva »

hansonkd wrote: post #34
zebomon wrote: post #19

My initial impression: it's very impressive and very exciting. My skeptical impression: it's complete hubris to conflate ARC or any benchmark with truly general intelligence. I know my skepticism here is identical to moving goalposts. More and more I am shifting my personal understanding of general intelligence as a phenomenon we will only ever be able to identify with the benefit of substantial retrospect. As it is…

It doesn't need to be general intelligence or perfectly map to human intelligence. All it needs to be is useful. Reading constant comments about LLMs can't be general intelligence or lack reasoning etc, to me seems like people witnessing the airplane and complaining that it isn't "real flying" because it isn't a bird flapping its wings (a large portion of the population held that point of view back then). It doesn't…

> It doesn't need to be general intelligence or perfectly map to human intelligence.

> All it needs to be is useful.

Computers were already useful.

The only definition we have for "intelligence" is human (or, generally, animal) intelligence. If LLMs aren't that, let's call it something else.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#117

Post by torginus »

maxdoop wrote: post #8

How much longer can I get paid $150k to write code ?

Well, considering they floated the $2000 subscription idea, and they still haven't revealed everything, they could still introduce the $2k sub with o3+agents/tool use, which means, till about next week.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#118

Post by antirez »

obblekk wrote: post #42

Human performance is 85% [1]. o3 high gets 87.5%. This means we have an algorithm to get to human level performance on this task. If you think this task is an eval of general reasoning ability, we have an algorithm for that now. There's a lot of work ahead to generalize o3 performance to all domains. I think this explains why many researchers feel AGI is within reach, now that we have an algorithm that works. Congrat…

NNs are not algorithms.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#120

Post by InkCanon »

spaceman_2020 wrote: post #31

Just as an aside, I've personally found o1 to be completely useless for coding. Sonnet 3.5 remains the king of the hill by quite some margin

I just asked o1 a simple yes or no question about x86 atomics and it did one of those A or B replies. The first answer was yes, the second answer was no.
Post reply on HN