Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

901–910 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#901

Post by mistermann »

qt31415926 wrote: post #129
phillipcarter wrote: post #53

Earlier quoted context omitted.

As excited as I am by this, I still feel like this is still just a small approximation of a small chunk of human reasoning ability at large. o3 (and whatever comes next) feels to me like it will head down the path of being a reasoning coprocessor for various tasks. But, still, this is incredibly impressive.

Which parts of reasoning do you think is missing? I do feel like it covers a lot of 'reasoning' ground despite its on the surface simplicity

Optimal phenomenological reasoning is going to be a tough nut to crack.

Luckily we don't know the problem exists, so in a cultural/phenomenological sense it is already cracked.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#902

Post by [deleted] »

tikkun wrote: post #844
tymonPartyLate wrote: post #841

Isn’t this like a brute force approach? Given it costs $ 3000 per task, thats like 600 GPU hours (h100 at Azure) In that amount of time the model can generate millions of chains of thoughts and then spend hours reviewing them or even testing them out one by one. Kind of like trying until something sticks and that happens to solve 80% of ARC. I feel like reasoning works differently in my brain. ;)

They're only allowed 2-3 guesses per problem. So even though yes it generates many candidates, it can't validate them - it doesn't have tool use or a verifier, it submits the best 2-3 guesses. https://www.lesswrong.com/posts/Rdwui3wHxCeKb7feK/getting-50...

[deleted]

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#903

Post by jeremyjh »

tim333 wrote: post #853
6gvONxR4sf7o wrote: post #222

Earlier quoted context omitted.

Human performance is much closer to 100% on this, depending on your human. It's easy to miss the dot in the corner of the headline graph in TFA that says "STEM grad."

A fair comparison might be average human. The average human isn't a STEM grad. It seems STEM grad approximately equals an IQ of 130. https://www.accommodationforstudents.com/student-blog/the-su... From a post elsewhere the scores on ARC-AGI-PUB are approx average human 64%, o3 87%. https://news.ycombinator.com/item?id=42474659 Though also elsewhere, o3 seems very expensive to operate. You could probably hire a PhD re…

Why would an average human be more fair than a trained human? The model is trained.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#904

Post by figure8 »

I have a very naive question.

Why is the ARC challenge difficult but coding problems are easy? The two examples they give for ARC (border width and square filling) are much simpler than pattern awareness I see simple models find in code everyday.

What am I misunderstanding? Is it that one is a visual grid context which is unfamiliar?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#905

Post by jeremyjh »

strangescript wrote: post #843
tymonPartyLate wrote: post #841

Isn’t this like a brute force approach? Given it costs $ 3000 per task, thats like 600 GPU hours (h100 at Azure) In that amount of time the model can generate millions of chains of thoughts and then spend hours reviewing them or even testing them out one by one. Kind of like trying until something sticks and that happens to solve 80% of ARC. I feel like reasoning works differently in my brain. ;)

"We have created artificial super intelligence, it has solved physics!" "Well, yeah, but its kind of expensive" -- this guy

Please do show me a novel result in physics from any LLM. You think "this guy" is stupid because he doesn't extrapolate from this $2MM test that nearly reproduces the work of a STEM graduate to a super intelligence that has already solved physics. Maybe you've got it backwards.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#906

Post by sn0wr8ven »

Incredibly impressive. Still can't really shake the feeling that this is o3 gaming the system more than it is actually being able to reason. If the reasoning capabilities are there, there should be no reason why it achieves 90% on one version and 30% on the next. If a human maintains the same performance across the two versions, an AI with reason should too.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#907

Post by ItsMattyG »

figure8 wrote: post #904

I have a very naive question. Why is the ARC challenge difficult but coding problems are easy? The two examples they give for ARC (border width and square filling) are much simpler than pattern awareness I see simple models find in code everyday. What am I misunderstanding? Is it that one is a visual grid context which is unfamiliar?

Francois'(the creator of ARC-AGI benchmark) whole point was that while they look the same, they're not. Coding is solving a familiar pattern in the same way (and fails when it' s NOT doing that, it just looks like it doesn't happen because it's seen SO MANY patterns in code). But the point of Arc AGI is to make each problem have to generalize in some new ay.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#908

Post by daveguy »

agumonkey wrote: post #838
bluecoconut wrote: post #201

Earlier quoted context omitted.

some other imporant quotes: "Average human off the street: 70-80%. STEM college grad: >95%. Panel of 10 random humans: 99-100%" -@fchollet on X So, considering that the $3400/task system isn't able to compete with STEM college grad yet, we still have some room (but it is shrinking, i expect even more compute will be thrown and we'll see these barriers broken in coming years) Also, some other back of envelope calculat…

who in this field is anticipating impact of near AGI for society ? maybe i'm too anxious but not planning for potential workless life seems dangerous (but maybe i'm just not following the right groups)

AGI would have a major impact on human work. Currently the hype is much greater than the reality. But it looks like we are starting to see some of the components of an AGI and that is cause for discussion of impact, but not panicked discussion. Even the chatbot customer service has to be trained on the domain. Still it is most useful in a few specific ways:

Routing to the correct human support

Providing FAQ level responses to the most common problems.

Providing a second opinion to the human taking the call.

So, even this most relevant domain for the technology doesn't eliminate human employment (because it's just not flexible or reliable enough yet).

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#909

Post by demirbey05 »

sn0wr8ven wrote: post #906

Incredibly impressive. Still can't really shake the feeling that this is o3 gaming the system more than it is actually being able to reason. If the reasoning capabilities are there, there should be no reason why it achieves 90% on one version and 30% on the next. If a human maintains the same performance across the two versions, an AI with reason should too.

I am not expert in llm reasoning but I think because of RL. You cannot use AlphaZero to play other games.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#910

Post by GaggiX »

sn0wr8ven wrote: post #906

Incredibly impressive. Still can't really shake the feeling that this is o3 gaming the system more than it is actually being able to reason. If the reasoning capabilities are there, there should be no reason why it achieves 90% on one version and 30% on the next. If a human maintains the same performance across the two versions, an AI with reason should too.

Humans and AIs are different, the next benchmark would be build so that it emphasize the weak points of current AI models where a human is expected to perform better, but I guess you can also make a benchmark that is the opposite, where humans struggle and o3 has an easy time.
Post reply on HN