Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

851–860 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#851

Post by freehorse »

strangescript wrote: post #843
tymonPartyLate wrote: post #841

Isn’t this like a brute force approach? Given it costs $ 3000 per task, thats like 600 GPU hours (h100 at Azure) In that amount of time the model can generate millions of chains of thoughts and then spend hours reviewing them or even testing them out one by one. Kind of like trying until something sticks and that happens to solve 80% of ARC. I feel like reasoning works differently in my brain. ;)

"We have created artificial super intelligence, it has solved physics!" "Well, yeah, but its kind of expensive" -- this guy

The problem is not that it is expensive, but that, most likely, it is not superintelligence. Superintelligence is not exploring the problem space semi-blindly, if the thounsands $$$ per task are actually spent for that. There is a reason the actual ARC-AGI prize requires efficiency, because the point is not "passing the test" but solving the framing problem of intelligence.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#852

Post by r-zip »

OldGreenYodaGPT wrote: post #621
Engineering-MD wrote: post #575

Can I just say what a dick move it was to do this as a 12 days of Christmas. I mean to be honest I agree with the arguments this isn’t as impressive as my initial impression, but they clearly intended it to be shocking/a show of possible AGI, which is rightly scary. It feels so insensitive to that right before a major holiday when the likely outcome is a lot of people feeling less secure in their career/job/life. Tha…

Blaming OpenAI for progress is like blaming a calendar for Christmas—it’s not the timing, it’s your unwillingness to adapt

Unwillingness to adapt to the destruction of the middle class and knowledge work is pretty reasonable tbh.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#853

Post by tim333 »

6gvONxR4sf7o wrote: post #222
obblekk wrote: post #42

Human performance is 85% [1]. o3 high gets 87.5%. This means we have an algorithm to get to human level performance on this task. If you think this task is an eval of general reasoning ability, we have an algorithm for that now. There's a lot of work ahead to generalize o3 performance to all domains. I think this explains why many researchers feel AGI is within reach, now that we have an algorithm that works. Congrat…

Human performance is much closer to 100% on this, depending on your human. It's easy to miss the dot in the corner of the headline graph in TFA that says "STEM grad."

A fair comparison might be average human. The average human isn't a STEM grad. It seems STEM grad approximately equals an IQ of 130. https://www.accommodationforstudents.com/student-blog/the-su...

From a post elsewhere the scores on ARC-AGI-PUB are approx average human 64%, o3 87%. https://news.ycombinator.com/item?id=42474659

Though also elsewhere, o3 seems very expensive to operate. You could probably hire a PhD researcher for cheaper.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#854

Post by amelius »

ryoshu wrote: post #607
nopinsight wrote: post #125

Let me go against some skeptics and explain why I think full o3 is pretty much AGI or at least embodies most essential aspects of AGI. What has been lacking so far in frontier LLMs is the ability to reliably deal with the right level of abstraction for a given problem. Reasoning is useful but often comes out lacking if one cannot reason at the right level of abstraction. (Note that many humans can't either when they…

Ask o3 is P=NP?

It will just answer with the current consensus on the matter.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#855

Post by spaceman_2020 »

vessenes wrote: post #86
spaceman_2020 wrote: post #31

Just as an aside, I've personally found o1 to be completely useless for coding. Sonnet 3.5 remains the king of the hill by quite some margin

To fill this out, I find o1-pro (and -preview when it was live) to be pretty good at filling in blindspots/spotting holistic bugs. I use Claude for day to day, and when Claude is spinning, o1 often can point out why. It's too slow for AI coding, and I agree that at default its responses aren't always satisfying. That said, I think its code style is arguably better, more concise and has better patterns -- Claude needs…

Claude has a habit of sometimes just getting “lost”

Like I’ll have it a project in Cursor and it will spin up ready to use components that use my site style, reference existing components, and follow all existing patterns

Then on some days, it will even forget what language the project is in and start giving me python code for a react project

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#856

Post by spaceman_2020 »

bitbuilder wrote: post #301
spaceman_2020 wrote: post #31

Just as an aside, I've personally found o1 to be completely useless for coding. Sonnet 3.5 remains the king of the hill by quite some margin

I find myself hoping between o1 and Sonnet pretty frequently these days, and my personal observation is that the quality of output from o1 scales more directly to the quality of the prompting you're giving it. In a way it almost feels like it's become too good at following instructions and simply just takes your direction more literally. It doesn't seem to take the initiative of going the extra mile of filling in the…

Have you found any tool or guide for writing better o1 prompts? This isn’t the first time I’ve heard this about o1 but no one seems to know how to prompt it

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#857

Post by lagrange77 »

OldGreenYodaGPT wrote: post #621
Engineering-MD wrote: post #575

Can I just say what a dick move it was to do this as a 12 days of Christmas. I mean to be honest I agree with the arguments this isn’t as impressive as my initial impression, but they clearly intended it to be shocking/a show of possible AGI, which is rightly scary. It feels so insensitive to that right before a major holiday when the likely outcome is a lot of people feeling less secure in their career/job/life. Tha…

Blaming OpenAI for progress is like blaming a calendar for Christmas—it’s not the timing, it’s your unwillingness to adapt

Wow, you just solved the ethics of technology in a one liner. Impressive.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#858

Post by javaunsafe2019 »

dmead wrote: post #787
obblekk wrote: post #42

Human performance is 85% [1]. o3 high gets 87.5%. This means we have an algorithm to get to human level performance on this task. If you think this task is an eval of general reasoning ability, we have an algorithm for that now. There's a lot of work ahead to generalize o3 performance to all domains. I think this explains why many researchers feel AGI is within reach, now that we have an algorithm that works. Congrat…

This is so strange. people think that an llm trained on programming questions and docs can do mundane tasks like this means intelligent? Come on. It really calls into question two things. 1. You don't know what you're talking about about. 2. You have a perverse incentive to believe this such that you will preach it to others and elevate some job salary range or stock. Either way, not a good look.

This

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#859

Post by zug_zug »

This is a lot of noise around what's clearly not even an order of magnitude to the way to AGI.

Here's my AGI test - Can the model make a theory of AGI validation that no human has suggested before, test itself to see if it qualifies, iterate, read all the literature, and suggest modifications to its own network to improve its performance?

That's what a human-level performer would do.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#860

Post by tymonPartyLate »

strangescript wrote: post #843
tymonPartyLate wrote: post #841

Isn’t this like a brute force approach? Given it costs $ 3000 per task, thats like 600 GPU hours (h100 at Azure) In that amount of time the model can generate millions of chains of thoughts and then spend hours reviewing them or even testing them out one by one. Kind of like trying until something sticks and that happens to solve 80% of ARC. I feel like reasoning works differently in my brain. ;)

"We have created artificial super intelligence, it has solved physics!" "Well, yeah, but its kind of expensive" -- this guy

Haha. Hopefully you’re right and solving the ARC puzzle translates to solving all of physics. I just remain skeptical about the OpenAI hype. They have a track record of exaggerating the significance of their releases and their impact on humanity.
Post reply on HN