Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

931–940 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#931

Post by farts_mckensy »

rvnx wrote: post #309
neuroelectron wrote: post #167

OpenAI spent approximately $1,503,077 to smash the SOTA on ARC-AGI with their new o3 model semi-private evals (100 tasks): 75.7% @ $2,012 total/100 tasks (~$20/task) with just 6 samples & 33M tokens processed in ~1.3 min/task and a cost of $2012 The “low-efficiency” setting with 1024 samples scored 87.5% but required 172x more compute. If we assume compute spent and cost are proportional, then OpenAI might have just…

It sounds like they essentially brute-forced the solutions ? Ask LLM for answer, answer for LLM to verify the answer. Ask LLM for answer, answer for LLM to verify the answer. Add a bit of randomness. Ask LLM for answer, answer for LLM to verify the answer. Add a bit of randomness. Repeat 5B times (this is what the paper says).

Evolution itself is the ultimate brute-force algorithm—it’s just applied over millennia. Trial and error, coupled with selection and refinement, is the only way to generate novelty when there’s no clear blueprint.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#933

Post by ozten »

demirbey05 wrote: post #909
sn0wr8ven wrote: post #906

Incredibly impressive. Still can't really shake the feeling that this is o3 gaming the system more than it is actually being able to reason. If the reasoning capabilities are there, there should be no reason why it achieves 90% on one version and 30% on the next. If a human maintains the same performance across the two versions, an AI with reason should too.

I am not expert in llm reasoning but I think because of RL. You cannot use AlphaZero to play other games.

Nope. AlphaZero taught itself to play games like chess, shogi, and Go through self-play, starting from random moves. It was not given any strategies or human gameplay data but was provided with the basic rules of each game to guide its learning process.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#934

Post by creer »

dyauspitr wrote: post #479
obblekk wrote: post #42

Human performance is 85% [1]. o3 high gets 87.5%. This means we have an algorithm to get to human level performance on this task. If you think this task is an eval of general reasoning ability, we have an algorithm for that now. There's a lot of work ahead to generalize o3 performance to all domains. I think this explains why many researchers feel AGI is within reach, now that we have an algorithm that works. Congrat…

I’ll believe it when the AI can earn money on its own. I obviously don’t mean someone paying a subscription to use the AI I mean, letting the AI lose on the Internet with only the goal of making money and putting it into a bank account.

You don't think there are already plenty of attempts out there?

When someone is "disinterested enough" to publish though, note the obvious way to launch a new fund or advisor with a good track record: crank out a pile of them, run them one or two years, discard the many losers and publish the one or two top winners. I.E. first you should be suspicious of why it's being published, then of how selected that result is.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#936

Post by neom »

codedokode wrote: post #932

I wonder, what is the main obstacle in making robots for mechanical tasks, like laying bricks, paving a road or working in the shaft? It doesn't look like something that requires lot of mathematical or programming skills, just good vision and manipulators.

https://www.youtube.com/watch?v=K1TrbI0BaaU

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#937

Post by wrsh07 »

figure8 wrote: post #904

I have a very naive question. Why is the ARC challenge difficult but coding problems are easy? The two examples they give for ARC (border width and square filling) are much simpler than pattern awareness I see simple models find in code everyday. What am I misunderstanding? Is it that one is a visual grid context which is unfamiliar?

I expect it largely has to do with "scale"

We have an enormous amount of high quality programming samples. From there it's relatively straightforward to bootstrap (similar to original versions of alphago - start with human games, improve via self play) using leetcode or other problems with a "right answer"

In contrast, the arc puzzles are relatively novel (why? Well, this has to do with the relative utility of solving an arc problem and programmer open source culture)

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#938

Post by xanderlewis »

cornholio wrote: post #916
sn0wr8ven wrote: post #906

Incredibly impressive. Still can't really shake the feeling that this is o3 gaming the system more than it is actually being able to reason. If the reasoning capabilities are there, there should be no reason why it achieves 90% on one version and 30% on the next. If a human maintains the same performance across the two versions, an AI with reason should too.

But does it matter if it "really, really" reasons in the human sense, if it's able to prove some famous math theorem or come up with a novel result in theoretical physics? While beyond current motels, that would be the final test of AGI capability.

That would be important, but as far as I know it hasn’t happened (despite how often it’s intimated that we’re on the verge of it happening).

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#939

Post by tintor »

miki123211 wrote: post #542
bluecoconut wrote: post #201

Earlier quoted context omitted.

some other imporant quotes: "Average human off the street: 70-80%. STEM college grad: >95%. Panel of 10 random humans: 99-100%" -@fchollet on X So, considering that the $3400/task system isn't able to compete with STEM college grad yet, we still have some room (but it is shrinking, i expect even more compute will be thrown and we'll see these barriers broken in coming years) Also, some other back of envelope calculat…

It's also worth keeping in mind that AIs are a lot less risky to deploy for businesses than humans. You can scale them up and down at any time, they can work 24/7 (including holidays) with no overtime pay and no breaks, they need no corporate campuses, office space, HR personnel or travel budgets, you don't have to worry about key employees going on sick/maternity leave or taking time off the moment they're needed mo…

“they won’t leak”

That one isn’t guaranteed. Many examples online of exfiltration attacks on LLMs.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#940

Post by salawat »

RivieraKid wrote: post #229
rvz wrote: post #73

Great results. However, let's all just admit it. It has well replaced journalists, artists and on its way to replace nearly both junior and senior engineers. The ultimate intention of "AGI" is that it is going to replace tens of millions of jobs. That is it and you know it. It will only accelerate and we need to stop pretending and coping. Instead lets discuss solutions for those lost jobs. So what is the replacement…

The economic theory answer is that people simply switch to jobs that are not yet replaceable by AI. Doctors, nurses, electricians, construction workers, police officers, etc. People in aggregate will produce more, consume more and work less.

>Doctors capped per year by law.

>Trades people only work after having something to do. If you don't have sufficient demand for builders, electricians, plumbers, etc... No one can afford to become one. Nevermind the fact that not everyone should be any of those things. Economics fails when the loop fails to close.

Post reply on HN