Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

501–510 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#501

Post by byyoung3 »

brysonreece wrote: post #461
OhioMan2943 wrote: post #456

I'm 22 and have no clue what I'm meant to do in a world where this is a thing. I'm moving to a semi rural, outdoorsy area where they teach data science and marine science and I can enjoy my days hiking, and the march of technology is a little slower. I know this will disrupt so much of our way of life, so I'm chasing what fun innocent years are left before things change dramatically.

It's worth noting that LLMs have been part of the tech zeitgeist for over two years and have had a pretty limited impact on hireability for roles, despite what people like the Klarna CEO are saying. Personally, I'm betting on two things: * The upward bound of compute/performance gains as we continue to iterate on LLMs. It simply isn't going to be feasible for a lot of engineers and businesses to run/train their own L…

o8 will probably be able to handle datacenter management

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#502

Post by fzzzy »

csomar wrote: post #487
m3kw9 wrote: post #111

Earlier quoted context omitted.

I would say they didn’t need to demo anything, because if you are gonna use the output code live on a demo it may make compile errors and then look stupid trying to fix it live

Models are predictable at 0 temperatures. They might have tested the output beforehand.

Models in practice haven't been deterministic at 0 temperature, although nobody knows exactly why. Either hardware or software bugs.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#503

Post by HeatrayEnjoyer »

croemer wrote: post #93

The programming task they gave o3-mini high (creating Python server that allows chatting with OpenAI API and run some code in terminal) didn't seem very hard? Strange choice of example for something that's claimed to be a big step forwards. YT timestamped link: https://www.youtube.com/watch?v=SKBG1sqdyIU&t=768s (thanks for the fixed link @photonboom) Updated: I gave the task to Claude 3.5 Sonnet and it worked first s…

Sonnet isn't a "mini" sized model. Try it with Haiku.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#505

Post by echelon »

spencerchubb wrote: post #453
bluecoconut wrote: post #141

Efficiency is now key. ~=$3400 per single task to meet human performance on this benchmark is a lot. Also it shows the bullets as "ARC-AGI-TUNED", which makes me think they did some undisclosed amount of fine-tuning (eg. via the API they showed off last week), so even more compute went into this task. We can compare this roughly to a human doing ARC-AGI puzzles, where a human will take (high variance in my subjective…

> Super exciting that OpenAI pushed the compute out this far it's even more exciting than that. the fact that you even can use more compute to get more intelligence is a breakthrough. if they spent even more on inference, would they get even better scores on arc agi?

Maybe it's not linear spend.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#506

Post by salter2 »

OhioMan2943 wrote: post #456

I'm 22 and have no clue what I'm meant to do in a world where this is a thing. I'm moving to a semi rural, outdoorsy area where they teach data science and marine science and I can enjoy my days hiking, and the march of technology is a little slower. I know this will disrupt so much of our way of life, so I'm chasing what fun innocent years are left before things change dramatically.

I'm the same age as you; I feel lost, erring in being a little too pessimistic.

Feels like I hit the real world just a couple years too late to get situated in a solid position. Years of obsession in attempt to catch up to the wizards, chasing the tech dream. But this, feels like this is it. Just watching the timebomb tick. I'd love to work on what feels like the final technology, but I'm not a freakshow like what these labs are hiring. At least I get to spectate the creation of humanity's greatest invention.

This announcement is just another gut punch, but at this point I should expect its inevitable. A Jason Voorhees AGI, slowly but surely to devour all the talents and skills information workers have to offer.

Apologies for the rambly and depressing post, but this is reality for anyone recently out or still in school.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#507

Post by __MatrixMan__ »

With only a 100x increase in cost, we improved performance by 0.1x and continued plotting this concave-down diminishing-returns type graph! Hurray for logarithmic x-axes!

Joking aside, better than ever before at any cost is an achievement, it just doesn't exactly scream "breakthrough" to me.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#508

Post by computerex »

danielmarkbruce wrote: post #344
modeless wrote: post #13

Congratulations to Francois Chollet on making the most interesting and challenging LLM benchmark so far. A lot of people have criticized ARC as not being relevant or indicative of true reasoning, but I think it was exactly the right thing. The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning. It's obvious to everyone that…

Highly challenging for LLMs because it has nothing to do with language. LLMs and their training processes have all kinds of optimizations for language and how it's presented. This benchmark has done a wonderful job with marketing by picking a great name. It's largely irrelevant for LLMs despite the fact it's difficult. Consider how much of the model is just noise for a task like this given the low amount of informati…

The benchmark is designed to test for AGI and intelligence, specifically the ability to solve novel problems.

If the hypothesis is that LLMs are the “computer” that drives the AGI then of course the benchmark is relevant in testing for AGI.

I don’t think you understand the benchmark and its motivation. ARC AGI benchmark problems are extremely easy and simple for humans. But LLMs fail spectacularly at them. Why they fail is irrelevant, the fact they fail though means that we don’t have AGI.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#509

Post by Havoc »

OhioMan2943 wrote: post #456

I'm 22 and have no clue what I'm meant to do in a world where this is a thing. I'm moving to a semi rural, outdoorsy area where they teach data science and marine science and I can enjoy my days hiking, and the march of technology is a little slower. I know this will disrupt so much of our way of life, so I'm chasing what fun innocent years are left before things change dramatically.

>I'm 22 and have no clue what I'm meant to do in a world where this is a thing.

For what it's worth that's probably an advantage versus the legions of people who are staring down the barrel of years invested into skills that may lose relevance very rapidly.

Post reply on HN