Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

351–360 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#351

Post by lacedeconstruct »

thisisthenewme wrote: post #313

I feel like AI is already changing how we work and live - I've been using it myself for a lot of my development work. Though, what I'm really concerned about is what happens when it gets smart enough to do pretty much everything better (or even close) than humans can. We're talking about a huge shift where first knowledge workers get automated, then physical work too. The thing is, our whole society is built around p…

I am pretty sure we will have a deep cultural repulsion from it and people will pay serious money to have an AI free experience, If AI becomes actually useful there is alot of areas that we dont even know how to tackle like medicine and biology, I dont think anything would change otherwise, AI will take jobs but it will open alot more jobs at much higher abstraction, 50 years ago the idea that a software engineer would become a get rich quick job would have been insane imo

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#352

Post by wyager »

demirbey05 wrote: post #336

It is not exactly AGI but huge step toward it. I would expect this step in 2028-2030. I cant really understand why people are happy with it, this technology is so dangerous that can disrupt whole society. It's neither like smartphone nor internet. What will happen to 3rd world countries. Lots of unsolved questions and world is not prepared for such a change. Lots of people will lose their jobs I am not even mentionin…

> What will happen to 3rd world countries

Probably less disruption than will happen in 1st world countries.

> No one will have chance to be rich anymore

It's strange to reach this conclusion from "look, a massive new productivity increase".

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#353

Post by lagrange77 »

sss111 wrote: post #69
maxdoop wrote: post #8

How much longer can I get paid $150k to write code ?

3 to 5 years, max. Traditional coding is going to be dead in the water. Optimistically, the junior SWE job will evolve but more realistically dedicated AI-based programming agents will end demand for Junior SWEs

Which implies that a few years later they will not become senior SWEs either.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#354

Post by jasondigitized »

hansonkd wrote: post #34
zebomon wrote: post #19

My initial impression: it's very impressive and very exciting. My skeptical impression: it's complete hubris to conflate ARC or any benchmark with truly general intelligence. I know my skepticism here is identical to moving goalposts. More and more I am shifting my personal understanding of general intelligence as a phenomenon we will only ever be able to identify with the benefit of substantial retrospect. As it is…

It doesn't need to be general intelligence or perfectly map to human intelligence. All it needs to be is useful. Reading constant comments about LLMs can't be general intelligence or lack reasoning etc, to me seems like people witnessing the airplane and complaining that it isn't "real flying" because it isn't a bird flapping its wings (a large portion of the population held that point of view back then). It doesn't…

This a thousand times.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#355

Post by sunaookami »

m3kw9 wrote: post #111
croemer wrote: post #93

The programming task they gave o3-mini high (creating Python server that allows chatting with OpenAI API and run some code in terminal) didn't seem very hard? Strange choice of example for something that's claimed to be a big step forwards. YT timestamped link: https://www.youtube.com/watch?v=SKBG1sqdyIU&t=768s (thanks for the fixed link @photonboom) Updated: I gave the task to Claude 3.5 Sonnet and it worked first s…

I would say they didn’t need to demo anything, because if you are gonna use the output code live on a demo it may make compile errors and then look stupid trying to fix it live

They actually did that the last time when they showed the apps integration. First try in Xcode didn't work.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#356

Post by adamgordonbell »

modeless wrote: post #13

Congratulations to Francois Chollet on making the most interesting and challenging LLM benchmark so far. A lot of people have criticized ARC as not being relevant or indicative of true reasoning, but I think it was exactly the right thing. The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning. It's obvious to everyone that…

There is a benchmark, NovelQA, that LLMs don't dominate when it feels like they should. The benchmark is to read a novel and answer questions about it.

LLMs are below human evaluation, as I last looked, but it doesn't get much attention.

Once it is passed, I'd like to see one that is solving the mystery in a mystery book right before it's revealed.

We'd need unpublished mystery novels to use for that benchmark, but I think it gets at what I think of as reasoning.

https://novelqa.github.io/

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#357

Post by Ancalagon »

demirbey05 wrote: post #336

It is not exactly AGI but huge step toward it. I would expect this step in 2028-2030. I cant really understand why people are happy with it, this technology is so dangerous that can disrupt whole society. It's neither like smartphone nor internet. What will happen to 3rd world countries. Lots of unsolved questions and world is not prepared for such a change. Lots of people will lose their jobs I am not even mentionin…

Same, I don’t really get the excitement. None of these companies are pushing for a utopian Star Trek society either with that power.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#358

Post by bcrosby95 »

zamadatix wrote: post #297
bluecoconut wrote: post #201

Earlier quoted context omitted.

some other imporant quotes: "Average human off the street: 70-80%. STEM college grad: >95%. Panel of 10 random humans: 99-100%" -@fchollet on X So, considering that the $3400/task system isn't able to compete with STEM college grad yet, we still have some room (but it is shrinking, i expect even more compute will be thrown and we'll see these barriers broken in coming years) Also, some other back of envelope calculat…

I don't follow how 10 random humans can beat the average STEM college grad and average humans in that tweet. I suspect it's really "a panel of 10 randomly chosen experts in the space" or something? I agree the most interesting thing to watch will be cost for a given score more than maximum possible score achieved (not that the latter won't be interesting by any means).

Two heads is better than 1. 10 is way better. Even if they aren't a field of experts. You're bound to get random people that remember random stuff from high school, college, work, and life in general, allowing them to piece together a solution.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#359

Post by tivert »

mensetmanusman wrote: post #74

I’m super curious as to whether this technology completely destroys the middle class, or if everyone becomes better off because productivity is going to skyrocket.

> I’m super curious as to whether this technology completely destroys the middle class, or if everyone becomes better off because productivity is going to skyrocket.

Even if productivity skyrockets, why would anyone assume the dividends would be shared with the "destroy[ed] middle class"?

All indications will be this will end up like the China Shock: "I lost my middle class job, and all I got was the opportunity to buy flimsy pieces of crap from a dollar store." America lacks the ideological foundations for any other result, and the coming economic changes will likely make building those foundations even more difficult if not impossible.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#360

Post by phillipcarter »

qt31415926 wrote: post #129
phillipcarter wrote: post #53

Earlier quoted context omitted.

As excited as I am by this, I still feel like this is still just a small approximation of a small chunk of human reasoning ability at large. o3 (and whatever comes next) feels to me like it will head down the path of being a reasoning coprocessor for various tasks. But, still, this is incredibly impressive.

Which parts of reasoning do you think is missing? I do feel like it covers a lot of 'reasoning' ground despite its on the surface simplicity

I think it's hard to enumerate the unknown, but I'd personally love to see how models like this perform on things like word problems where you introduce red herrings. Right now, LLMs at large tend to struggle mightily to understand when some of the given information is not only irrelevant, but may explicitly serve to distract from the real problem.
Post reply on HN