Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

881–890 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#881

Post by zitterbewegung »

miki123211 wrote: post #542
bluecoconut wrote: post #201

Earlier quoted context omitted.

some other imporant quotes: "Average human off the street: 70-80%. STEM college grad: >95%. Panel of 10 random humans: 99-100%" -@fchollet on X So, considering that the $3400/task system isn't able to compete with STEM college grad yet, we still have some room (but it is shrinking, i expect even more compute will be thrown and we'll see these barriers broken in coming years) Also, some other back of envelope calculat…

It's also worth keeping in mind that AIs are a lot less risky to deploy for businesses than humans. You can scale them up and down at any time, they can work 24/7 (including holidays) with no overtime pay and no breaks, they need no corporate campuses, office space, HR personnel or travel budgets, you don't have to worry about key employees going on sick/maternity leave or taking time off the moment they're needed mo…

Having AI "tarnish the reputation of your company" encompasses so much in regard to AI when it can receive input and be manipulated by others such as Tai from Microsoft and many other outcomes where there is a true risk for AI deployment.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#882

Post by 0points »

acchow wrote: post #729
bluecoconut wrote: post #201

Earlier quoted context omitted.

some other imporant quotes: "Average human off the street: 70-80%. STEM college grad: >95%. Panel of 10 random humans: 99-100%" -@fchollet on X So, considering that the $3400/task system isn't able to compete with STEM college grad yet, we still have some room (but it is shrinking, i expect even more compute will be thrown and we'll see these barriers broken in coming years) Also, some other back of envelope calculat…

> ~doubling every 2-2.5 years) puts us at 20~25 years. The trend for power consumption of compute (Megaflops per watt) has generally tracked with Koomey’s law for a doubling every 1.57 years Then you also have model performance improving with compression. For example, Llama 3.1’s 8B outperforming the original Llama 65B

Then you will just have the issue of supplying enough of power to support this "linear" growth of yours.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#883

Post by gliptic »

msoad wrote: post #873

There are new research where chain of thoughts is happening in latent spaces and not in English. They demonstrated better results since language is not as expressive as those concepts that can be represented in the layers before decoder. I wonder if o3 is doing that?

"You can tell the RL is done properly when the models cease to speak English in their chain of thought" -- Karpathy

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#885

Post by macrolime »

tymonPartyLate wrote: post #841

Isn’t this like a brute force approach? Given it costs $ 3000 per task, thats like 600 GPU hours (h100 at Azure) In that amount of time the model can generate millions of chains of thoughts and then spend hours reviewing them or even testing them out one by one. Kind of like trying until something sticks and that happens to solve 80% of ARC. I feel like reasoning works differently in my brain. ;)

The trick with AlphaGo was brute force combined with learning to extract strategies from brute force using reinforcement learning, that's what we'll see here. So maybe it costs a million dollars in compute to get a high score, but use reinforcement learning ala alphazero to learn from the process and it won't cost a million dollars next time and let it do lots of hard benchmarks, math problems and coding tasks and it'll keep getting better and better.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#886

Post by 0points »

cchance wrote: post #304
scotty79 wrote: post #67

Earlier quoted context omitted.

Still it's comparing average human level performance with best AI performance. Examples of things o3 failed at are insanely easy for humans.

You'd be surprised what the AVERAGE human fails to do that you think is easy, my mom can't fucking send an email without downloading a virus, i have a coworker that believes beyond a shadow of a doubt the world is flat. The Average human is a lot dumber than people on hackernews and reddit seem to realize, shit the people on mturk are likely smarter than the AVERAGE person

Your examples are just examples of lack of information. That's not a measure for intelligence.

As a contrary point, most people think they are smarter than they really are.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#887

Post by esafak »

stevenhuang wrote: post #652
Engineering-MD wrote: post #575

Can I just say what a dick move it was to do this as a 12 days of Christmas. I mean to be honest I agree with the arguments this isn’t as impressive as my initial impression, but they clearly intended it to be shocking/a show of possible AGI, which is rightly scary. It feels so insensitive to that right before a major holiday when the likely outcome is a lot of people feeling less secure in their career/job/life. Tha…

This is a you problem. Yes there will be pain in short term, but it will be worth it in long term. Many of us look forward to what a future with AGI can do to help humanity and hopefully change society for the better, mainly to achieve a post scarcity economy.

How are you going to make housing, healthcare, etc. not scarce, and pay for them?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#888

Post by cmiles74 »

miki123211 wrote: post #542
bluecoconut wrote: post #201

Earlier quoted context omitted.

some other imporant quotes: "Average human off the street: 70-80%. STEM college grad: >95%. Panel of 10 random humans: 99-100%" -@fchollet on X So, considering that the $3400/task system isn't able to compete with STEM college grad yet, we still have some room (but it is shrinking, i expect even more compute will be thrown and we'll see these barriers broken in coming years) Also, some other back of envelope calculat…

It's also worth keeping in mind that AIs are a lot less risky to deploy for businesses than humans. You can scale them up and down at any time, they can work 24/7 (including holidays) with no overtime pay and no breaks, they need no corporate campuses, office space, HR personnel or travel budgets, you don't have to worry about key employees going on sick/maternity leave or taking time off the moment they're needed mo…

"...they need no corporate campuses, office space..."

This is a big downside of AI, IMHO. Those offices need to be filled! ;-)

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#889

Post by rapjr9 »

Does anyone have a feeling for how latency (from asking a question/API call to getting an answer/API return) is progressing with new models? I see 1.3 minutes/task and 13.8 minutes/task mentioned in the page on evaluating O3. Efficiency gains that also reduce latency will be important and some of them will come from efficiency in computation, but as models include more and more layers (layers of models for example) the overall latency may grow and faster compute times inside each layer may only help somewhat. This could have large effects on usability.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#890

Post by thegeomaster »

torginus wrote: post #763
hamburga wrote: post #627

I’m not sure if people realize what a weird test this is. They’re these simple visual puzzles that people can usually solve at a glance, but for the LLMs, they’re converted into a json format, and then the LLMs have to reconstruct the 2D visual scene from the json and pick up the patterns. If humans were given the json as input rather than the images, they’d have a hard time, too.

Not sure how much that matters - I'm not an AI expert, but I did some intro courses where we had to train a classifier to recognize digits. How it worked basically was that we fed each pixel of the 2d grid of the image into an input of the network, essentially flattening it in a similar fashion. It worked just fine, and that was a tiny network.

The classifier was likely a convolutional network, so the assumption of the image being a 2D grid was baked into the architecture itself - it didn't have to be represented via the shape of the input for the network to use it.
Post reply on HN