OpenAI O3 breakthrough high score on ARC-AGI-PUB

Post by melagonster » Sat, Dec 21, 2024, 3:13 PM UTC

throw83288 wrote: post #550

hcwilk wrote: post #537

I just graduated college, and this was a major blow. I studied Mechanical Engineering and went into Sales Engineering because cause I love technology and people, but articles like this do nothing but make me dread the future. I have no idea what to specialize in, what skills I should master, or where I should be spending my time to build a successful career. Seems like we’re headed toward a world where you automate s…

This is me as well. Either: 1) Just give up computing entirely, the field I've been dreaming about since childhood. Perhaps if I immiserate myself with a dry regulated engineering field or trade I would perhaps survive to recursive self-improvement, but if anything the length it takes to pivot (I am a Junior in College that has already done probably 3/4th of my CS credits) means I probably couldn't get any foothold u…

Post by atleastoptimal » Sat, Dec 21, 2024, 3:21 PM UTC

istjohn wrote: post #373

w4 wrote: post #319

The cost to run the highest performance o3 model is estimated to be somewhere between $2,000 and $3,400 per task.[1] Based on these estimates, o3 costs about 100x what it would cost to have a human perform the exact same task. Many people are therefore dismissing the near-term impact of these models because of these extremely expensive costs. I think this is a mistake. Even if very high costs make o3 uneconomic for b…

Your economic analysis is deeply flawed. If there was anything that valuable and that required that much manpower, it would already have driven up the cost of labor accordingly. The one property that could conceivably justify a substantially higher cost is secrecy. After all, you can't (legally) kill a human after your project ends to ensure total secrecy. But that takes us into thriller novel territory.

Post by HarHarVeryFunny » Sat, Dec 21, 2024, 3:27 PM UTC

cchance wrote: post #304

scotty79 wrote: post #67

Earlier quoted context omitted.

Still it's comparing average human level performance with best AI performance. Examples of things o3 failed at are insanely easy for humans.

You'd be surprised what the AVERAGE human fails to do that you think is easy, my mom can't fucking send an email without downloading a virus, i have a coworker that believes beyond a shadow of a doubt the world is flat. The Average human is a lot dumber than people on hackernews and reddit seem to realize, shit the people on mturk are likely smarter than the AVERAGE person

Post by atleastoptimal » Sat, Dec 21, 2024, 3:28 PM UTC

solidasparagus wrote: post #619

Imnimo wrote: post #52

Whenever a benchmark that was thought to be extremely difficult is (nearly) solved, it's a mix of two causes. One is that progress on AI capabilities was faster than we expected, and the other is that there was an approach that made the task easier than we expected. I feel like the there's a lot of the former here, but the compute cost per task (thousands of dollars to solve one little color grid puzzle??) suggests t…

Or the test wasn't testing anything meaningful, which IMO is what happened here. I think ARC was basically looking at the distribution of what AI is capable of, picked an area that it was bad at and no one had cared enough to go solve, and put together a benchmark. And then we got good at it because someone cared and we had a measurement. Which is essentially the goal of ARC. But I don't much agree that it is any mea…

Post by tim333 » Sat, Dec 21, 2024, 3:33 PM UTC

Engineering-MD wrote: post #575

Can I just say what a dick move it was to do this as a 12 days of Christmas. I mean to be honest I agree with the arguments this isn’t as impressive as my initial impression, but they clearly intended it to be shocking/a show of possible AGI, which is rightly scary. It feels so insensitive to that right before a major holiday when the likely outcome is a lot of people feeling less secure in their career/job/life. Tha…