Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

671–680 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#671

Post by john_minsk »

qt31415926 wrote: post #129
phillipcarter wrote: post #53

Earlier quoted context omitted.

As excited as I am by this, I still feel like this is still just a small approximation of a small chunk of human reasoning ability at large. o3 (and whatever comes next) feels to me like it will head down the path of being a reasoning coprocessor for various tasks. But, still, this is incredibly impressive.

Which parts of reasoning do you think is missing? I do feel like it covers a lot of 'reasoning' ground despite its on the surface simplicity

My personal 5 cents is that reasoning will be there when LLM gives you some kind of outcome and then when questioned about it can explain every bit of result it produced.

For example, if we asked an LLM to produce an image of a "human woman photorealistic" it produces result. After that you should be able to ask it "tell me about its background" and it should be able to explain "Since user didn't specify background in the query I randomly decided to draw her standing in front of a fantasy background of Amsterdam iconic houses. Usually Amsterdam houses are 3 stories tall, attached to each other and 10 meters wide. Amsterdam houses usually have cranes on the top floor, which help to bring goods to the top floor since doors are too narrow for any object wider than 1m. The woman stands in front of the houses approximately 25 meters in front of them. She is 1,59m tall, which gives us correct perspective. It is 11:16am of August 22nd which I used to calculate correct position of the sun and align all shadows according to projected lighting conditions. The color of her skin is set at RGB:xxxxxx randomly" etc.

And it is not too much to ask LLMs for it. LLMs have access to all the information above as they read all the internet. So there is definitely a description of Amsterdam architecture, what a human body looks like or how to correctly estimate time of day based on shadows (and vise versa). The only thing missing is logic that connects all this information and which is applied correctly to generate final image.

I like to think about LLMs as a fancy genius compressing engines. They took all the information in the internet, compressed it and are able to cleverly query this information for end user. It is a tremendously valuable thing, but if intelligence emerges out of it - not sure. Digital information doesn't necessarily contain everything needed to understand how it was generated and why.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#672

Post by esafak »

tripletao wrote: post #628
hcwilk wrote: post #537

I just graduated college, and this was a major blow. I studied Mechanical Engineering and went into Sales Engineering because cause I love technology and people, but articles like this do nothing but make me dread the future. I have no idea what to specialize in, what skills I should master, or where I should be spending my time to build a successful career. Seems like we’re headed toward a world where you automate s…

I feel like many people are reacting to the string "AGI" in the benchmark name, and not to the actual result. The tasks in question are to color squares in a grid, maintaining the geometric pattern of the examples. Unlike most other benchmarks where LLMs have shown large advances (in law, medicine, etc.), this benchmark isn't directly related to any practically useful task. Rather, the benchmark is notable because it…

I think you did not watch the full video. The model performs at PhD level on maths questions, and expert level at coding.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#673

Post by esafak »

prpl wrote: post #629
hcwilk wrote: post #537

I just graduated college, and this was a major blow. I studied Mechanical Engineering and went into Sales Engineering because cause I love technology and people, but articles like this do nothing but make me dread the future. I have no idea what to specialize in, what skills I should master, or where I should be spending my time to build a successful career. Seems like we’re headed toward a world where you automate s…

In 2016 I was asked by an Uber driver in Pittsburgh when his job would be obsolete (I’d worked around Zoox people quite a bit and Uber basically was all-in at CMU. I told him it was at least 5 years, probably 10, though he was sure it would be 2. I was arguably “right”, 2023-ish is probably going to be the date people put down in the books, but the future isn’t evenly distributed. It’s at least another 5 years, and m…

Just don't have kids?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#674

Post by myko »

throw83288 wrote: post #550
hcwilk wrote: post #537

I just graduated college, and this was a major blow. I studied Mechanical Engineering and went into Sales Engineering because cause I love technology and people, but articles like this do nothing but make me dread the future. I have no idea what to specialize in, what skills I should master, or where I should be spending my time to build a successful career. Seems like we’re headed toward a world where you automate s…

This is me as well. Either: 1) Just give up computing entirely, the field I've been dreaming about since childhood. Perhaps if I immiserate myself with a dry regulated engineering field or trade I would perhaps survive to recursive self-improvement, but if anything the length it takes to pivot (I am a Junior in College that has already done probably 3/4th of my CS credits) means I probably couldn't get any foothold u…

spend a little time learning how to use LLMs and i think you'll be less scared. they're not that good at doing the job of a software developer.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#675

Post by ImaCake »

hamburga wrote: post #627

I’m not sure if people realize what a weird test this is. They’re these simple visual puzzles that people can usually solve at a glance, but for the LLMs, they’re converted into a json format, and then the LLMs have to reconstruct the 2D visual scene from the json and pick up the patterns. If humans were given the json as input rather than the images, they’d have a hard time, too.

Yeah, this entire thread seems utterly detached from my lived experience. LLMs are immensely useful for me at work but they certainly don't come close to the hype spouted by many commenters here. It would be great if it could handle more of our quite modest codebase but it's not able to yet

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#677

Post by sakopov »

jebarker wrote: post #646
sakopov wrote: post #642

Maybe I'm missing something vital, but how does anything that we've seen AI do up until this point or explained in this experiment even hint at AGI? Can any of these models ideate? Can they come up with technologies and tools? No and it's unlikely they will any time soon. However, they can make engineers infinitely more productive.

You need to define ideate, tools and technologies to answer those questions. Not to mention that it's quite possible humans do those things through re-combination of learned ideas similarly to how these reasoning models are suggested to be working.

Every technological advancement that we've seen in software engineering - be it in things like Postgres, Kubernetes and Cloud Infrastructure - came out from truly novel ideas. AI seems to generate outputs that appear novel but are they really? It's capable of synthesizing and combining vast amounts of information in creative ways but it's deriving everything from existing patterns found within its training data. Truly novel ideas require thinking outside the box. It's combination of cognitive, emotional and environmental factors which go beyond pattern recognition. How close are we to achieving this? Everyone seems to be shaking in their boots because we might lose our job safety in tech, but I don't see any intelligence here.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#678

Post by qingcharles »

iandanforth wrote: post #372
bluecoconut wrote: post #201

Earlier quoted context omitted.

some other imporant quotes: "Average human off the street: 70-80%. STEM college grad: >95%. Panel of 10 random humans: 99-100%" -@fchollet on X So, considering that the $3400/task system isn't able to compete with STEM college grad yet, we still have some room (but it is shrinking, i expect even more compute will be thrown and we'll see these barriers broken in coming years) Also, some other back of envelope calculat…

Let's say that Google is already 1 generation ahead of nvidia in terms of efficient AI compute. ($1700) Then let's say that OpenAI brute forced this without any meta-optimization of the hypothesized search component (they just set a compute budget). This is probably low hanging fruit and another 2x in compute reduction. ($850) Then let's say that OpenAI was pushing really really hard for the numbers and was willing t…

Until 2022 most AI research was aimed at improving the quality of the output, not the quantity.

Since then there has been a tsunami of optimizations in the way training and inference is done. I don't think we've even begun to find all the ways that inference can be further optimized at both hardware and software levels.

Look at the huge models that you can happily run on an M3 Mac. The cost reduction in inference is going to vastly outpace Moore's law, even as chip design continues on its own path.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#679

Post by jakebasile »

stevenhuang wrote: post #652
Engineering-MD wrote: post #575

Can I just say what a dick move it was to do this as a 12 days of Christmas. I mean to be honest I agree with the arguments this isn’t as impressive as my initial impression, but they clearly intended it to be shocking/a show of possible AGI, which is rightly scary. It feels so insensitive to that right before a major holiday when the likely outcome is a lot of people feeling less secure in their career/job/life. Tha…

This is a you problem. Yes there will be pain in short term, but it will be worth it in long term. Many of us look forward to what a future with AGI can do to help humanity and hopefully change society for the better, mainly to achieve a post scarcity economy.

Surely the elites that control this fancy new technology will share the benefits with all of us _this_ time!
Post reply on HN