I’m super curious as to whether this technology completely destroys the middle class, or if everyone becomes better off because productivity is going to skyrocket.
Is anyone here aware of the latest research that tries to predict the outcome? Please share - super curious as well
OpenAI O3 breakthrough high score on ARC-AGI-PUB
241–250 of 1001 posts
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#242Earlier quoted context omitted.
>Achieving human-level performance in the ARC benchmark, as well as top human performance in GPQA, Codeforce, AIME, and Frontier Math strongly suggests the model can potentially solve any problem at the human level if it possesses essential knowledge about it. The article notes, "o3 still fails on some very easy tasks". What explains these failures if o3 can solve "any problem" at the human level? Do these failed cas…
Great point. I'd love to see what these easy tasks are and would be happy to revise my hypothesis accordingly. o3's intelligence is unlikely to be a strict superset of human intelligence. It is certainly superior to humans in some respects and probably inferior in others. Whether it's sufficiently generally intelligent would be both a matter of definition and empirical fact.
https://x.com/fchollet/status/1870172872641261979
https://x.com/fchollet/status/1870173137234727219
I would definitely consider them legitimately easy for humans.
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#243Complete aside here: I used to do work with amputees and prosthetics. There is a standardized test (and I just cannot remember the name) that fits in a briefcase. It's used for measuring the level of damage to the upper limbs and for prosthetic grading. Basically, it's got the dumbest and simplest things in it. Stuff like a lock and key, a glass of water and jug, common units of currency, a zipper, etc. It tests if y…
In that sense, the goalposts haven’t moved in a long time despite claims from AI enthusiasts that people are constantly moving goalposts.
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#244The programming task they gave o3-mini high (creating Python server that allows chatting with OpenAI API and run some code in terminal) didn't seem very hard? Strange choice of example for something that's claimed to be a big step forwards. YT timestamped link: https://www.youtube.com/watch?v=SKBG1sqdyIU&t=768s (thanks for the fixed link @photonboom) Updated: I gave the task to Claude 3.5 Sonnet and it worked first s…
I would say they didn’t need to demo anything, because if you are gonna use the output code live on a demo it may make compile errors and then look stupid trying to fix it live
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#245Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#246OpenAI spent approximately $1,503,077 to smash the SOTA on ARC-AGI with their new o3 model semi-private evals (100 tasks): 75.7% @ $2,012 total/100 tasks (~$20/task) with just 6 samples & 33M tokens processed in ~1.3 min/task and a cost of $2012 The “low-efficiency” setting with 1024 samples scored 87.5% but required 172x more compute. If we assume compute spent and cost are proportional, then OpenAI might have just…
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#247Great. Now we have to think of a new way to move the goalposts.
This is just as silly as claiming that people "moved the goalposts" when a computer beat Kasparov at chess to claim that it wasn't AGI: it wasn't a good test and some people only realize this after the computer beat Kasparov but couldn't do much else. In this case the ARC maintainers specifically have stated that this is a necessary but not sufficient test of AGI (I personally think it is neither).
o3 can do much much more. There is nothing narrow about SOTA LLMs. They are already General. It doesn't matter what ARC Maintainers have said. There is no common definition of General that LLMs fail to meet. It's not a binary thing.
By the time a single machine covers every little test humanity can devise, what comes out of that is not 'AGI' as the words themselves mean but a General Super Intelligence.
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#248A bit puzzling to me. Why does it matter ?
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#249Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#250Another meaningless benchmark, another month—it’s like clockwork at this point. No one’s going to remember this in a month; it’s just noise. The real test? It’s not in these flashy metrics or minor improvements. The only thing that actually matters is how fast it can wipe out the layers of middle management and all those pointless, bureaucratic jobs that add zero value. That’s the true litmus test. Everything else? I…