Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

251–260 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#251

Earlier quoted context omitted.

Quote from the creators of the AGI-ARC benchmark: "Passing ARC-AGI does not equate achieving AGI, and, as a matter of fact, I don't think o3 is AGI yet. o3 still fails on some very easy tasks, indicating fundamental differences with human intelligence."

Yeah the real goalpost is reliable intelligence. A supposed phd level AI failing simple problems is a red flag that we’re still missing something.

You've never met a Doctor who couldn't figure out how to work their email? Or use street smarts? You can have a PHD but be unable to reliably handle soft skills, or any number of things you might 'expect' someone to be able to do.

Just playing devils' advocate or nitpicking the language a bit...

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#252

It sucks that I would love to be excited about this... but I mostly feel anxiety and sadness.

We’re enabling a huge swath of humanity being put out of work so a handful of billionaires can become trillionaires.

And also the solving of hundreds of diseases that ail us.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#253

Let me go against some skeptics and explain why I think full o3 is pretty much AGI or at least embodies most essential aspects of AGI. What has been lacking so far in frontier LLMs is the ability to reliably deal with the right level of abstraction for a given problem. Reasoning is useful but often comes out lacking if one cannot reason at the right level of abstraction. (Note that many humans can't either when they…

GPQA scores are mostly from pre-training, against content in the corpus. They have gone silent but look at the GPT4 technical report which calls this out. We are nowhere close to what Sam Altman calls AGI and transformers are still limited to what uniform-TC0 can do. As an example the Boolean Formula Value Problem is NC1-complete, thus beyond transformers but trivial to solve with a TM. As it is now proven that the f…

Isn't any physically realizable computer (including our brains) limited to what uniform-TC0 can do?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#254

It sucks that I would love to be excited about this... but I mostly feel anxiety and sadness.

Anxiety and sadness are actually mild emotional responses to the dissolution of human culture. Nick Land in 1992:

"It is ceasing to be a matter of how we think about technics, if only because technics is increasingly thinking about itself. It might still be a few decades before artificial intelligences surpass the horizon of biological ones, but it is utterly superstitious to imagine that the human dominion of terrestrial culture is still marked out in centuries, let alone in some metaphysical perpetuity. The high road to thinking no longer passes through a deepening of human cognition, but rather through a becoming inhuman of cognition, a migration of cognition out into the emerging planetary technosentience reservoir, into 'dehumanized landscapes ... emptied spaces' where human culture will be dissolved. Just as the capitalist urbanization of labour abstracted it in a parallel escalation with technical machines, so will intelligence be transplanted into the purring data zones of new software worlds in order to be abstracted from an increasingly obsolescent anthropoid particularity, and thus to venture beyond modernity. Human brains are to thinking what mediaeval villages were to engineering: antechambers to experimentation, cramped and parochial places to be.

[...]

Life is being phased-out into something new, and if we think this can be stopped we are even more stupid than we seem." [0]

Land is being ostracized for some of his provocations, but it seems pretty clear by now that we are in the Landian Accelerationism timeline. Engaging with his thought is crucial to understanding what is happening with AI, and what is still largely unseen, such as the autonomization of capital.

[0] https://retrochronic.com/#circuitries

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#255

Complete aside here: I used to do work with amputees and prosthetics. There is a standardized test (and I just cannot remember the name) that fits in a briefcase. It's used for measuring the level of damage to the upper limbs and for prosthetic grading. Basically, it's got the dumbest and simplest things in it. Stuff like a lock and key, a glass of water and jug, common units of currency, a zipper, etc. It tests if y…

> We had hand prosthetics that could play Mozart at 5x speed on a baby grand, but could not pick up a silver dollar or zip a jacket even a little bit. "

I must be missing something, how can they be able to play Mozart at 5x speed with their prosthetics but not zip a jacket? They could press keys but not do tasks requiring feedback?

Or did you mean they used to play Mozart at 5x speed before they became amputees?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#256

OpenAI spent approximately $1,503,077 to smash the SOTA on ARC-AGI with their new o3 model semi-private evals (100 tasks): 75.7% @ $2,012 total/100 tasks (~$20/task) with just 6 samples & 33M tokens processed in ~1.3 min/task and a cost of $2012 The “low-efficiency” setting with 1024 samples scored 87.5% but required 172x more compute. If we assume compute spent and cost are proportional, then OpenAI might have just…

By my estimates, for this single benchmark, this is comparable cost to training a ~70B model from scratch today. Literally from 0 to a GPT-3 scale model for the compute they ran on 100 ARC tasks.

I double checked with some flop estimates (P100 for 12 hours = Kaggle limit, they claim ~100-1000x for O3-low, and x172 for O3-high) so roughly on the order of 10^22-10^23 flops.

In another way, using H100 market price $2/chip -> at $350k, that's ~175k hours. Or 10^24 FLOPs in total.

So, huge margin, but 10^22 - 10^24 flop is the band I think we can estimate.

These are the scale of numbers that show up in the chinchilla optimal paper, haha. Truly GPT-3 scale models.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#257

Earlier quoted context omitted.

It doesn't need to be general intelligence or perfectly map to human intelligence. All it needs to be is useful. Reading constant comments about LLMs can't be general intelligence or lack reasoning etc, to me seems like people witnessing the airplane and complaining that it isn't "real flying" because it isn't a bird flapping its wings (a large portion of the population held that point of view back then). It doesn't…

> to me seems like people witnessing the airplane and complaining that it isn't "real flying" because it isn't a bird flapping its wings To me it is more like there is someone jumping on a pogo ball while flapping their arms and saying that they are flying whenever they hop off the ground. Skeptics say that they are not really flying, while adherents say that "with current pogo ball advancements, they will be flying…

Is that what Elon Musk was trying to do on stage?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#258

Direct quote from the ARC-AGI blog: “SO IS IT AGI? ARC-AGI serves as a critical benchmark for detecting such breakthroughs, highlighting generalization power in a way that saturated or less demanding benchmarks cannot. However, it is important to note that ARC-AGI is not an acid test for AGI – as we've repeated dozens of times this year. It's a research tool designed to focus attention on the most challenging unsolve…

> Also, 1 odd thing I noticed is that the graph in their blog post shows the top 2 scores as “tuned”

Something I missed until I scrolled back to the top and reread the page was this

> OpenAI's new o3 system - trained on the ARC-AGI-1 Public Training set

So yeah, the results were specifically from a version of o3 trained on the public training set

Which on the one hand I think is a completely fair thing to do. It's reasonable that you should teach your AI the rules of the game, so to speak. There really aren't any spoken rules though, just pattern observation. Thus, if you want to teach the AI how to play the game, you must train it.

On the other hand though, I don't think the o1 models nor Claude were trained on the dataset, in which case it isn't a completely fair competition. If I had to guess, you could probably get 60% on o1 if you trained it on the public dataset as well.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#259
post #236

Complete aside here: I used to do work with amputees and prosthetics. There is a standardized test (and I just cannot remember the name) that fits in a briefcase. It's used for measuring the level of damage to the upper limbs and for prosthetic grading. Basically, it's got the dumbest and simplest things in it. Stuff like a lock and key, a glass of water and jug, common units of currency, a zipper, etc. It tests if y…

It would be interesting to see trick questions. Like in your test a hand grenade and a pin - don't pull the pin. Or maybe a mousetrap? but maybe that would be defused? in the ai test... or Global Thermonuclear War, the only winning move is...

to move first!

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#260
post #236

Complete aside here: I used to do work with amputees and prosthetics. There is a standardized test (and I just cannot remember the name) that fits in a briefcase. It's used for measuring the level of damage to the upper limbs and for prosthetic grading. Basically, it's got the dumbest and simplest things in it. Stuff like a lock and key, a glass of water and jug, common units of currency, a zipper, etc. It tests if y…

It would be interesting to see trick questions. Like in your test a hand grenade and a pin - don't pull the pin. Or maybe a mousetrap? but maybe that would be defused? in the ai test... or Global Thermonuclear War, the only winning move is...

Gaming streams being in the training data, it might pull the pin because "that's what you do".
Post reply on HN