Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

761–770 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#761

Post by nopinsight »

duluca wrote: post #689

The first computers cost millions of dollars and filled entire rooms to accomplish what we would now consider simple computational tasks. That same computing power now fits into the width of a finger nail. I don’t get how technologists balk at the cost of experimental tech or assume current tech will run at the same efficiency for decades to come and melt the planet into a puddle. AGI won’t happen until you can fit e…

Many of humans' capabilities are pretrained with massive computing through evolution. Inference results of o3 and its successors might be used to train the next generation of small models to be highly capable. Recent advances in the capabilities of small models such as Gemini-2.0 Flash suggest the same.

Recent research from NVIDIA suggests such an efficiency gain is quite possible in the physical realm as well. They trained a tiny model to control the full body of a robot via simulations.

---

"We trained a 1.5M-parameter neural network to control the body of a humanoid robot. It takes a lot of subconscious processing for us humans to walk, maintain balance, and maneuver our arms and legs into desired positions. We capture this “subconsciousness” in HOVER, a single model that learns how to coordinate the motors of a humanoid robot to support locomotion and manipulation."

...

"HOVER supports any humanoid that can be simulated in Isaac. Bring your own robot, and watch it come to life!"

More here: https://x.com/DrJimFan/status/1851643431803830551

---

This demonstrates that with proper training, small models can perform at a high level in both cognitive and physical domains.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#763

Post by torginus »

hamburga wrote: post #627

I’m not sure if people realize what a weird test this is. They’re these simple visual puzzles that people can usually solve at a glance, but for the LLMs, they’re converted into a json format, and then the LLMs have to reconstruct the 2D visual scene from the json and pick up the patterns. If humans were given the json as input rather than the images, they’d have a hard time, too.

Not sure how much that matters - I'm not an AI expert, but I did some intro courses where we had to train a classifier to recognize digits. How it worked basically was that we fed each pixel of the 2d grid of the image into an input of the network, essentially flattening it in a similar fashion. It worked just fine, and that was a tiny network.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#764

Post by bigprof »

nopinsight wrote: post #761
duluca wrote: post #689

The first computers cost millions of dollars and filled entire rooms to accomplish what we would now consider simple computational tasks. That same computing power now fits into the width of a finger nail. I don’t get how technologists balk at the cost of experimental tech or assume current tech will run at the same efficiency for decades to come and melt the planet into a puddle. AGI won’t happen until you can fit e…

Many of humans' capabilities are pretrained with massive computing through evolution. Inference results of o3 and its successors might be used to train the next generation of small models to be highly capable. Recent advances in the capabilities of small models such as Gemini-2.0 Flash suggest the same. Recent research from NVIDIA suggests such an efficiency gain is quite possible in the physical realm as well. They…

> Similarly, many of humans' capabilities are pretrained with massive computing through evolution.

Hmm .. my intuition is that humans' capabilities are gained during early childhood (walking, running, speaking .. etc) ... what are examples of capabilities pretrained by evolution, and how does this work?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#765

Post by internet_points »

modeless wrote: post #13

Congratulations to Francois Chollet on making the most interesting and challenging LLM benchmark so far. A lot of people have criticized ARC as not being relevant or indicative of true reasoning, but I think it was exactly the right thing. The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning. It's obvious to everyone that…

> The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning.

One might also interpret that as "the fact that models which are studying to the test are getting better at the test" (Goodhart's law), not that they're actually reasoning.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#766

Post by globalise83 »

8n4vidtmkvmk wrote: post #735
duluca wrote: post #689

The first computers cost millions of dollars and filled entire rooms to accomplish what we would now consider simple computational tasks. That same computing power now fits into the width of a finger nail. I don’t get how technologists balk at the cost of experimental tech or assume current tech will run at the same efficiency for decades to come and melt the planet into a puddle. AGI won’t happen until you can fit e…

I thought you were going to say that now we're back to bigger-than-room sized computers that cost many millions just to perform the same tasks we could 40 years ago. I of course mean we're using these LLMs for a lot of tasks that they're inappropriate for, and a clever manually coded algorithm could do better and much more efficiently.

The LLMs are now writing their own algorithms to answer questions. Not long before they can design a more efficient algorithm to complete any feasible computational task, in a millionth of the time needed by the best human.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#767

Post by Existenceblinks »

duluca wrote: post #689

The first computers cost millions of dollars and filled entire rooms to accomplish what we would now consider simple computational tasks. That same computing power now fits into the width of a finger nail. I don’t get how technologists balk at the cost of experimental tech or assume current tech will run at the same efficiency for decades to come and melt the planet into a puddle. AGI won’t happen until you can fit e…

Honestly, it doesn't need to be local, API is some 200ms away is ok-ish, make it 50ms it will be practically usable for every majority of interaction.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#768

Post by owenpalmer »

Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it.

I = E / K

where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge.

For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happens to have more prior knowledge of math than student B. In this case, the intelligence of B is greater than the intelligence of A, even though they have the same effectiveness. B was able to "figure out" the math, without using any of the "tricks" that A already knew.

Now back to the question of whether or not prior knowledge is required. As K approaches 0, intelligence approaches infinity. But when K=0, intelligence is undefined. Tada! I think that answers the question.

Most LLM benchmarks simply measure effectiveness, not intelligence. I conceptualize LLMs as a person with a photographic memory and a low IQ of 85, who was given 100 billion years to learn everything humans have ever created.

IK = E

low intelligence * vast knowledge = reasonable effectiveness

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#769

Post by deneas »

miga89 wrote: post #738

How do the organisers keep the private test set private? Does openAI hand them the model for testing? If they use a model API, then surely OpenAI has access to the private test set questions and can include it in the next round of training? (I am sure I am missing something.)

They have two sets, a fully private one where the models run isolated and the semi-private one where they run models accessed over the internet.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#770

Post by Woodi »

So article seriously and scientifically states:

"Our programs compilation (AI) gave 90% of correct answers in test 1. We expect that in test 2 quality of answers will degenerate to below random monkey pushing buttons levels. Now more money is needed to prove we hit blind alley."

Hurray ! Put limited version of that on everybody phones !

Post reply on HN