Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

781–790 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#781

Post by olalonde »

zamadatix wrote: post #297
bluecoconut wrote: post #201

Earlier quoted context omitted.

some other imporant quotes: "Average human off the street: 70-80%. STEM college grad: >95%. Panel of 10 random humans: 99-100%" -@fchollet on X So, considering that the $3400/task system isn't able to compete with STEM college grad yet, we still have some room (but it is shrinking, i expect even more compute will be thrown and we'll see these barriers broken in coming years) Also, some other back of envelope calculat…

I don't follow how 10 random humans can beat the average STEM college grad and average humans in that tweet. I suspect it's really "a panel of 10 randomly chosen experts in the space" or something? I agree the most interesting thing to watch will be cost for a given score more than maximum possible score achieved (not that the latter won't be interesting by any means).

Even if you assume that non STEM grads are dumb, isn't there a good probability of having a STEM graduate among 10 random humans?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#782

Post by razodactyl »

norir wrote: post #220
nopinsight wrote: post #125

Let me go against some skeptics and explain why I think full o3 is pretty much AGI or at least embodies most essential aspects of AGI. What has been lacking so far in frontier LLMs is the ability to reliably deal with the right level of abstraction for a given problem. Reasoning is useful but often comes out lacking if one cannot reason at the right level of abstraction. (Note that many humans can't either when they…

Personally I find "human-level" to be a borderline meaningless and limiting term. Are we now super human as a species relative to ourselves just five years ago because of our advances in developing computer programs that better imitate what many (but far from all) of us were already capable of doing? Have we reached a limit to human potential that can only be surpassed by digital machines? Who decides what human leve…

No I think we're just tired and depressed as a species... Existing systems work to a degree but aren't living up to their potential of increasing happiness according to technological capabilities.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#783

Post by spacebanana7 »

lorepieri wrote: post #777
owenpalmer wrote: post #768

Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…

There should be also a factor about resource consumption. See here: https://lorenzopieri.com/pgii/

Also perhaps a factor (with diminishing returns) for response speed?

All else equal, a student who gets 100% on a problem set in 10 minutes is more intelligent than one with the same score after 120 minutes. Likewise an LLM that can respond in 2 seconds is more impressive than one which responds in 30 seconds.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#784

Post by someothherguyy »

owenpalmer wrote: post #768

Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…

https://en.wikipedia.org/wiki/Fluid_and_crystallized_intelli...

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#785

Post by keiferski »

Engineering-MD wrote: post #575

Can I just say what a dick move it was to do this as a 12 days of Christmas. I mean to be honest I agree with the arguments this isn’t as impressive as my initial impression, but they clearly intended it to be shocking/a show of possible AGI, which is rightly scary. It feels so insensitive to that right before a major holiday when the likely outcome is a lot of people feeling less secure in their career/job/life. Tha…

The vast majority of people who will lose jobs to AI aren’t following AGI benchmarks, or even know what AGI is short for.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#786

Post by hanspeter »

wangii wrote: post #775
owenpalmer wrote: post #768

Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…

Interesting formulation! it captures the intuition of the "smartness" when solving a problem. However, what about asking good questions or proposing conjectures?

Aren't those solutions to problems as well?

Find the best questions to ask. Find the best hypothesis to suggest.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#787

Post by dmead »

obblekk wrote: post #42

Human performance is 85% [1]. o3 high gets 87.5%. This means we have an algorithm to get to human level performance on this task. If you think this task is an eval of general reasoning ability, we have an algorithm for that now. There's a lot of work ahead to generalize o3 performance to all domains. I think this explains why many researchers feel AGI is within reach, now that we have an algorithm that works. Congrat…

This is so strange. people think that an llm trained on programming questions and docs can do mundane tasks like this means intelligent? Come on.

It really calls into question two things.

1. You don't know what you're talking about about.

2. You have a perverse incentive to believe this such that you will preach it to others and elevate some job salary range or stock.

Either way, not a good look.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#788

Post by exe34 »

Jensson wrote: post #442
exe34 wrote: post #88

Earlier quoted context omitted.

> the other is that there was an approach that made the task easier than we expected. from reading Dennett's philosophy, I'm convinced that that's how human intelligence works - for each task that "only a human could do that", there's a trick that makes it easier than it seems. We are bags of tricks.

> We are bags of tricks. We are trick generators, that is what it means to be a general intelligence. Adding another trick in the bag doesn't make you a general intelligence, being able to discover and add new tricks yourself makes you a general intelligence.

generating tricks is itself a trick that relies on an enormous bag of tricks we inherited through evolution by the process of natural selection.

the new tricks don't just pop into our heads even though it seems that way. nobody ever woke up and devised a new trick in a completely new field without spending years learning about that field or something adjacent to it. even the new ideas tend to be an old idea from a different field applied to a new field. tricks stand on the shoulders of giants.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#789

Post by croes »

stocknoob wrote: post #650
ripped_britches wrote: post #544

Sad to see everyone so focused on compute expense during this massive breakthrough. GPT-2 originally cost $50k to train, but now can be trained for ~$150. The key part is that scaling test-time compute will likely be a key to achieving AGI/ASI. Costs will definitely come down as is evidenced by precedents, Moore’s law, o3-mini being cheaper than o1 with improved performance, etc.

It’s wild, are people purposefully overlooking that inference costs are dropping 10-100x each year? https://a16z.com/llmflation-llm-inference-cost/ Look at the log scale slope, especially the orange MMLU > 83 data points.

A bit early for a every year claim not to mention what all these AI is used for.

In some parts of the internet it’s you hardly find real content only AI spam.

It will get worse the cheaper it gets.

Think of email spam.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#790

Post by PhilippGille »

owenpalmer wrote: post #744
miga89 wrote: post #738

How do the organisers keep the private test set private? Does openAI hand them the model for testing? If they use a model API, then surely OpenAI has access to the private test set questions and can include it in the next round of training? (I am sure I am missing something.)

I wouldn't be surprised if the term "benchmark fraud" will soon been coined.

Benchmark fraud is not a novel concept. Outside of LLMs for example smartphone manufacturers detect benchmarks and disable or reduce CPU throttling: https://www.theregister.com/2019/09/30/samsung_benchmarking_...
Post reply on HN