Is it just me or does looking at the ARC-AGI example questions at the bottom... make your brain hurt?
OpenAI O3 breakthrough high score on ARC-AGI-PUB
61–70 of 1001 posts
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#62So now not only are the models closed, but so are their evals?! This is a "semi-private" eval. WTH is that supposed to mean? I'm sure the model is great but I refuse to take their word for it.
The private evaluation set is private from the public/OpenAI so companies can't train on those problems and cheat their way to a high score by overfitting.
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#63Human performance is 85% [1]. o3 high gets 87.5%. This means we have an algorithm to get to human level performance on this task. If you think this task is an eval of general reasoning ability, we have an algorithm for that now. There's a lot of work ahead to generalize o3 performance to all domains. I think this explains why many researchers feel AGI is within reach, now that we have an algorithm that works. Congrat…
It's not saturated. 85% is average human performance, not "best human" performance. There is still room for the model to go up to 100% on this eval.
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#64My initial impression: it's very impressive and very exciting. My skeptical impression: it's complete hubris to conflate ARC or any benchmark with truly general intelligence. I know my skepticism here is identical to moving goalposts. More and more I am shifting my personal understanding of general intelligence as a phenomenon we will only ever be able to identify with the benefit of substantial retrospect. As it is…
These comments are getting ridiculous. I remember when this test was first discussed here on HN and everyone agreed that it clearly proves current AI models are not "intelligent" (whatever that means). And people tried to talk me down when I theorised this test will get nuked soon - like all the ones before. It's time people woke up and realised that the old age of AI is over. This new kind is here to stay and it will take over the world. And you better guess it'll be sooner rather than later and start to prepare.
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#65Uhh...some of us are apparently living under a rock, as this is the first time I hear about o3 and I'm on HN far too much every day
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#66So now not only are the models closed, but so are their evals?! This is a "semi-private" eval. WTH is that supposed to mean? I'm sure the model is great but I refuse to take their word for it.
the evals are the question/answers, ARC-AGI doesn't share the questions and answers for a portion so that models can't be trained on them, the public ones... the public knows the questions so theres a chance they could have been at least partially been trained on the question (if not the actual answer).
Thats how i understand it
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#67Human performance is 85% [1]. o3 high gets 87.5%. This means we have an algorithm to get to human level performance on this task. If you think this task is an eval of general reasoning ability, we have an algorithm for that now. There's a lot of work ahead to generalize o3 performance to all domains. I think this explains why many researchers feel AGI is within reach, now that we have an algorithm that works. Congrat…
Still it's comparing average human level performance with best AI performance. Examples of things o3 failed at are insanely easy for humans.
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#68Besides higher scores - is there any improvements for a general use? Like asking to help setup home assistant etc etc?
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#69How much longer can I get paid $150k to write code ?
3 to 5 years, max. Traditional coding is going to be dead in the water. Optimistically, the junior SWE job will evolve but more realistically dedicated AI-based programming agents will end demand for Junior SWEs
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#70My initial impression: it's very impressive and very exciting. My skeptical impression: it's complete hubris to conflate ARC or any benchmark with truly general intelligence. I know my skepticism here is identical to moving goalposts. More and more I am shifting my personal understanding of general intelligence as a phenomenon we will only ever be able to identify with the benefit of substantial retrospect. As it is…
These comments are getting ridiculous. I remember when this test was first discussed here on HN and everyone agreed that it clearly proves current AI models are not "intelligent" (whatever that means). And people tried to talk me down when I theorised this test will get nuked soon - like all the ones before. It's time people woke up and realised that the old age of AI is over. This new kind is here to stay and it wil…
What kind of preparation are you suggesting?