OpenAI O3 breakthrough high score on ARC-AGI-PUB • Hacker News Board

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#741

Post by DiscourseFan » Sat, Dec 21, 2024, 7:47 AM UTC

a little from column A, a little from column B

I don't think this is AGI; nor is it something to scoff at. Its impressive, but its also not human-like intelligence. Perhaps human-like intelligence is not the goal, since that would imply we have even a remotely comprehensive understanding of the human mind. I doubt the mind operates as a single unit anyway, a human's first words are "Mama," not "I am a self-conscious freely self-determining being that recognizes my own reasoning ability and autonomy." And the latter would be easily programmable anyway. The goal here might, then, be infeasible: the concept of free will is a kind of technology in and of itself, it has already augmented human cognition. How will these technologies not augment the "mind" such that our own understanding of our consciousness is altered? And why should we try to determine ahead of time what will hold weight for us, why the "human" part of the intelligence will matter in the future? Technology should not be compared to the world it transforms.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#742

Post by 7734128 » Sat, Dec 21, 2024, 7:49 AM UTC

miga89 wrote: post #738

How do the organisers keep the private test set private? Does openAI hand them the model for testing? If they use a model API, then surely OpenAI has access to the private test set questions and can include it in the next round of training? (I am sure I am missing something.)

I suppose that's why they are calling it "semi-private".

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#743

Post by freehorse » Sat, Dec 21, 2024, 7:57 AM UTC

7734128 wrote: post #742

miga89 wrote: post #738

How do the organisers keep the private test set private? Does openAI hand them the model for testing? If they use a model API, then surely OpenAI has access to the private test set questions and can include it in the next round of training? (I am sure I am missing something.)

I suppose that's why they are calling it "semi-private".

And why o3 or any OpenAI llm is not evaluated in the actual private dataset.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#744

Post by owenpalmer » Sat, Dec 21, 2024, 7:57 AM UTC

miga89 wrote: post #738

How do the organisers keep the private test set private? Does openAI hand them the model for testing? If they use a model API, then surely OpenAI has access to the private test set questions and can include it in the next round of training? (I am sure I am missing something.)

I wouldn't be surprised if the term "benchmark fraud" will soon been coined.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#745

Post by [deleted] » Sat, Dec 21, 2024, 7:57 AM UTC

Engineering-MD wrote: post #575

Can I just say what a dick move it was to do this as a 12 days of Christmas. I mean to be honest I agree with the arguments this isn’t as impressive as my initial impression, but they clearly intended it to be shocking/a show of possible AGI, which is rightly scary. It feels so insensitive to that right before a major holiday when the likely outcome is a lot of people feeling less secure in their career/job/life. Tha…

[deleted]

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#746

Post by digitcatphd » Sat, Dec 21, 2024, 8:01 AM UTC

o3 fixes the fundamental limitation of the LLM paradigm – the inability to recombine knowledge at test time – and it does so via a form of LLM-guided natural language program search

> This is significant, but I am doubtful it will be as meaningful as people expect aside from potentially greater coding tasks. Without a 'world model' that has a contextual understanding of what it is doing, things will remain fundamentally throttled.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#747

Post by dyauspitr » Sat, Dec 21, 2024, 8:02 AM UTC

attentionmech wrote: post #21

Isn't this at the level now where it can sort of self improve. My guess is that they will just use it to improve the model and the cost they are showing per evaluation will go down drastically. So, next step in reasoning is open world reasoning now?

I don’t believe so. If it’s at the point where you could just plug it into a bunch of camera feeds around the world and it could only filter out a useful training set for itself out of that data then we truly would have AGI. I don’t think it’s there yet.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#748

Post by handzhiev » Sat, Dec 21, 2024, 8:04 AM UTC

__MatrixMan__ wrote: post #507

With only a 100x increase in cost, we improved performance by 0.1x and continued plotting this concave-down diminishing-returns type graph! Hurray for logarithmic x-axes! Joking aside, better than ever before at any cost is an achievement, it just doesn't exactly scream "breakthrough" to me.

How much time does a top sprinter take a 100 m run for compared to a mediocre sprinter?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#749

Post by owenpalmer » Sat, Dec 21, 2024, 8:05 AM UTC

lumost wrote: post #696

duluca wrote: post #689

The first computers cost millions of dollars and filled entire rooms to accomplish what we would now consider simple computational tasks. That same computing power now fits into the width of a finger nail. I don’t get how technologists balk at the cost of experimental tech or assume current tech will run at the same efficiency for decades to come and melt the planet into a puddle. AGI won’t happen until you can fit e…

The concern here is mainly on practicality. The original mainframes did not command startup valuations counted in fractions of the US economy, they did qualify for billions in investment. This is a great milestone, but OpenAI will not be successful charging 10x the cost of a human to perform a task.

> OpenAI will not be successful charging 10x the cost of a human to perform a task.

True, but they might be successful charging 20x for 2x the skill of a human.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#750

Post by dyauspitr » Sat, Dec 21, 2024, 8:05 AM UTC

para_parolu wrote: post #406

thisisthenewme wrote: post #313

I feel like AI is already changing how we work and live - I've been using it myself for a lot of my development work. Though, what I'm really concerned about is what happens when it gets smart enough to do pretty much everything better (or even close) than humans can. We're talking about a huge shift where first knowledge workers get automated, then physical work too. The thing is, our whole society is built around p…

From IT bubble it’s very easy to have impression that AI will replace most people. Most of people on my street do not work in IT. Teacher, nurse, hobby shop owner, construction workers, etc. Surely programming and other virtual work may become less paid job but it’s not end of the world.

Honestly with o3 levels of reasoning generating control software for robots on the fly, none of the above seem safe. For a decade or two at the most if that.