Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-6 Astra

openai.com

Post reply on HN
Print view View original

771–780 of 1001 posts

Re: GPT-6 Astra

#771

Post by gravypod »

intenex wrote: post #249

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

> For the many people who resist the AGI label possibly ever being achieved, I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.

I think I have the following questions about what AGI would look like:

1. Do you expect an AGI to be able to competently do any knowledge work that an able human does today?

I think this is implied by the "General" component. I would assume that anything we call AGI would be able to do any of these tasks if given the time and reference material needed.

2. How would you expect AGI to handle edge cases? (Missing context, no known solution, under specified instructions, over specified instructions)

I would expect an agent to be able to look at the context that work exists in and correctly attenuate it's intentions for these goals. Simpler solutions, more thorough reporting, etc based on the need.

3. In my work I attend meetings, write reports, write code, research things, etc. Would AGI be able to reliably do that?

I would say that AGI would need to do this. I would classify this as the "Intelligence" component. Obtaining context, building a model of a problem, solving it, and convincing others.

4. Would it be able to inspire trust in itself? Trust can be established through verification of it's outputs, the construction of introspective tools, no hallucinations, etc.

I would say yes to this as well. It would be a component of the "Intelligence" to know that buy in is more important than the completion of a task.

To these points, will Astra be able to do these things? If not, I would hesitate to call it AGI.

Re: GPT-6 Astra

#772

Post by Langleigh »

openaiscooked wrote: post #709
dalemhurley wrote: post #285

OpenAI is killing it now that they are more focused. Killing projects like Sora et al have seen it go from irrelevant to level footing with Anthropic. Sol is so much better than Fable 5. Then we get Astra (yet to use it) few days after Fable 5.1 (which is very impressive). Codex is slightly better than Claude Code. Good on Sam Altman getting back to basics and turning OpenAI around.

Killing Sora was one of the worst mistakes they ever made

100% they should have not given up on video.

That announcement is when I stopped paying attention to them.

Re: GPT-6 Astra

#774

Post by qsera »

intenex wrote: post #249

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

>what would make you think Astra is yet to be AGI...

Can it detect if I feed bullshit (by bullshit I mean stuff that contradicts with its own existing "knowledge") in its training data? If not, then I think it is a good indicator that it is not intelligent at all, let alone AGI...

And I think discussions on whether these models are AGI or not are AI marketing triggered. And that is exactly what these statements are targeting....HN appear to have fallen for it, as usual...

Re: GPT-6 Astra

#775

Post by geraneum »

drivebyhooting wrote: post #541

For people skeptical of AGI. Consider the following: 15 years ago if you were the sole proprietor of these models, would you be able to hold a dozen remote junior engineer jobs? Maybe even more? These models could certainly pass all interviews with flying colors and even survive independently in a company role. I think sole ownership of AI 15 years ago could be worth north of $10 million per year. Just as rank-and-fi…

For people delirious about AGI, you don’t get to it by redefining it favorably.

Re: GPT-6 Astra

#776

Post by nerevarthelame »

irthomasthomas wrote: post #457
intenex wrote: post #249

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

A model that can't beat gemini flash 3.8 on deepSWE is not AGI. I would not be surprised if ARC skills don't carry over to real tasks. In that case, training for ARC could even hurt real world performance. I have't looked in a while, but I wonder if there has been any research testing ARCs predictive power?

I feel like I'm completely missing something with ARC-AGI. The tasks are so limited in scale and very black and white, which do not at all map to real-life challenges.

I do think it's impressive that LLMs can reliably solve them, and I recognize LLMs are getting much better at navigating more ambiguous and expansive tasks. But I'm not impressed by any person who can solve ARC-AGIs, and nor would I even look down on a person who couldn't solve them all. I'd certainly never consider ARC-AGI results when deciding whether to hire someone.

Re: GPT-6 Astra

#777

Post by zarzavat »

tintor wrote: post #673

- OpenAI claims Astra beats all benchmarks (compared to Fable and Opus, except "Humanity's Last Exam (w/ tools)"): https://openai.com/index/gpt-6-astra/ - Artificial Analysis scores Astra (max effort) as 61 points on intelligence, behind Opus 5. https://artificialanalysis.ai/models/gpt-6-astra Who is wrong here? Some benchmark results in Astra page for Fable and Opus are blank (-). What is Artificial Analysis intelli…

> Humanity's Last Exam (w/ tools)

This is one of the only benchmarks that actually matters for testing the frontier however. Other benchmarks can be gamed by simply being more persistent, but HLE is a diverse set of open-ended research-level questions. It tests domain knowledge and problem solving skills. Burning more reasoning tokens may help somewhat but not as much as e.g. coding benchmarks.

Re: GPT-6 Astra

#779

Post by ramblerman »

kulkarniamey wrote: post #734

The model is probably excellent. The problem here is AGI having various definitions and many of them getting narrowed down to whatever makes benchmark numbers look good.

I agree, but I also long considered llm's stochastic parrots. Then this year happened.

Opus/Sol are easily far smarter programmers than I, and this thing supposedly blows them out of the water. Once an LLM is a better doctor, researcher, biologist, chemist, mathematician, physicist than any human is that not AGI?

It didn't arrive in the form I would have ever imagined, but it's hard to say its not (imo).

Re: GPT-6 Astra

#780

Post by DaSHacka »

tintor wrote: post #697
quyleanh wrote: post #642

> We also tested Astra on SRE-Bench [15], a benchmark that measures whether models can reverse engineer software binaries to understand its core logic without access to raw source code. Astra solved 88.0% of tasks in a single attempt and 99.2% within four attempts, compared with 55.9% and 68.7% for GPT‑5.6 Sol, respectively. So the closed source application should open its source in near future? [15] https://arxiv.or…

Not if OpenAI considers reverse engineering an offensive cybersecurity skill.

Surprisingly, I've had really good luck with reverse engineering on frontier models (without being part of CVP or similar). It's more the exploit development/PoC that it locks up on, which, as I use `pi`, I just switch the model to Kimi K3 to finish up making the PoC.

Ironically, due to the stringent guardrails on American models that exist to avoid giving adversaries a leg-up in cybersecurity, I end up feeding dozens of 0days straight to the CCP lol

Post reply on HN