GPT-6 Astra • Hacker News Board

Re: GPT-6 Astra

#501

Post by Yizahi » Thu, Sep 03, 2026, 10:21 PM UTC

intenex wrote: post #248

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

> I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.

Can Astra, or any other model explain how exactly it reached this or that output result? Start with a simple query of asking to add 55+66 for example. (no LLM program can do that)

Can Astra, or any other model refuse to answer or go on "thinking" in a orthogonal direction on it's own?

That's just two quick ideas, I'm pretty sure cognition scientists can invent better and wider range of checks.

Re: GPT-6 Astra

#502

Post by BadBrands » Thu, Sep 03, 2026, 10:21 PM UTC

So they’re copying Gemini with the whole star motif?

I guess it makes sense they are unoriginal.

like Zuck, @sama never invented anything or innovated at all - just took other people’s ideas

Re: GPT-6 Astra

#503

Post by ertgbnm » Thu, Sep 03, 2026, 10:23 PM UTC

abixb wrote: post #227

I want to take a step back: So, this is GPT-6 -- the natural number version release comparable to GPT-4 and GPT-5 from the past few years. The ARC-AGI-3 score is obviously impressive at 99.9% (we'll need to wait for more details on how they used the response API harness on GPT-6 Astra, wrt reasoning retention and compaction), but every other benchmarks seems to be a relatively modest improvement, comparable with any…

This is a very mundane release compared to GPT-4 and GPT-5. I think they probably scaled back a bit after the lukewarm response to the GPT-5 announcement. But it still very weird that there wasn't even a livestream,

Re: GPT-6 Astra

#504

Post by ChadMoran » Thu, Sep 03, 2026, 10:23 PM UTC

dalemhurley wrote: post #284

OpenAI is killing it now that they are more focused. Killing projects like Sora et al have seen it go from irrelevant to level footing with Anthropic. Sol is so much better than Fable 5. Then we get Astra (yet to use it) few days after Fable 5.1 (which is very impressive). Codex is slightly better than Claude Code. Good on Sam Altman getting back to basics and turning OpenAI around.

Sol better than Fable? What? I've found it to basically be on part with Opus and I max out 2 accounts on both providers every week.

Re: GPT-6 Astra

#505

Post by HDBaseT » Thu, Sep 03, 2026, 10:24 PM UTC

"Claude Fable 5 and 5.1 are not included in LifeSciBench Gold v1, GeneBench Pro v13, and MedChemBench because they refuse the majority of questions in these evaluations.12"

Sounds about right. Alignment is important, but also being able to do mundane tasks is important too.

Re: GPT-6 Astra

#506

Post by ObnoxiousProxy » Thu, Sep 03, 2026, 10:24 PM UTC

Yizahi wrote: post #501

intenex wrote: post #248

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

> I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward. Can Astra, or any other model explain how exactly it reached this or that output result? Start with a simple query of asking to add 55+66 for example. (no LLM program can do that) Can Astra, or any other model refuse to answer or go on "th…

In my opinion this is goal post moving. Humans do many things that we cannot fully explain either without post decision rationalization, and not all intelligent humans are deeply introspective.

Re: GPT-6 Astra

#507

Post by jameson » Thu, Sep 03, 2026, 10:26 PM UTC

intenex wrote: post #248

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

I agree that it's misleading but harness is now an essential part of LLM's effectiveness. It's safe to assume that LLM-alone-AGI is not coming anytime soon, given most of the frontier LLM vendors are developing their own harness.

Also the training dataset is proprietary and they'll drive the LLM's behavior, so it make sense for the vendors to invest in the harness and bake in prompts that work best with their models.

Re: GPT-6 Astra

#508

Post by rjtc » Thu, Sep 03, 2026, 10:26 PM UTC

I am really confused on how it can saturate ARC-AGI but still perform poorly on aggregated benchmarks:

https://artificialanalysis.ai/models

Perhaps if it was allowed this custom harness for all benchmarks it would similarily saturate?

Re: GPT-6 Astra

#509

Post by nater5000 » Thu, Sep 03, 2026, 10:27 PM UTC

thomasahle wrote: post #460

abixb wrote: post #227

I want to take a step back: So, this is GPT-6 -- the natural number version release comparable to GPT-4 and GPT-5 from the past few years. The ARC-AGI-3 score is obviously impressive at 99.9% (we'll need to wait for more details on how they used the response API harness on GPT-6 Astra, wrt reasoning retention and compaction), but every other benchmarks seems to be a relatively modest improvement, comparable with any…

• 98.6% on ARC-AGI-3 • 97.6% on frontier math • 95.9% on CAD • 100% on ExploitBench Nothing modest about it

Except the release announcement. You know, the thing the OP you're responding to is specifically pointing out?

Re: GPT-6 Astra

#510

Post by doctoboggan » Thu, Sep 03, 2026, 10:28 PM UTC

pera wrote: post #387

intenex wrote: post #248

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

To each their own. Personally I will start feeling the AGI as soon as we move from chatting about benchmark results to learn that some lab just announced the discovery of tens of novel treatments for rare diseases. Maybe I'm too boring but it seems quite pointless to have this same prediction game every time a new model is released.

Wouldn’t that be ASI? I.e. surpassing humans by outputting novel treatments at a far greater rate than normal humans?