GPT-6 Astra • Hacker News Board

Re: GPT-6 Astra

#341

Post by waffletower » Thu, Sep 03, 2026, 8:59 PM UTC

simianwords wrote: post #301

intenex wrote: post #248

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

> where I am reasonably confident that there's essentially nothing that I am better than Fable No. Humans are still better at super long context learning. Once that is beat you are completely correct.

I am much better at listening to Charli XCX than Fable, and much better at driving a Nissan Leaf than Fable (and much better than Tesla at driving a Tesla).

Re: GPT-6 Astra

#342

Post by gavinray » Thu, Sep 03, 2026, 8:59 PM UTC

mvkel wrote: post #263

intenex wrote: post #248

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

Take it from the mouth of the creator of ARC-AGI: When we released ARC 3, I got asked, "when do you think a frontier model will saturate it?", and I answered "in about a year, though it depends on how much it gets explicitly targeted" That was 6 months ago, so the progress that Astra represents happened about 2x faster than I anticipated. I think the speed of progress will surprise a lot of people, and what the new m…

Human brains have difficulty reasoning about exponential growth.

Re: GPT-6 Astra

#343

Post by cryptoz » Thu, Sep 03, 2026, 9:00 PM UTC

uludag wrote: post #302

intenex wrote: post #248

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

Wouldn't "general intelligence" require so much more than scoring well (or even amazingly) on benchmarks? Like what about having some "AGI model" embodied in something (maybe humanoid), and test it by having it step in an assortment of cars and park them. Does bodily-kinesthetic intelligence account for nothing? Humans are intelligent creatures and can dynamically adapt to the physical shape of a variety of vehicles…

What you've described is just a new benchmark, though. It'll be called CarParkBench, various embodied LLMs will then be run against that benchmark, and some will score better than others.

I do see where you're going, but that's already what's happening: we have so many different benchmarks because there's no real single way to test for general intelligence.

Also, it takes a human probably at least a decade of world experience, growth, learning, etc, to pass your benchmark. I'm quite confident that it will be very soon that an embodied LLM will pass your new benchmark, much sooner than a human would take if born today.

Re: GPT-6 Astra

#344

Post by silver_sun » Thu, Sep 03, 2026, 9:00 PM UTC

XCSme wrote: post #327

tintor wrote: post #78

ARC AGI-3 saturated by Astra! https://arcprize.org/leaderboard

The no-reasoning version scores 35% while the low reasoning one scores 17%? What?

It's simulating the Dunning-Kruger effect.

Re: GPT-6 Astra

#345

Post by sashank_1509 » Thu, Sep 03, 2026, 9:01 PM UTC

XCSme wrote: post #335

It's fun, but every new model release makes me even less interested to create cool stuff. Like, what's the point, if the next AI can do it in 5 seconds?

Agreed

Re: GPT-6 Astra

#346

Post by gavinray » Thu, Sep 03, 2026, 9:01 PM UTC

jdprgm wrote: post #288

Is anyone else just exhausted by the pace of all this. The models change constantly and relentlessly and so does the pricing, basically weekly at this point between all the labs. It feels nearly impossible to have any rigorous approach when choosing a particular model and price point for a task and more like blindly picking one. The time period needed to actually get familiar with various models to a degree you can i…

  > Is anyone else just exhausted by the pace of all this.

This is only the beginning. We are in the infancy of AI, progress will continue to accelerate until some filtering event or energy limitation happens.

Re: GPT-6 Astra

#347

Post by re-thc » Thu, Sep 03, 2026, 9:02 PM UTC

Pikamander2 wrote: post #310

jdprgm wrote: post #288

Is anyone else just exhausted by the pace of all this. The models change constantly and relentlessly and so does the pricing, basically weekly at this point between all the labs. It feels nearly impossible to have any rigorous approach when choosing a particular model and price point for a task and more like blindly picking one. The time period needed to actually get familiar with various models to a degree you can i…

That's how cutting edge tech has always worked. Imagine buying a shiny new PC in the 90s only to see it become practically obsolete within a year.

Hardware definitely has longer lifecycle than AI model releases at this point.

You don't see Nvidia and AMD fighting every other month over the latest cards.

Re: GPT-6 Astra

#348

Post by jeffybefffy519 » Thu, Sep 03, 2026, 9:03 PM UTC

dalemhurley wrote: post #284

OpenAI is killing it now that they are more focused. Killing projects like Sora et al have seen it go from irrelevant to level footing with Anthropic. Sol is so much better than Fable 5. Then we get Astra (yet to use it) few days after Fable 5.1 (which is very impressive). Codex is slightly better than Claude Code. Good on Sam Altman getting back to basics and turning OpenAI around.

Its funny, my experience with Sol has been awful. It really overworks problems and tracks into areas it does not need to...

I just dont get how its good for some, and bad for others. It makes me suspect that the models performance is not even against problem sets and it really is just a probabilistic prediction machine. Which then makes me very skeptical of GPT-6 Astra, because if their big claim is Computer Use then it is probably bad in a bunch of other areas.

Re: GPT-6 Astra

#349

Post by balefulboy » Thu, Sep 03, 2026, 9:03 PM UTC

damsta wrote: post #230

Why release it now instead waiting those few days until it is available for everybody?

Because they saw how much hype Glasswing was getting in April

Re: GPT-6 Astra

#350

Post by serjester » Thu, Sep 03, 2026, 9:03 PM UTC

Exciting but it’s priced at 2.5X Sol - we haven’t seen pricing this high since GPT 4.5. We will see if the real world use cases outweigh the sticker shock.