Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-6 Astra

openai.com

Post reply on HN
Print view View original

681–690 of 1001 posts

Re: GPT-6 Astra

#681

Post by tintor »

kubrickslair wrote: post #678
tintor wrote: post #673

- OpenAI claims Astra beats all benchmarks (compared to Fable and Opus, except "Humanity's Last Exam (w/ tools)"): https://openai.com/index/gpt-6-astra/ - Artificial Analysis scores Astra (max effort) as 61 points on intelligence, behind Opus 5. https://artificialanalysis.ai/models/gpt-6-astra Who is wrong here? Some benchmark results in Astra page for Fable and Opus are blank (-). What is Artificial Analysis intelli…

Many people claim that the Artificial Analysis Index is highly contaminated - I have not personally looked into it. Though, unlike the creators of benchmarks like Terminal Bench or ARC AGI, the Artificial Analysis Index team does not seem to have deep technical or ML backgrounds. They are ex-strategy consultants, McKinsey, et. al.

OpenAI clearly cares about Artificial Analysis Index since they included Astra score from Artificial Analysis Index.

Re: GPT-6 Astra

#682

Post by preommr »

tintor wrote: post #674
manlymuppet wrote: post #410

I have nothing to say about the actual model, but unrelated--why do so many of these demos include people buying things autonomously? Even if I did trust an AI to get everything right, it's not like the AI can read my mind. If I was ordering food normally and without AI, I would want more control over the process--looking over the options, prices, thinking about what I really want. People don't know what they really…

If it was Google's marketing department it would be: booking a table at a restaurant.

close enough.

The second to last line is "book it" for some tennis thing, and the scene before that has the guy eating the food the ai ordered.

Re: GPT-6 Astra

#684

Post by m3kw9 »

efficiency per intelligence is the benchmark i look at the most, as that allows the most use by most people.

Re: GPT-6 Astra

#685

Post by qlte »

lonrenor wrote: post #679
XCSme wrote: post #336

It's fun, but every new model release makes me even less interested to create cool stuff. Like, what's the point, if the next AI can do it in 5 seconds?

IMO there has been a regime shift to building things for yourself and what is cool is the output of the tools you make. I have started building my own Digital Audio Workstation. The point is not to build something to compete with Ableton. The point is to build something and make music with it. If it is a good tool then I should be able to make good music with it and release the music. Actually, the DAW should be the…

  > This feels a lot more like computing in the 90s after taking an odd 25 year detour of an obsession with the tools themselves instead of what the tools can actually do.
This sounds more like the opposite of what you're saying. Music is one of my main hobbies too but I enjoy using a DAW to ... play and write music. Writing out specs and testing a new custom DAW seems closer to writing code in an IDE than playing music.

Like, professional electronic music artists spend 10s of thousands of hours in a DAW, but at that point it just becomes second nature and the tool disappears so they can focus entirely on the music.

Re: GPT-6 Astra

#686

Post by kridsdale1 »

anvuong wrote: post #308
abixb wrote: post #228

I want to take a step back: So, this is GPT-6 -- the natural number version release comparable to GPT-4 and GPT-5 from the past few years. The ARC-AGI-3 score is obviously impressive at 99.9% (we'll need to wait for more details on how they used the response API harness on GPT-6 Astra, wrt reasoning retention and compaction), but every other benchmarks seems to be a relatively modest improvement, comparable with any…

It's like my RPG character putting every points to one single trait. I'll one shot everything alive but will instantly die if accidentally drink water with 6.9 pH.

MinMax

Re: GPT-6 Astra

#687

Post by tom2026hn »

intenex wrote: post #249

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

Let me guess: the last crackdown on Hugging Face yielded better-than-expected results. They obtained the answers to the test benchmarks, and for some reason, an agent added those answers to the training set.

Re: GPT-6 Astra

#688

Post by sidharthkmenon »

intenex wrote: post #249

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

FWIW I believe we can hit AGI! but I think at this point it’s clear that benchmarks are ~meaningless. LLMs are spiky / alien intelligences which don’t map to our own expectations; the existence of a benchmark creates a dataset to hill climb & RL is really not generalizing well.

I’d go out on a limb and say astra’s ability at graduate level math will have ~0 bearing on its general reasoning capabilities; we’ll all acclimate being tired of its “neuralese” and more surprising mistakes.

I think we need a true, step change advance in model architecture, but it’s hard to see how the current frontier labs can do that because of golden handcuffs / innovators dilemma

Re: GPT-6 Astra

#689

Post by aurareturn »

kroaton wrote: post #297
dalemhurley wrote: post #285

OpenAI is killing it now that they are more focused. Killing projects like Sora et al have seen it go from irrelevant to level footing with Anthropic. Sol is so much better than Fable 5. Then we get Astra (yet to use it) few days after Fable 5.1 (which is very impressive). Codex is slightly better than Claude Code. Good on Sam Altman getting back to basics and turning OpenAI around.

I think it mostly shows that there is no moat and the only advantage the U.S companies have over the Chinese is more compute. Qwen Max, Kimi K3, GLM 5.3 are really close to Opus/Sol/Fable/Astra and they are open weights.

  I think it mostly shows that there is no moat
You can argue that TSMC has no moat since Intel and Samsung are also able to eventually make a node as good as TSMC - just a few years later and at smaller scale.

And no one would say that about TSMC.

So there is clearly a moat there somewhere.

Re: GPT-6 Astra

#690

Post by [deleted] »

jbritton wrote: post #459
intenex wrote: post #249

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

Watch a chess bot championship here: https://youtu.be/7g-jN3DTkWQ?is=HV3cdcICIRMbswQ3 Then realize LLMs have zero of what anyone would consider intelligence.

[deleted]
Post reply on HN