Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-6 Astra

openai.com

Post reply on HN
Print view View original

751–760 of 1001 posts

Re: GPT-6 Astra

#751

Post by scandals »

tom2026hn wrote: post #678
intenex wrote: post #248

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

Let me guess: the last crackdown on Hugging Face yielded better-than-expected results. They obtained the answers to the test benchmarks, and for some reason, an agent added those answers to the training set.

Per the Fireship video, it was less that the answers were in the training set and more that the ability to calculate the flag on Exploitbench was left in from the previous test that went awry.

Scoring 100% is easy if noone checks your work

https://youtu.be/0Rp9KJCEIvg

Re: GPT-6 Astra

#752

Post by Marha01 »

lwansbrough wrote: post #616
intenex wrote: post #248

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

To me AGI has always meant sentience. And only since we’ve discovered that you can have something that is intelligent without it being apparently sentient that we’ve changed the definition to being, I suppose, more exactly aligned with the namesake. A real AGI, like the ones from science fiction, would make Astra look like a child’s toy. And I guess more concretely I would expect it to inhibit the following propertie…

> To me AGI has always meant sentience.

Sentience and intelligence are different things. Many humans and animals are pretty dumb, yet they are sentient. AI is intelligent, but non-sentient (hopefully!).

Re: GPT-6 Astra

#753

Post by winrid »

XCSme wrote: post #335

It's fun, but every new model release makes me even less interested to create cool stuff. Like, what's the point, if the next AI can do it in 5 seconds?

lol have you actually tried to build anything useful e2e? Leaving the AI to itself gives horrendous results.

Re: GPT-6 Astra

#754

Post by winrid »

gavinray wrote: post #355
XCSme wrote: post #335

It's fun, but every new model release makes me even less interested to create cool stuff. Like, what's the point, if the next AI can do it in 5 seconds?

> Like, what's the point, if the next AI can do it in 5 seconds? Live a life doing whatever makes you happy. Post-work society is an inevitability if we don't destroy our planet.

How do you imagine a post work society where people can have inequality? I don't want to be equal. I want to do more.

Re: GPT-6 Astra

#755

Post by a2ff6eeb0 »

ggsp wrote: post #747
intenex wrote: post #248

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

If you define AGI as "can do the work of a human sitting at a computer, end to end", then I'd say comparing yourself to it on a specific skill is the wrong test. Can you hand it a role and walk away for a day/week/month? I can’t yet. I think that I'd want at least two things it doesn't have: the ability to retain what it learned yesterday (without me carrying it in the context window and thus micromanaging it), and t…

I agree, we're not at that kind of long horizon capability yet. You still need a human in the loop to do manual testing. For whatever reason, we managed to automate the skill before we managed to automate the focus.

So, for now, humans need to stay in the loop and do low skilled labor to keep the skilled work the models do from going off the rails.

Re: GPT-6 Astra

#756

Post by ssl-3 »

echoangle wrote: post #548
manlymuppet wrote: post #406

I have nothing to say about the actual model, but unrelated--why do so many of these demos include people buying things autonomously? Even if I did trust an AI to get everything right, it's not like the AI can read my mind. If I was ordering food normally and without AI, I would want more control over the process--looking over the options, prices, thinking about what I really want. People don't know what they really…

That’s exactly the problem I have with all this agent ideas too. Imagine you had a human concierge that is just waiting for your instructions and is as smart or a bit smarter than you. Would you just tell them “plan this holiday for me” or “order this food”? I don’t even trust my friends to get this right, why would I give this to someone else?

As a realistic construct:

When I tell a bot to find the best value per volume for a reasonable quantity of unscented Dawn dish soap [so I can buy that], then: It often makes a complete mess of this seemingly-simple operation.

(And yeah, that is an actual thing that I've tried to accomplish with voice commands while standing in my kitchen and doing some dishes. It seems very simple, and it did not go well.

Maybe when we get the basics figured out we can start worrying about how inept it is at doing vacation planning.

It seems that this kind of thing isn't sorted at all, and that this is a very real problem for those who are in the bot business: These missed opportunities leave money on the table.)

Re: GPT-6 Astra

#757

Post by m11a »

sensanaty wrote: post #560
manlymuppet wrote: post #406

I have nothing to say about the actual model, but unrelated--why do so many of these demos include people buying things autonomously? Even if I did trust an AI to get everything right, it's not like the AI can read my mind. If I was ordering food normally and without AI, I would want more control over the process--looking over the options, prices, thinking about what I really want. People don't know what they really…

Despite access to """"""AGI""""""" all the marketing teams at these companies can only dream up 2 things, buying plane tickets and online shopping autonomously. Sometimes they're feeling extra spicy and throw in sorting emails or something along those lines. I suspect it's because it's tailored towards VCs and other similar rich ghouls as a replacement for their overworked and underpaid secretaries

Many VCs also dislike these examples, I believe. I'm doubtful this is what they're being pitched.

As for public releases: I wonder if it's because these examples are easy to relate to. Many websites are just a long tail of industry or use-case specific stuff. What's valuable to me probably means nothing to you. This is unlikely to resonate with people-wit-large (and LLMs are marketed broadly) or requires the reader to think (and marketing that requires thinking is bad these days).

Second, it's arguably a good litmus test. If it still can't do the worn out examples of plane tickets and shopping, which would be a good assumption since we've been demo'd these use-cases for 2 years at this point, then ...

Re: GPT-6 Astra

#759

Post by naishoya »

Toutouxc wrote: post #745
intenex wrote: post #248

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…

> I am reasonably confident that there's essentially nothing that I am better than Fable at despite generally being substantively above average on human benchmarks I genuinely don’t understand how an adult can say this with a straight face. I can take any single of my hobbies, start a mildly advanced conversation with Fable about the hobby and, within 5-10 turns, get it to contradict itself about something fundamenta…

People are ultimately capable of self deception. Believing that one is 'generally ... substantively above average on human benchmarks' may be more indicative of the brittleness of the claimant's human benchmarks.

Your observations expose the brittleness of the benchmarks being used for Fable, where the 'reasonable confident' claimant is working in the problem space of those two benchmarks, and you're demonstrating the failure at depth of the very capability facade for the model.

Sure, both the model and the confident human get the first layer right in some benchmark, but at least the model, and probably the human as well reach a collapsing probability of accuracy quite quickly. They may be completely and unconditionally right that Fable is more intelligent than they are, but that has little relevance at intelligence in depth as measured against your hobbies and the risk of unsafe or false responses.

Being smarter than a select group of people under test conditions is not conclusive regarding AGI. Moving the goalposts to declare AGI via a shallow benchmark, when a model could be proven wrong by almost anyone with basic competency given a few turns of iterative depth is where the 'grift' resides.

We're so far from anything approaching actual AGI, and it is highly speculative to infer that the current approach to Machine Learning as applied in LLM is even on a path that leads to AGI. But sales and promotions teams gotta pose, and we can all hate both the game and the players.

Re: GPT-6 Astra

#760

Post by sbochins »

XCSme wrote: post #335

It's fun, but every new model release makes me even less interested to create cool stuff. Like, what's the point, if the next AI can do it in 5 seconds?

Then learn how to be creative and use the ai better?
Post reply on HN