Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.6

openai.com

Post reply on HN
Print view View original

971–980 of 1001 posts

Re: GPT-5.6

#971

Post by tedsanders »

alimhaq wrote: post #124
realty_geek wrote: post #3

Wow, the "Agents' Last Exam" graph looks unreal!

I mean the y axis is deceptive to make it seem like greater gains since it starts at 30%, when in reality the differences aren't great. Even worse, it's not a fair comparison: they purposefully just used "adaptive" instead of "max" for Fable. What about the graph looked so unreal to you?

> Even worse, it's not a fair comparison: they purposefully just used "adaptive" instead of "max" for Fable.

We agree models should be compared on a fair basis. Unfortunately, adaptive was the only publicly available number. Anthropic doesn't generally let us run their models for evals, so we rely on whatever Anthropic or third parties have published. In this case, the Agents' Last Exam leaderboard has Fable Adaptive, but not Fable Max.

https://agents-last-exam.org/leaderboard

Would have loved to publish a full curve for Fable if anyone makes the data available.

Although we do bias toward publishing evals where we're ahead, we have historically been unafraid to publish evals where we're behind (e.g., GDPval). The point is give people useful information to decide what's best, not to trick people.

Edit: Now I see there's a second entry with xhigh effort. Not sure if that was added or recently or we skipped it.

(I work at OpenAI.)

Re: GPT-5.6

#972

Post by douglee650 »

camillomiller wrote: post #899
wkjagt wrote: post #893

I just watched the video on their launch page and I am really not sure how I feel about it. On one side, it's cool that these people get to start businesses and stuff using ChatGPT (assuming these are true stories), but how much of the business is really them? And how much does this business rely on a chat bot always being present as a kind of know it all employee? Maybe I'm just being naive or old fashioned (haven't…

I would add to this that, to me and to many friends in their 30-40s, using AI models to achieve something we used our brains to achieve feels... empty? wrong? soulless? Sure, a lot of menial work can be relegated to the models, and it's ok, most of the time, but you finish a day of work with the inability to shake a precise new feeling: that you haven't really achieved something, even if you shipped more than on an a…

Not manually cleansing data and building apps with AI to do the parts you can't or don't want to do is a massive lift.

I can do things that were far beyond my reach — suddenly. Almost all at once.

"Now you can discover you suck at marketing and business" :D

Re: GPT-5.6

#973

Post by hdemirev »

Have been testing Luna against other small models for production customer support uses cases, finding negligible performance impact when comparing against the substantial cost increase. Claims re: reduced token use also don't seem reproducible. Wrote a quick blog with some sample findings (marketing content but the findings are real): https://valiopt.com/blog/gpt-5-6-customer-support-cost-perfo...

Re: GPT-5.6

#974

Post by matheusmoreira »

insanitybit wrote: post #721
inciampati wrote: post #118

Earlier quoted context omitted.

This is a major reason why I and a number of biologists I've talked to have canceled their anthropic accounts recently. Not working is not working.

I'm writing a programming language with a "capability security model". That's enough to trigger Fable, it won't work on the language. It's hilarious. The mere presence of the word "security" seems to be enough to trip it up.

Anthropic refuses to allow Fable to code review my interpreter's memory safety. It was funny at first, then it became disappointing, then insulting, and finally utterly infuriating because I remembered the fact I'm actually paying for this nonsense.

Cancelled my subscription today. Hope OpenAI isn't patronizing like Anthropic. I don't want to hear about their "safety" bullshit ever again.

Re: GPT-5.6

#975

Post by tandr »

arikrahman wrote: post #28
Syntaf wrote: post #6

Ok long time Claude Code user here; lately I've started to realize there's other great models out there I should be trying, but I'm hesitant to leave Claude Code behind for something new. What's the consensus today on codex vs claude code, does it really matter anymore?

For me the biggest shift was using Deepseek through an American provider with reasonix as the harness, making cache hits at a rate of practically free.

Which provider do you use, if you don't mind sharing? I tried Digital Ocean (looking for Zero retention and no training), but their context limits are rather small for DeepSeek inference.

Re: GPT-5.6

#976

Post by tedsanders »

jiggawatts wrote: post #179
tedsanders wrote: post #93

Earlier quoted context omitted.

As usual, even though GPT-5.6 is releasing today, the rollout in ChatGPT and Codex will be gradual over many hours so that we can make sure service remains stable for everyone (same as our previous launches). We usually start with Pro/Enterprise accounts and then work our way down to Plus. We know it's slightly annoying to have to wait a random amount of time, but we do it this way to keep service maximally stable. T…

Is this bug fixed with 5.6? If not, it probably doesn’t matter which version Codex users are getting because the overall result is dramatically worse than stated by Open AI advertising: https://github.com/openai/codex/issues/30364

Not entirely fixed yet, but should be rarer with 5.6. Don’t have a quantification, unfortunately.

Re: GPT-5.6

#977

Post by gertlabs »

CuriouslyC wrote: post #883
gertlabs wrote: post #784

Earlier quoted context omitted.

We have it slightly ahead of Fable in our multi-agent coding evaluations. Fable's main advantage is that its average solution size is smaller. However, GPT 5.6 Sol is a substantial improvement from GPT 5.4/5.5 which would write verbose, defensive code. 31KB for GPT 5.4/5.5 down to 26KB for GPT 5.6 Sol, with better performance for Sol. Fable scores slightly lower, but with an average solution size of 12.2 KB. Data at…

Seems quite kind to Gemini models.

Gemini models struggle with agentic coding/tool use/exploration, but they are actually quite smart in one-shot reasoning. They're not as far behind as people think. It's mostly post-training and productization issues, which are easier to fix than pre-training/mid-training issues.

Agentic coding data: https://gertlabs.com/rankings?mode=agentic_coding

Re: GPT-5.6

#978

Post by gertlabs »

dgfl wrote: post #862
gertlabs wrote: post #784

Earlier quoted context omitted.

We have it slightly ahead of Fable in our multi-agent coding evaluations. Fable's main advantage is that its average solution size is smaller. However, GPT 5.6 Sol is a substantial improvement from GPT 5.4/5.5 which would write verbose, defensive code. 31KB for GPT 5.4/5.5 down to 26KB for GPT 5.6 Sol, with better performance for Sol. Fable scores slightly lower, but with an average solution size of 12.2 KB. Data at…

This looks like a good benchmark. Time and time again I keep giving OpenAI models the chance to win me back, but Opus (and Fable especially) just writes more elegant code and is a significantly more productive rubber duck for interactive discussions. I feel vindicated seeing your description of verbose and defensive code, and I’m a bit disappointed that 5.6 Sol’s solution is still >5x longer than the human solution a…

The human solutions are all written in Python, which creates a significant length bias, whereas the AI models are assigned to create solutions randomly distributed across 11 relevant programming languages, most of which are inherently more verbose than Python.

I have not broken down the comment/code ratio, but that's actually a really interesting idea for a metric.

I would also like to test Cursor, but our policy is to only test models available on public routers for now.

Re: GPT-5.6

#979

Post by ryukoposting »

jitl wrote: post #955
ryukoposting wrote: post #954

Earlier quoted context omitted.

> Avoid generic brevity instructions y'know, I don't think I will. I really, truly want one-word answers to any binary or multiple-choice question. If I want more, I will ask for it once the model has given its answer.

that is a specific brevity instruction!

Yeah, but good luck getting any model to obey it.

Re: GPT-5.6

#980

Post by insanitybit »

matheusmoreira wrote: post #974
insanitybit wrote: post #721

Earlier quoted context omitted.

I'm writing a programming language with a "capability security model". That's enough to trigger Fable, it won't work on the language. It's hilarious. The mere presence of the word "security" seems to be enough to trip it up.

Anthropic refuses to allow Fable to code review my interpreter's memory safety. It was funny at first, then it became disappointing, then insulting, and finally utterly infuriating because I remembered the fact I'm actually paying for this nonsense. Cancelled my subscription today. Hope OpenAI isn't patronizing like Anthropic. I don't want to hear about their "safety" bullshit ever again.

I've had zero issues with Codex. If it flags something it seems to have a slower "review before proceeding" phase but it does proceed.
Post reply on HN