Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.6

openai.com

Post reply on HN
Print view View original

601–610 of 1001 posts

Re: GPT-5.6

#601

Post by XCSme »

10xDev wrote: post #234
meetpateltech wrote: post #154

GPT-5.6 Sol sets a new SOTA on ARC-AGI-3: 7.8% Sol is the first verified frontier model to ever beat an ARC-AGI-3 game https://arcprize.org/results/openai-gpt-5-6

Seeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learning. We are probably going to need a lot more GPUs.

Not always, in some cases, changing to a higher reasoning makes the AI doubt itself too much, and skip over the correct answer by overcomplicating the problem and polluting the context.

It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.

Re: GPT-5.6

#602

Post by lsllc »

XCSme wrote: post #597

GPT-5.6 is a really good model, and quite cheap. I can finally replace GPT-5.3-Codex for my Tool Calling in n8n. Here's my benchmark results for GPT-5.6: https://aibenchy.com/?q=gpt-5.6 (the high reasoning variants are still running, uploading them soon too) EDIT: The high variants are there too, enjoy the hamsters[0]. [0]: https://aibenchy.com/showcase/?q=gpt-5.6

Interesting that Sol (low) did better than Sol (medium) in your benchmark (and is barely more expensive than Terra). I too have been using 5.3 codex as a cheap-but-good model and are switching to Terra (xhigh).

Re: GPT-5.6

#603

Post by daveguy »

tshaddox wrote: post #485
ravenstine wrote: post #94

Earlier quoted context omitted.

> Avoid generic brevity instructions That part is confusing because it's not like they provide an example of how default GPT-5.6 output compares with GPT-5.5 both with default output and prompted for brevity. Whenever I use such prompts, it's usually because I want the model to give me the gist in a few sentences. I'd be stunned if GPT-5.6 was that concise by default. I would think that could "break" a lot of things…

If you conceptualize this as “there is an appropriate amount of brevity for each situation” then it would be expected for a better model to use different amounts of brevity if it gets better at determining the appropriate amount. My view is that popular models by default output wildly excessive amounts of prose for nearly every use case, so if this changes in a new model that’s a pure win.

The models don't get better, except when a new one is released. Their performance depends solely on the model training before release and how well you curate the context you feed it. That's it. Contrary to popular belief these things are not intelligent.

Re: GPT-5.6

#604

Post by dawnerd »

minimaxir wrote: post #266
bigyabai wrote: post #177

Earlier quoted context omitted.

The naming convention is bizarre and doesn't really mean anything to normies. Trying to pick between "Sol" and "Terra" is like asking the average person if they want the Max or the Ultra chip.

What about Haiku, Sonnet, and Opus?

Also just as confusing. Shouldn’t take away from the point though. Both can be bad names.

Re: GPT-5.6

#605

Post by William_BB »

37374848 wrote: post #216
danielsamuels wrote: post #173

Earlier quoted context omitted.

It's worth noting that OpenAI recently came out saying, "We don't think SWEBench Pro is worth reporting any more" - https://openai.com/index/separating-signal-from-noise-coding...

[flagged]

People will downvote you because this comment is "not appropriate" for HN, but there were countless conversations on HN about how important these benchmarks are.

I am literally LOLing at HN right now

Re: GPT-5.6

#606

Post by cmrdporcupine »

alex0015 wrote: post #591
cmrdporcupine wrote: post #385

Earlier quoted context omitted.

I almost immediately ran into "This request requires additional safety checks, which can take extra time. Hang tight or retry with a faster model for a quicker response, though it may be less capable of handling complex requests." Which is something I've never seen with codex before, and I wasn't doing anything funky. Just writing CUDA kernels and benchmarks for them.

I was getting that regularly last week with regular 5.5 medium on the plus plan. I was doing benchmarking for a photo editor in Swift.

Interesting. Never seen it before. Now it's just constant and there's nothing sensitive about this work.

Re: GPT-5.6

#607

Post by XCSme »

XCSme wrote: post #597

GPT-5.6 is a really good model, and quite cheap. I can finally replace GPT-5.3-Codex for my Tool Calling in n8n. Here's my benchmark results for GPT-5.6: https://aibenchy.com/?q=gpt-5.6 (the high reasoning variants are still running, uploading them soon too) EDIT: The high variants are there too, enjoy the hamsters[0]. [0]: https://aibenchy.com/showcase/?q=gpt-5.6

Here's all 3 (medium), and GPT-5.5

It GPT-5.6 doesn't seem to be a lot smarter than 5.5, but it is faster, cheaper, more efficient and more consistent:

https://aibenchy.com/compare/openai-gpt-5-6-sol-medium/opena...

Re: GPT-5.6

#608

Post by anematode »

AgentMasterRace wrote: post #453
ddxv wrote: post #199

I'm disappointed these models continue to be closed source and so expensive. Open weight models being 10x or more cheaper is just so much more of an unlock than incremental gains for me.

because they're stealing from the frontier models. they're gaming the benchmarks. look how bad glm 5.2 is on cursors evals. gmhit garbage , but it gets glazed as God tier.

They're stealing, eh?

Re: GPT-5.6

#609

Post by XCSme »

lsllc wrote: post #602
XCSme wrote: post #597

GPT-5.6 is a really good model, and quite cheap. I can finally replace GPT-5.3-Codex for my Tool Calling in n8n. Here's my benchmark results for GPT-5.6: https://aibenchy.com/?q=gpt-5.6 (the high reasoning variants are still running, uploading them soon too) EDIT: The high variants are there too, enjoy the hamsters[0]. [0]: https://aibenchy.com/showcase/?q=gpt-5.6

Interesting that Sol (low) did better than Sol (medium) in your benchmark (and is barely more expensive than Terra). I too have been using 5.3 codex as a cheap-but-good model and are switching to Terra (xhigh).

Yeah, for some reason the (low) versions do really well, like they think directly of the solution instead of going around all the edge-cases and getting lost in one of them.

Re: GPT-5.6

#610

Post by XCSme »

lsllc wrote: post #602
XCSme wrote: post #597

GPT-5.6 is a really good model, and quite cheap. I can finally replace GPT-5.3-Codex for my Tool Calling in n8n. Here's my benchmark results for GPT-5.6: https://aibenchy.com/?q=gpt-5.6 (the high reasoning variants are still running, uploading them soon too) EDIT: The high variants are there too, enjoy the hamsters[0]. [0]: https://aibenchy.com/showcase/?q=gpt-5.6

Interesting that Sol (low) did better than Sol (medium) in your benchmark (and is barely more expensive than Terra). I too have been using 5.3 codex as a cheap-but-good model and are switching to Terra (xhigh).

Also for most, there doesn't seem to be a big difference between (medium) and (high).
Post reply on HN