GPT-5.4 • Hacker News Board

Re: GPT-5.4

#591

Post by cperciva » Fri, Mar 06, 2026, 1:40 AM UTC

FrankBooth wrote: post #305

sillysaurusx wrote: post #251

Earlier quoted context omitted.

You may want to look over this thread from cperciva: https://x.com/cperciva/status/2029645027358495156 I too tried Codex and found it similarly hard to control over long contexts. It ended up coding an app that spit out millions of tiny files which were technically smaller than the original files it was supposed to optimize, except due to there being millions of them, actual hard drive usage was 18x larger. It seemed…

What’s the connection with context size in that thread? It seems more like an instruction following problem.

Yeah, I would definitely characterize it as an instruction following problem. After a few more round trips I got it to admit that "my earlier passes leaned heavily on build/tests + targeted reads, which can miss many “deep” bugs that only show up under specific conditions or with careful semantic review" and then asking it to "Please do a careful semantic review of files, one by one." started it on actually reviewing code.

Mind you, the bugs it reported were mostly bogus. But at least I was eventually able to convince it to try.

Re: GPT-5.4

#592

Post by varenc » Fri, Mar 06, 2026, 1:47 AM UTC

ozgung wrote: post #274

Chance-Device wrote: post #13

I’m sure the military and security services will enjoy it.

Did they publish its scores on military benchmarks, like on ArtificialSuperSoldier or Humanity's Last War?

I was pretty bummed to discover these aren't real benchmarks.

Re: GPT-5.4

#593

Post by XCSme » Fri, Mar 06, 2026, 1:51 AM UTC

QRe wrote: post #588

XCSme wrote: post #579

Looking ok, but nothing special: https://aibenchy.com/model/openai-gpt-5-4-medium/

Does this LLM benchmark have any actual credibility? I get why they chose to not publish the actual tests but I find it highly dubious that there are only 15 tests and Gemini 3 Flash performs best.

I actually made it, so I'm not sure if it has credibility, but the tests are simply various (quite simple) questions, and models are just tested on it. I am also surprised Gemini 3 Flash does so well (note that only the MEDIUM reasoning does exceptionally well).

When I look at the results, it does make sense though. Higher models (like Gemini 3 pro) tend to overthink, doubt themselves and go with the wrong solution.

Claude usually fails in subtle ways, sometimes due to formatting or not respecting certain instructions.

From the Chinese models, Qwen 3.5 Plus (Qwen3.5-397B-A17B) does extremely well, and I actually started using it on a AI system for one of my clients, and today they sent me an email they were impressed with one response the AI gave to a customer, so it does translate in real-world usage.

I am not testing any specific thing, the categories there are just as a hint as what the tests are about.

I just added this page to maybe provide a bit more transparency, without divulging the tests: https://aibenchy.com/methodology/

Re: GPT-5.4

#594

Post by triage8004 » Fri, Mar 06, 2026, 1:52 AM UTC

gizmodo59 wrote: post #538

baxtr wrote: post #478

Earlier quoted context omitted.

I picked up Claude today after being away and using only ChatGPT and Gemini for a while. I was pretty impressed with how they’ve improved user experience. If I had to guess, I’d say Anthropic has better product people who put more attention to detail in these areas.

ChatGPT has given more for my 20$ than any other vendor. And that’s not even considering codex which is so good and the limits are much much higher

They are all losing money on probably all levels of the packages if you max them out

Re: GPT-5.4

#595

Post by CamperBob2 » Fri, Mar 06, 2026, 1:59 AM UTC

rurban wrote: post #465

delaminator wrote: post #342

Earlier quoted context omitted.

two great problems in computing naming things cache invalidation off by one errors

Biggest problem right now in computing: Out of tokens until end of month

More like, "Out of DRAM until end of world"

Re: GPT-5.4

#596

Post by Nition » Fri, Mar 06, 2026, 2:01 AM UTC

0123456789ABCDE wrote: post #301

skrebbel wrote: post #244

Earlier quoted context omitted.

What the hell is a "safety score for violence"?

read here: https://deploymentsafety.openai.com/gpt-5-4-thinking/disallo...

I was sure the parent comment was a joke about OpenAI's recent deal with the DoD. But no, there it is, disallowing violence down from 90.9% of the time to 83.1%.

Re: GPT-5.4

#597

Post by beambot » Fri, Mar 06, 2026, 2:08 AM UTC

Philip-J-Fry wrote: post #374

I find it quite funny how this blog post has a big "Ask ChatGPT" box at the bottom. So you might think you could ask a question about the contents of the blog post, so you type the text "summarise this blog post". And it opens a new chat window with the link to the blog post followed by "summarise this blog post". Only to be told "I can't access external URLs directly, but if you can paste the relevant text or descri…

It's like opening copilot in a word doc and it telling you it can't see the document in its context

Re: GPT-5.4

#598

Post by le-mark » Fri, Mar 06, 2026, 2:12 AM UTC

lubesGordi wrote: post #421

tedsanders wrote: post #59

Earlier quoted context omitted.

Yeah, long context vs compaction is always an interesting tradeoff. More information isn't always better for LLMs, as each token adds distraction, cost, and latency. There's no single optimum for all use cases. For Codex, we're making 1M context experimentally available, but we're not making it the default experience for everyone, as from our testing we think that shorter context plus compaction works best for most p…

It's funny that the context window size is such a thing still. Like the whole LLM 'thing' is compression. Why can't we figure out some equally brilliant way of handling context besides just storing text somewhere and feeding it to the llm? RAG is the best attempt so far. We need something like a dynamic in flight llm/data structure being generated from the context that the agent can query as it goes.

That’s actually a pretty cool idea. When I think about my internal mental model of a codebase I’m working on it’s definitely a compacted lossy thing that evolves as I learn more.

Re: GPT-5.4

#599

Post by vostrocity » Fri, Mar 06, 2026, 2:22 AM UTC

abustamam wrote: post #523

baxtr wrote: post #478

Earlier quoted context omitted.

I picked up Claude today after being away and using only ChatGPT and Gemini for a while. I was pretty impressed with how they’ve improved user experience. If I had to guess, I’d say Anthropic has better product people who put more attention to detail in these areas.

I agree! I recently migrated from ChatGPT to Claude and it is just superior in every way. It doesn't blather on the at the end ask me for clarification. It's succinct and clarifies vital information before providing a solution.

Voice input is still far less accurate than OpenAI's unfortunately, otherwise I would have already switched.

Re: GPT-5.4

#600

Post by reval » Fri, Mar 06, 2026, 2:23 AM UTC

beambot wrote: post #597

Philip-J-Fry wrote: post #374

I find it quite funny how this blog post has a big "Ask ChatGPT" box at the bottom. So you might think you could ask a question about the contents of the blog post, so you type the text "summarise this blog post". And it opens a new chat window with the link to the blog post followed by "summarise this blog post". Only to be told "I can't access external URLs directly, but if you can paste the relevant text or descri…

It's like opening copilot in a word doc and it telling you it can't see the document in its context

This is infuriating. However, for those in this situation, know this: it works if the document or spreadsheet is in OneDrive. I just wish Copilot told you this instead of asking you to upload the doc.