Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5

openai.com

Post reply on HN
Print view View original

711–720 of 1001 posts

Re: GPT-5.5

#711

Post by dcre »

gausswho wrote: post #209
h14h wrote: post #106

This seems huge for subscription customers. Looking at the Artificial Analysis numbers, 5.5 at medium effort yields roughly the intelligence as 5.4 (xhigh) while using less than a fifth the tokens. As long as tokens count roughly equally towards subscription plan usage between 5.5 & 5.4, you can look at this as effectively a 5x increase in usage limits.

As someone who always leaves intelligence at default, and am ok with existing models, should I be shifting gears more manually as providers sell us newer models? Is medium or lower better than free/cheaper models?

SOTA models on medium are probably still better than free or cheap models, but you should experiment.

Re: GPT-5.5

#712

Post by doug_durham »

ofjcihen wrote: post #470
sharts wrote: post #455

Earlier quoted context omitted.

One might argue that it’s not too too different from higher level abstractions when using libraries. You get things done faster, write less code, library handles some internal state/memory management for you. Would one be uneasy about calling a library to do stuff than manually messing around with pointers and malloc()? For some, yes. For others, it’s a bit freeing as you can do more high-level architecture without g…

I see this comparison made constantly and for me it misses the mark. When you use abstractions you are still deterministically creating something you understand in depth with individual pieces you understand. When you vibe something you understand only the prompt that started it and whether or not it spits out what you were expecting. Hence feeling lost when you suddenly lose access to frontier models and take a look…

In what world to developers “understand” pieces like React, Pandas, or Cuda? Developers only have a superficial understanding of the tools they are developing with.

Re: GPT-5.5

#713

Post by subhobroto »

mikestorrent wrote: post #537
noosphr wrote: post #528

Earlier quoted context omitted.

LLMs upend a few centuries of labor theory. The current market is predicated on the assumption that labor is atomic and has little bargaining power (minus unions). While capital has huge bargaining power and can effectively put whatever price it wants on labor (in markets where labor is plentiful, which is most of them). What happens to a company used to extracting surplus value from labor when the labor is provided…

I am still trying to figure out the business model of open weights. Like... it's wonderful that there are open LLMs, super happy about it, good for everyone, but why are there these? What is the advantage to their companies to release them?

> What is the advantage to their companies to release them?

It's a distribution strategy. It costs something to serve the models - let's say $5/1M tokens.

If Qwen required $5 from anyone who was curious so you could even begin to test it out, a lot of people just wouldn't.

Now Qwen could offer a "free" tier, but it's infinitely cheaper to provide the weights and let people run it themselves including opening up the ability for anyone else on the planet to test it against other (open weight) models.

The costs to build the open weight models are sunk, but the costs to serve them, get them tested are not.

It's also precisely why the .NET SDK is free or the ESP32 SDK is free - they sell more Microsoft or ESP32 products.

Re: GPT-5.5

#714

Post by tedsanders »

dakolli wrote: post #344
silvertaza wrote: post #267

Still huge hallucination rate, unfortunately at 86%. To compare, Opus sits at 36%. Source: https://artificialanalysis.ai/models?omniscience=omniscience...

This indicates they want this behavior, they know the person asking the question probably doesn't understand the problem entirely (or why would they be asking), so they'd prefer a confident response, regardless of outcomes, because the point is to sell the technologies competency (and the perception thereof), not the capabilities, to a bunch of people that have no clue what they're talking about. LLMs will ruin your…

We don't want hallucinations either, I promise you.

A few biased defenses:

- I'll note that this eval doesn't have web search enabled, but we train our models to use web search in ChatGPT, Codex, and our API. I'd be curious to see hallucination rates with web search on.

- This eval only measures binary attempted vs did not attempt, but doesn't really reward any sort of continuous hedging like "I think it's X, but to be honest I'm not sure."

- On the flip side, GPT-5.5 has the highest accuracy score.

- With any rate over 1% (whether 30% or 70%), you should be verifying anything important anyway.

- On our internal eval made from de-identified ChatGPT prompts that previously elicited hallucinations, we've actually been improving substantially from 5.2 to 5.4 to 5.5. So as always, progress depends on how you measure it.

- Models that ask more clarifying questions will do better on this eval, even if they are just as likely to hallucinate after the clarifying question.

Still, Anthropic has done a great job here and I hope we catch up to them on this eval in the future.

Re: GPT-5.5

#715

Post by calf »

dakolli wrote: post #344
silvertaza wrote: post #267

Still huge hallucination rate, unfortunately at 86%. To compare, Opus sits at 36%. Source: https://artificialanalysis.ai/models?omniscience=omniscience...

This indicates they want this behavior, they know the person asking the question probably doesn't understand the problem entirely (or why would they be asking), so they'd prefer a confident response, regardless of outcomes, because the point is to sell the technologies competency (and the perception thereof), not the capabilities, to a bunch of people that have no clue what they're talking about. LLMs will ruin your…

On ChatGPT 5.3 Plus subscription I find that long informal chats tend to reveal unsatisfactory answers and biases, at this point after 10 rounds of replies I end up having to correct it so much that it starts to agree with my initial arguments full circle. I don't see how this behavior is acceptable or safe for real work. Like are programmers and engineers using LLMs completely differently than I'm doing, because the underlying technology is fundamentally the same.

Re: GPT-5.5

#716

Post by NegativeLatency »

doug_durham wrote: post #712
ofjcihen wrote: post #470

Earlier quoted context omitted.

I see this comparison made constantly and for me it misses the mark. When you use abstractions you are still deterministically creating something you understand in depth with individual pieces you understand. When you vibe something you understand only the prompt that started it and whether or not it spits out what you were expecting. Hence feeling lost when you suddenly lose access to frontier models and take a look…

In what world to developers “understand” pieces like React, Pandas, or Cuda? Developers only have a superficial understanding of the tools they are developing with.

Some developers, I usually end up fixing bugs in OSS I use

Re: GPT-5.5

#717

Post by doug_durham »

noosphr wrote: post #532
sharts wrote: post #455

Earlier quoted context omitted.

One might argue that it’s not too too different from higher level abstractions when using libraries. You get things done faster, write less code, library handles some internal state/memory management for you. Would one be uneasy about calling a library to do stuff than manually messing around with pointers and malloc()? For some, yes. For others, it’s a bit freeing as you can do more high-level architecture without g…

A library is deterministic. LLMs are not. That we let a generation of software developers rot their brains on js frameworks is finally coming back to bite us. We can build infinite towers of abstraction on top of computers because they always give the same results. LLMs by comparison will always give different results. I've seen it first hand when a $50,000 LLM generated (but human guided) code base just stops workin…

Why would that necessarily happen? With an LLM you have perfect knowledge of the code. At any time you can understand any part of your code by simply asking the LLM to explain it. It is one of the super powers of the tools. They also accelerate debugging by allowing you to have comprehensive logging. With that logging the LLM can track down the source of problems. You should try it.

Re: GPT-5.5

#718

Post by drusepth »

_alternator_ wrote: post #330

> One engineer at NVIDIA who had early access to the model went as far as to say: "Losing access to GPT‑5.5 feels like I've had a limb amputated.” This quote is more sinister than I think was intended; it likely applies to all frontier coding models. As they get better, we quickly come to rely on them for coding. It's like playing a game on God Mode. Engineers become dependent; it's truly addictive. This matches my o…

> It's literally higher leverage for me to go for a walk if Claude goes down than to write code because if I come back refreshed and Claude is working an hour later then I'll make more progress than mentally wearing myself out reading a bunch of LLM generated code trying to figure out how to solve the problem manually.

Taking more breaks and "not working" during the work day sounds like something we should probably be striving to work towards more as a society.

Re: GPT-5.5

#719

Post by rattray »

simianwords wrote: post #329
silvertaza wrote: post #267

Still huge hallucination rate, unfortunately at 86%. To compare, Opus sits at 36%. Source: https://artificialanalysis.ai/models?omniscience=omniscience...

There's something off with this because Haiku should not be that good.

I've been very curious about that too. I wonder if it's actually much better at admitting when it doesn't know something, because it thinks it's a "dumber model". But I haven't played with this at all myself.

Re: GPT-5.5

#720

Post by doug_durham »

theappsecguy wrote: post #495
sharts wrote: post #455

Earlier quoted context omitted.

One might argue that it’s not too too different from higher level abstractions when using libraries. You get things done faster, write less code, library handles some internal state/memory management for you. Would one be uneasy about calling a library to do stuff than manually messing around with pointers and malloc()? For some, yes. For others, it’s a bit freeing as you can do more high-level architecture without g…

I would argue it couldn't be more different. I can dive into the source code of any library, inspect it. I can assess how reliable a library is and how popular. Bugs aside, libraries are deterministic. I don't see why this parallel keeps getting made over and over again.

I can dive into the source code of LLM generated code too. Indeed it is better because you have tools to document it better than a library that you use.
Post reply on HN