Developers are choosing older AI models • Hacker News Board

Re: Developers are choosing older AI models

#121

Post by infecto » Wed, Nov 05, 2025, 1:58 PM UTC

feintruled wrote: post #116

Some missing context (pun intended) is that Augment code has recently switched to a per-token instead of per-message pricing model. This hasn't gone down particularly well, but that's another story. But it may well be that users drop back to older models in the expectation it will use less tokens. Personally, I stopped using GPT-5 as it would just be tool call after tool call without ever stopping to tell you what th…

My take is the models matter but the tool call integration is the most important piece and differentiator.

Re: Developers are choosing older AI models

#122

Post by tifa2up » Wed, Nov 05, 2025, 1:59 PM UTC

internet_points wrote: post #50

tifa2up wrote: post #42

Earlier quoted context omitted.

For large context (up to 100K tokens in some cases). We found that GPT-5: a) has worse instruction following; doesn't follow the system prompt b) produces very long answers which resulted in a bad ux c) has 125K context window so extreme cases resulted in an error

Interesting. https://www.robert-glaser.de/prompts-as-programs-in-gpt-5/ claims GPT-5 has amazing!1!! instruction following. Is your use-case very different, or is this yet another case of "developer A got lucky, developer B tested more things"?

Think it varies by use case. It didn't do well with long context

Re: Developers are choosing older AI models

#123

Post by infecto » Wed, Nov 05, 2025, 2:00 PM UTC

s1mplicissimus wrote: post #23

nicce wrote: post #11

Earlier quoted context omitted.

Most people can’t affort the GPUs for local models if you want to get close to cloud capabilities.

Most people I know can't afford to leak business insider information to 3rd party SaaS providers, so it's unfortunately not really an option.

This is a poor take imo. Depends on the industry but the worlds businesses run on the shoulders of companies like Microsoft and heavily use OneDrive/Sharepoint. Most entities, even those with sensitive information are legally comfortable with that arrangement. Using a LLM does not change much so long as the MSA is similar.

Re: Developers are choosing older AI models

#124

Post by infecto » Wed, Nov 05, 2025, 2:01 PM UTC

Antibabelic wrote: post #75

ruszki wrote: post #63

Earlier quoted context omitted.

But… they do all the time. Almost everybody uses some mix of Office, Slack, Notion, random email providers, random “security” solutions etc. The exception is the opposite. The only thing prevents info leaking is ToS, and there are options for that even with LLMs. Nothing changed regarding that.

In my personal experience, it's very common for big companies to host email, messengers, conferencing software on their own servers.

In my experience it’s very common for big companies to not host. Think Fortune 500 type companies. Most are legally happy with their MSA and reasonably confident in security standards.

Re: Developers are choosing older AI models

#125

Post by mbesto » Wed, Nov 05, 2025, 2:08 PM UTC

tifa2up wrote: post #14

We tried GPT-5 for a RAG use case, and found that it performs worse than 4.1. We reverted and didn't look back.

How do you objectively tell whether a model "performs" better than another?

Re: Developers are choosing older AI models

#126

Post by belval » Wed, Nov 05, 2025, 2:13 PM UTC

mbesto wrote: post #125

tifa2up wrote: post #14

We tried GPT-5 for a RAG use case, and found that it performs worse than 4.1. We reverted and didn't look back.

How do you objectively tell whether a model "performs" better than another?

Not the original commenter but I work in the space and we have large annotated datasets with "gold" evidence that we want to retrieve, the evaluation of new models is actually very quantitative.

Re: Developers are choosing older AI models

#127

Post by infecto » Wed, Nov 05, 2025, 2:15 PM UTC

brazukadev wrote: post #66

iberator wrote: post #38

Earlier quoted context omitted.

That's out of touch for 90% of developers worldwide

Today. But what about in 5 years? Would you bet we will be paying hundreds of billions to OpenAI yearly or buying consumer GPUs? I know what I will be doing.

Paying for compute in the cloud. That’s what I am betting on. Multiple providers, different data center players. There may be healthy margins for them but I would bet it’s always going to be relatively cheaper for me to pay for the compute rather than manage it myself.

Re: Developers are choosing older AI models

#128

Post by Buttons840 » Wed, Nov 05, 2025, 2:20 PM UTC

This is how the bubble pops.

I've been thinking the AI bubble wouldn't pop, because even the AI advances we've already seen can change the majority of industries if it is carefully integrated with existing technology. But if there's a mass movement to use older and/or smaller models, then yeah, all the money going into newer bigger models will pop.

Or, maybe the training datasets getting polluted with AI slop will mean that new models are worse than old models. That would pop the industry.

Or, maybe the GPT-4 era was the golden era for AI, and making them bigger and better is just overfitting (in the classical machine learning sense of the word) and is both worse and more expensive. This would pop the industry too.

I guess there's a few ways for the industry to pop, but this trend of using older models makes me especially skeptical of AI.

Re: Developers are choosing older AI models

#129

Post by dhumph » Wed, Nov 05, 2025, 2:24 PM UTC

kristianp wrote: post #89

KronisLV wrote: post #3

For development use cases, I switched to Sonnet 4.5 and haven't looked back. I mean, sure, sometimes I also use GPT-5 (and mini) and Gemini 2.5 Pro (and Flash), and also Cerebras Code just switched to providing GLM 4.6 instead of the previous Qwen3 Coder so those as well, but in general the frontier models are pretty good for development and I wouldn't have much reason to use something like Sonnet 4 or 3.7 or whateve…

What tool are you using to enable switching between so many models?

Cline & Open router

Re: Developers are choosing older AI models

#130

Post by jennyholzer » Wed, Nov 05, 2025, 2:31 PM UTC

Buttons840 wrote: post #128

This is how the bubble pops. I've been thinking the AI bubble wouldn't pop, because even the AI advances we've already seen can change the majority of industries if it is carefully integrated with existing technology. But if there's a mass movement to use older and/or smaller models, then yeah, all the money going into newer bigger models will pop. Or, maybe the training datasets getting polluted with AI slop will me…

Since the day GPT-5 released, I've felt quite confident that the GPT-4 era was the golden era for AI.

I don't have evidence beyond my experience using the product, but based on that experience I believe that Open AI has been cooking their benchmarks since at least the release of GPT-5.