Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is minimal downside to switching to open models

marble.onl

Post reply on HN
Print view View original

51–60 of 351 posts

Re: There is minimal downside to switching to open models

#51

Post by Barbing »

realusername wrote: post #39
kgeist wrote: post #11

Earlier quoted context omitted.

Every new proprietary model is "groundbreaking" and "look, it just solved task X that no other model could solve," only to be referred to as "that crappy previous-generation model" a month later. So yeah, I'm totally fine using Kimi-2.7, GLM-5.2 or Deepseek-v4. I think we've already hit the ceiling and most improvements now seem to be from harness improvements and slightly better RL to improve reasoning/tool calling.

There's also a lot of benchmark trickery going on, it's becoming harder to see how the latest models really improved. The top models also seem to have inconsistent performance depending on the time of day and how far we are from the next release.

Interesting, Claude might be doing better since I last checked:

https://marginlab.ai/trackers/claude-code-historical-perform...

There were at least a couple of these degradation trackers.

Re: There is minimal downside to switching to open models

#52

Post by [deleted] »

julianlam wrote: post #6

I think it's interesting that people write off open weight models because they're "a few months behind" proprietary models. I know LLMs move at the speed of light (especially these past few quarters), but if Opus and GPT "a few months ago" were really like open weight models, then there's really no reason to not switch, especially for those who were using these models a few months ago. Your codebase didn't change, so…

[deleted]

Re: There is minimal downside to switching to open models

#53

Post by sleepyeldrazi »

causality0 wrote: post #25

I know open models have gotten quite good in many tasks such as coding or composition, but are there any that can access the internet and retrieve data like ChatGPT, Claude, etc can? I do have to admit I have recently begun wishing I could pay five dollars a month for a "just answer the fucking question" plan that would give me results without the guardrails and without the constant simpering and ego-stroking. I keep…

That's why I like qwen3.6 27B, it has 0 ego, it knows that it doesn't have complete world knowledge, so when it sees a web_search tool it searches all the time. Even qwen3.5 9B is mostly search-eager (but given the size, it's weaker on reasoning on the results if that's needed). I use a stock pi harness with only web_search and web_fetch (cleans up the html to only keep text) tools defined.

I have given up on making Opus actually retrieve online information for me. At this point I only query it side by side with qwen to laugh at how it didn't even attempt to search properly, and how a small local model is beating it every time. Gemini is very fast for searching, but somehow miss-sources all the time.

Re: There is minimal downside to switching to open models

#54

Post by windexh8er »

nemomarx wrote: post #21
TacticalCoder wrote: post #9

Earlier quoted context omitted.

> I think it's interesting that people write off open weight models because they're "a few months behind" proprietary models. The really interesting thing is that it's typically those very same accounts who were explaining, a few months ago, that thanks to their commercial model they were gaining so much time and producing so much fantastic code. A few months passes and suddenly the open-source model have caught up w…

My most charitable interpretation that there's some honeymoon effect for each release, and people genuinely feel very productive and useful for 2-3 months. By the time the next big model release happens they've seen some issues or run into something that makes them feel like the new model will fix all that and improve their flow so much, etc. Not unusual in the tech space, but this has been basically constantly happe…

They are generally referred to as the Kool-Aid drinkers. There's always something holding them back from open models. It's no different than the argument in the article. I've been daily driving Linux for well over 20 years at this point and while things have gotten easier they haven't gotten that much easier. There's always been a distro that's focused on new users or ease of use. I used to take for granted the Linux distro ecosystem but now worry how Microsoft, Apple and others will continue to try and legislate compute into a corner. I can appreciate good engineering, but when I look at OS X and Windows they're both failing end users in different ways.

Just like the OS ecosystem I think we'll see a similar trajectory with OAI, Anthropic and Google but on a much accelerated time scale. I think the lobbying has begun to lock in their fate for revenue - because none of them give a shit about their users. I do hope, however, that Anthropic continues to over rotate and continue to gimp their models into uselessness. I just asked Opus 4.8 the other day to look at some code as an adversary and summarize areas that should be addressed. Nothing specific and it shut down the conversation. However starting a new prompt and prodding the model from a different angle yielded the results I asked for directly. Pick a lane. Or, don't and continue to lose industry respect and consideration.

Re: There is minimal downside to switching to open models

#56

Post by 827a »

julianlam wrote: post #6

I think it's interesting that people write off open weight models because they're "a few months behind" proprietary models. I know LLMs move at the speed of light (especially these past few quarters), but if Opus and GPT "a few months ago" were really like open weight models, then there's really no reason to not switch, especially for those who were using these models a few months ago. Your codebase didn't change, so…

Intelligence is maybe a few months behind. But cost sadly is further behind. GLM-5.2 has a deceptively high cost during day-to-day usage for e.g. coding because 1) it has to think a ton more than GPT-5.5/Opus-4.8 to get to competitive results; 2) many providers are still figuring out caching; and 3) API pricing for Codex/Claude can be as high as 40x more than subscription pricing, which distorts the market.

Re: There is minimal downside to switching to open models

#57

Post by tr_user »

causality0 wrote: post #25

I know open models have gotten quite good in many tasks such as coding or composition, but are there any that can access the internet and retrieve data like ChatGPT, Claude, etc can? I do have to admit I have recently begun wishing I could pay five dollars a month for a "just answer the fucking question" plan that would give me results without the guardrails and without the constant simpering and ego-stroking. I keep…

isn't that just in the harness?

Re: There is minimal downside to switching to open models

#59

Post by targafarian »

bonesss wrote: post #50
realusername wrote: post #39

Earlier quoted context omitted.

There's also a lot of benchmark trickery going on, it's becoming harder to see how the latest models really improved. The top models also seem to have inconsistent performance depending on the time of day and how far we are from the next release.

I’m an LLM fan, but from an engineering perspective the idea of building atop services that palpably fluctuate in capacity, performance, and capability is nutty . Even with minor automation I feel like I can watch OpenAI and Anthropic engineers fiddling in real-time. Tuesdays behaviour changes by Thursday, 10AMs production isn’t possible at 11:30AM. Nutty.

I chilled significantly on using Google for anything to do with business due to API (and offering) stability. (Still use Google for personal things.) But AI models seem orders of magnitude more fluid, so to my risk-averse eye, they're nothing I'd base my own business on.

Re: There is minimal downside to switching to open models

#60

Post by manyatoms »

matheusmoreira wrote: post #35
kgeist wrote: post #11

Earlier quoted context omitted.

Every new proprietary model is "groundbreaking" and "look, it just solved task X that no other model could solve," only to be referred to as "that crappy previous-generation model" a month later. So yeah, I'm totally fine using Kimi-2.7, GLM-5.2 or Deepseek-v4. I think we've already hit the ceiling and most improvements now seem to be from harness improvements and slightly better RL to improve reasoning/tool calling.

There's at least the possibility that they intentionally degrade the models as time passes. We can't really verify that we're getting what we're paying for all of the time. All the more reason to invest in local inference.

Unless what you're getting is really explicitly spelled out in a contract, you should flatly assume that they're doing whatever they like whenever they like.
Post reply on HN