Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is minimal downside to switching to open models

marble.onl

Post reply on HN
Print view View original

81–90 of 351 posts

Re: There is minimal downside to switching to open models

#81

Post by OtomotO »

awakeasleep wrote: post #63
matheusmoreira wrote: post #43

Earlier quoted context omitted.

Current prices are insane but at this point I'm starting to feel like it's an existential issue. I'm not a US citizen. At any point the USA could come up with some arbitrary export controls. Not having a computer capable of running at least Qwen is starting to actually seem risky to me. At least it's going to be usable as a very high end gaming PC.

Why would you buy and build everything before the low probability catastrophe strikes, though? You don’t get any benefit from switching early and you pay a big opportunity cost.

Because you will not be the only one struggling to get the hardware in the "unlikely" case the POTUS blurts out another fart.

Re: There is minimal downside to switching to open models

#82

Post by OtomotO »

Aurornis wrote: post #37
julianlam wrote: post #6

I think it's interesting that people write off open weight models because they're "a few months behind" proprietary models. I know LLMs move at the speed of light (especially these past few quarters), but if Opus and GPT "a few months ago" were really like open weight models, then there's really no reason to not switch, especially for those who were using these models a few months ago. Your codebase didn't change, so…

> I think it's interesting that people write off open weight models because they're "a few months behind" proprietary models I experiment a lot with the open models and I’m getting tired of this trope. I’m not yet convinced that even the best open weight models are equal to Opus from “a few months” ago. I know what the benchmarks say. I had higher hopes. My real experience just doesn’t match the benchmarks. I also do…

> My real experience just doesn’t match the benchmarks.

That's exactly the problem I have... with Anthropic and "Open""AI"

Re: There is minimal downside to switching to open models

#83

Post by OtomotO »

itake wrote: post #48
CJefferson wrote: post #29

Earlier quoted context omitted.

One big advantage I’ve found — people get attached to models (including me). With open models if you find one that works perfectly for you but the next version doesn’t, you can run the old one forever (or someone will for you)

But… the models will fall behind. As libraries and languages and tool calling updates or the world knowledge changes, the models decay. Personally, I don’t like the change, but it’s just how technology works so I’d rather move with the flow than try to stick my foot down and freeze time.

No problem, "AI" will just write its own frameworks and libs then!

Re: There is minimal downside to switching to open models

#84

Post by maybe_pablo »

LPisGood wrote: post #73
matheusmoreira wrote: post #35

Earlier quoted context omitted.

There's at least the possibility that they intentionally degrade the models as time passes. We can't really verify that we're getting what we're paying for all of the time. All the more reason to invest in local inference.

People talk about this a lot. What I have never seen is a discussion of methods they might employ to degrade the models. Let’s say I’m a bad faith LLM operator, and I want to degrade my model so the next release looks better and people want to switch to the more expensive one. How would I do that?

Weight quantization, n-expert capping, routing to smaller model, context window truncation, aggressive sampling constraints, lossy speculative decoding and probably more.

Re: There is minimal downside to switching to open models

#85

Post by Tepix »

LPisGood wrote: post #73
matheusmoreira wrote: post #35

Earlier quoted context omitted.

There's at least the possibility that they intentionally degrade the models as time passes. We can't really verify that we're getting what we're paying for all of the time. All the more reason to invest in local inference.

People talk about this a lot. What I have never seen is a discussion of methods they might employ to degrade the models. Let’s say I’m a bad faith LLM operator, and I want to degrade my model so the next release looks better and people want to switch to the more expensive one. How would I do that?

Use quantisation.

Re: There is minimal downside to switching to open models

#86

Post by simianwords »

coffinbirth wrote: post #70

> Open models are served via various means, some by the companies that released them and some by third parties like OpenRouter. Unfortunately, both of these routes are dodgier in terms of privacy and data sharing, and I would not feel the same comfort sending API calls containing client or confidential data to them. That's why I'm using eurouter.ai with the following routing rule for all my requests: { "model": "glm-…

GDPR compliant llm was a joke a few months back but here we are

Re: There is minimal downside to switching to open models

#87

Post by throwaway27448 »

coffinbirth wrote: post #70

> Open models are served via various means, some by the companies that released them and some by third parties like OpenRouter. Unfortunately, both of these routes are dodgier in terms of privacy and data sharing, and I would not feel the same comfort sending API calls containing client or confidential data to them. That's why I'm using eurouter.ai with the following routing rule for all my requests: { "model": "glm-…

Why use EU specifically? I get not trusting the US, of course, but surely the EU isn't far behind in its desire to spy on its own citizens. Do you not live there?

Re: There is minimal downside to switching to open models

#88

Post by nessex »

LPisGood wrote: post #73
matheusmoreira wrote: post #35

Earlier quoted context omitted.

There's at least the possibility that they intentionally degrade the models as time passes. We can't really verify that we're getting what we're paying for all of the time. All the more reason to invest in local inference.

People talk about this a lot. What I have never seen is a discussion of methods they might employ to degrade the models. Let’s say I’m a bad faith LLM operator, and I want to degrade my model so the next release looks better and people want to switch to the more expensive one. How would I do that?

They would quantize the model. That'd make it cheaper to run, and have slightly worse output but it would still generate outputs with a similar feel, derived from a compressed version of the same knowledge base etc.

They wouldn't even need to do this uniformly, quantized versions of the model could be routed only a subset of the requests. They could do this to nerf the old model, or more likely just to give themselves more hardware to run the new one on by handling more requests on less hardware. Or to handle increased request volume as traffic ramps up faster than hardware can be provisioned.

Playing with local models at various quants, the degradation can be hard to spot. Sometimes it's only noticeable in aggregate. And even then, you never really know if you just got unlucky with a bad response due to RNG.

I've had Opus 4.6 fall into some weirdly incoherent loops that I rarely see from even Sonnet, that felt like the kind of thing I got frequently with Qwen3.5 9B on local. And the above applies... Was that just bad RNG? Or was my request to Opus routed to some lower quality variant? There's no great way for me to tell for any given request, nor any way to guarantee Anthropic _didn't_ do that.

Re: There is minimal downside to switching to open models

#89

Post by OtomotO »

I am absolutely pro local and true open source models.

Personally I haven't seen any productivity gain since Opus 4.5 times.

But: I can't fully get behind the opinion that (so called) "open source models" are simply superior and will be in the future, because when I asked some models who they are, they answered with "I am Claude from Anthropic", which could mean they have been trained by exfiltrating Claude.

I have NO moral objection to this, as Anthropic and "Open""AI".also trained their models on anything they could get their hands on.

It's more about the question: can and will these models be updated, even if Anthropic et al fail. Who's gonna pay for training then? What's their incentive? Have we reached a plateau?

Re: There is minimal downside to switching to open models

#90

Post by earthnail »

throwaway27448 wrote: post #87
coffinbirth wrote: post #70

> Open models are served via various means, some by the companies that released them and some by third parties like OpenRouter. Unfortunately, both of these routes are dodgier in terms of privacy and data sharing, and I would not feel the same comfort sending API calls containing client or confidential data to them. That's why I'm using eurouter.ai with the following routing rule for all my requests: { "model": "glm-…

Why use EU specifically? I get not trusting the US, of course, but surely the EU isn't far behind in its desire to spy on its own citizens. Do you not live there?

From all the large governmental institutions, the EU is the one currently holding up traditional western values. That gives it street cred in this subject.
Post reply on HN