Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is minimal downside to switching to open models

marble.onl

Post reply on HN
Print view View original

61–70 of 351 posts

Re: There is minimal downside to switching to open models

#61

Post by Aurornis »

bnj wrote: post #41

I’ve been wanting to get better acquainted with local inference but I don’t have the hardware, which has made me think about something I haven’t seen discussed, which is local collaboratives. The economics makes it seem like a group of people joining together to run good hardware and an open model might make sense, but I haven’t seen anything like this mentioned. Have I been missing it? I think it would be pretty nea…

The reason you don't see more of this is because everyone does the math, realizes it's not a good deal, and then gives up on the idea.

There's a post at the top of /r/localllama about this exact math right now: https://www.reddit.com/r/LocalLLaMA/comments/1ubrcwj/tokenom...

TL;DR: Running GLM 5.2 is going to cost about $20K minimum, and that's going to be painfully slow compared to the cloud hosted versions. Even the estimates where the server is computing tokens 24/7 you can't break even for several years.

The only reason to run locally is if complete data privacy is your top concern. You pay a high premium for that.

Re: There is minimal downside to switching to open models

#62

Post by marcus_holmes »

brucehoult wrote: post #38
marcus_holmes wrote: post #34

Earlier quoted context omitted.

Macs are expensive hardware, but I'm always seeing people running LLMs on them. Is anyone running on cheaper generic hardware and Linux?

A Mac is cheaper than a high end GPU with the same amount of RAM.

ah, right, so it's about Apple Silicon being fast enough to use instead of a GPU?

Re: There is minimal downside to switching to open models

#63

Post by awakeasleep »

matheusmoreira wrote: post #43
taytus wrote: post #40

Earlier quoted context omitted.

At current prices, and considering these OS Models' performance, investing in local inference sounds like a bad idea.

Current prices are insane but at this point I'm starting to feel like it's an existential issue. I'm not a US citizen. At any point the USA could come up with some arbitrary export controls. Not having a computer capable of running at least Qwen is starting to actually seem risky to me. At least it's going to be usable as a very high end gaming PC.

Why would you buy and build everything before the low probability catastrophe strikes, though? You don’t get any benefit from switching early and you pay a big opportunity cost.

Re: There is minimal downside to switching to open models

#65

Post by CamperBob2 »

Aurornis wrote: post #37
julianlam wrote: post #6

I think it's interesting that people write off open weight models because they're "a few months behind" proprietary models. I know LLMs move at the speed of light (especially these past few quarters), but if Opus and GPT "a few months ago" were really like open weight models, then there's really no reason to not switch, especially for those who were using these models a few months ago. Your codebase didn't change, so…

> I think it's interesting that people write off open weight models because they're "a few months behind" proprietary models I experiment a lot with the open models and I’m getting tired of this trope. I’m not yet convinced that even the best open weight models are equal to Opus from “a few months” ago. I know what the benchmarks say. I had higher hopes. My real experience just doesn’t match the benchmarks. I also do…

Have you found anything specific that the full-precision quant of GLM 5.2 can't do that Opus 4.8 can? I haven't, so far.

5.2 lives up to the hype. I don't find it to be the best at anything except coding. But for coding... yeah, it lives up to the hype. Not quite Opus 4.8-level, but I would feel comfortable comparing it to 4.5, at least if it had vision capabilities.

Re: There is minimal downside to switching to open models

#66

Post by inigyou »

matheusmoreira wrote: post #35
kgeist wrote: post #11

Earlier quoted context omitted.

Every new proprietary model is "groundbreaking" and "look, it just solved task X that no other model could solve," only to be referred to as "that crappy previous-generation model" a month later. So yeah, I'm totally fine using Kimi-2.7, GLM-5.2 or Deepseek-v4. I think we've already hit the ceiling and most improvements now seem to be from harness improvements and slightly better RL to improve reasoning/tool calling.

There's at least the possibility that they intentionally degrade the models as time passes. We can't really verify that we're getting what we're paying for all of the time. All the more reason to invest in local inference.

What if the new model is exactly as good as the last model on launch day but better than the last model was on the new model's launch day because it was degraded? Every single time?

Re: There is minimal downside to switching to open models

#67

Post by inigyou »

awakeasleep wrote: post #63
matheusmoreira wrote: post #43

Earlier quoted context omitted.

Current prices are insane but at this point I'm starting to feel like it's an existential issue. I'm not a US citizen. At any point the USA could come up with some arbitrary export controls. Not having a computer capable of running at least Qwen is starting to actually seem risky to me. At least it's going to be usable as a very high end gaming PC.

Why would you buy and build everything before the low probability catastrophe strikes, though? You don’t get any benefit from switching early and you pay a big opportunity cost.

because as soon as it strikes computer hardware will be completely unavailable to buy?

Re: There is minimal downside to switching to open models

#68

Post by newwttbreak »

wilj wrote: post #32
causality0 wrote: post #25

I know open models have gotten quite good in many tasks such as coding or composition, but are there any that can access the internet and retrieve data like ChatGPT, Claude, etc can? I do have to admit I have recently begun wishing I could pay five dollars a month for a "just answer the fucking question" plan that would give me results without the guardrails and without the constant simpering and ego-stroking. I keep…

> I know open models have gotten quite good in many tasks such as coding or composition, but are there any that can access the internet and retrieve data like ChatGPT, Claude, etc can? The things you describe are just tool calling, they're a feature of whatever harness you use. Use OpenCode, pi.dev, or maki.sh with any of the open models. > I do have to admit I have recently begun wishing I could pay five dollars a m…

What are good resources and forums where I can figure out these system prompts to bypass guardrails, atleast on agents?

Re: There is minimal downside to switching to open models

#69

Post by moomoo11 »

julianlam wrote: post #6

I think it's interesting that people write off open weight models because they're "a few months behind" proprietary models. I know LLMs move at the speed of light (especially these past few quarters), but if Opus and GPT "a few months ago" were really like open weight models, then there's really no reason to not switch, especially for those who were using these models a few months ago. Your codebase didn't change, so…

ok but your competition using the latest models has an advantage

not all of us are doing noob shit lol

Re: There is minimal downside to switching to open models

#70

Post by coffinbirth »

> Open models are served via various means, some by the companies that released them and some by third parties like OpenRouter. Unfortunately, both of these routes are dodgier in terms of privacy and data sharing, and I would not feel the same comfort sending API calls containing client or confidential data to them.

That's why I'm using eurouter.ai with the following routing rule for all my requests:

  {
    "model": "glm-5.2",
    "models": [
      "deepseek-v4-pro",
      "deepseek-v4-flash"
    ],
    "provider": {
      "allow_fallbacks": true,
      "data_collection": "deny",
      "data_residency": "EU",
      "max_retention_days": 0,
      "eu_owned": true
    }
  }
Sure, it's quite expensive, but at least on a legal side data privacy is ensured. I trust them more than e.g. Anthropic, OpenAI or OpenRouter.

Personally, I find it morally unacceptable to use U.S. AI tools, because I do not want to support them financially and thus support the crimes they are involved in[1].

[1]: https://news.ycombinator.com/item?id=48512339

Post reply on HN