Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The unbearable cheapness of open weight models

jamesoclaire.com

Post reply on HN
Print view View original

141–150 of 195 posts

Re: The unbearable cheapness of open weight models

#141

Post by hakfoo »

manwithopinions wrote: post #124
Tuna-Fish wrote: post #112

> What worries me about this is that Anthropic and OpenAI seem to have backed themselves into a corner of high costs. Can they reasonably decrease their prices by 20-50x to compete with DeepSeek or Xiaomi’s Mimo? They have high prices, not high costs. They will obviously keep prices as high as they can for as long as they can, while keeping demand up. Once demand starts to fall, so will the prices. > Are these models…

So why are they losing so much money? Money is made on the subset of inference that is charged at cost + margin via their APIs. API usage is so high because customers are still finding their feet, trying to understand how to measure the value they get from their spend, erring on the side of spend. Yes, in a world of unmeasured value and tokenmaxxing, inference is profitable on SOTA models because all capacity is bein…

> Outside of coding, almost every business case for AI doesn’t need above human intelligence, it doesn’t even need human intelligence, or half a human intelligence, a business can extract a lot of value from a machine that has a fraction of a human’s intelligence.

What the business world actually needed isn't intelligence, it's VBA with a bit of polish on it.

Yeah, people want tools to distill reports, and puff nonsense into bigger nonsense, but to a remarkable degree, this doesn't require an LLM. In fact, the alternatives might be preferrable by offering more consistency/repeatability and efficiency (I am so sick of watching a LLM babble for 5 minutes on something a regex would do in 5 seconds).

The genius of the LLM industry is that it avoids "programming anxiety" by hiding it behind a friendly-ish UI and not calling it programming. It's another in the string of innovations like "hide the file system" and "removing user programmability" so we can sell it back to you.

Re: The unbearable cheapness of open weight models

#142

Post by drillsteps5 »

dwaite wrote: post #93

Where's a few good places to go to learn more about open weight models, both running hosted and running locally?

Aside from googling "how to download and run open weights model" check out localllama (yes 3Ls) subreddit. Huggingface.co is where many of them are published.

There's many providers that run open weights models and give you access. Many decent open weights models cannot be run on consumer-grade hardware (DeepSeek, GLM, many others).

Re: The unbearable cheapness of open weight models

#143

Post by drillsteps5 »

Open weights models are cheap in the context of the article (when you run inference in the cloud) because they are free. When I pay for inference for running DeepSeek open weights model I only pay the inference service provider for compute/memory/storage/network throughput. The model itself is free, the developer isn't getting a dime.

Developing these things is NOT free, there's a lot of labor, hardware, compute/memory/storage/network that goes into that. Who's paying for all this? Chinese govt? Developers themselves? What's the revenue model here?

I absolutely LOVE ability to either run them locally or access inference providers on the cheap, but having a hard time understanding the financial side of this.

Re: The unbearable cheapness of open weight models

#144

Post by drillsteps5 »

MoonWalk wrote: post #115

I'd appreciate an explanation of what "open weight model" means. Is it a "weight model" that is open, or a model with open weights (so should be "open-weight model"), or is it weights that can be applied to a model? Are weights separable from a model? And if not, what is the point of saying "open-weight model" instead of just "open model?" To the newcomer, it's hard to determine what the components of an AI system ar…

As another commenter said a "model" is a file (or group of files, there's multiple formats available; GGUF format is all in one file for example). You download it to the hardware of your choice (ie your own desktop with NVIDIA GPU). You run the inference engine (llama-cpp, ollama,lm studio etc) and tell it where the downloaded model is and it runs inference (so you can start chatting with it, or run agents).

"Open weights model" means the developer made the model available for everyone for free. You can download it from huggingface.co for example and do whatever you want with it.

Why "open weights" and not "open source"? Because the "source code" for LLM would include things like training data, training methodologies and tools, so that you can do the training and produce the model (files) yourself. That would be like compiling from source code. Which is not done with these models, it's company's know-how, they only share the end result.

It's more analogous to "freeware" which is what we traditionally call freely distributed binary executable files. But people started calling them "open weights" instead and the term stuck.

Re: The unbearable cheapness of open weight models

#145

Post by vrighter »

Tuna-Fish wrote: post #134
manwithopinions wrote: post #124

Earlier quoted context omitted.

So why are they losing so much money? Money is made on the subset of inference that is charged at cost + margin via their APIs. API usage is so high because customers are still finding their feet, trying to understand how to measure the value they get from their spend, erring on the side of spend. Yes, in a world of unmeasured value and tokenmaxxing, inference is profitable on SOTA models because all capacity is bein…

> So why are they losing so much money? Mostly training. Claude didn't just get to be so good at coding by magic, it was suddenly so good because they did truly staggering amounts of RLHF and RLAIF on it. They are still doing that today, on any tasks they can figure out how to evaluate it on. This is capex for them. Their margins on inference are >90% today for tokens they sell (plans are hard to count, but still pro…

you forgot that to not have a knowledge cutoff and fall behing, you need to always be training new models. It matters jack shit if inference is cheap, if you are forced to do training anyway to stay "competitive"

Re: The unbearable cheapness of open weight models

#146

Post by mikae1 »

vrighter wrote: post #145
Tuna-Fish wrote: post #134

Earlier quoted context omitted.

> So why are they losing so much money? Mostly training. Claude didn't just get to be so good at coding by magic, it was suddenly so good because they did truly staggering amounts of RLHF and RLAIF on it. They are still doing that today, on any tasks they can figure out how to evaluate it on. This is capex for them. Their margins on inference are >90% today for tokens they sell (plans are hard to count, but still pro…

you forgot that to not have a knowledge cutoff and fall behing, you need to always be training new models. It matters jack shit if inference is cheap, if you are forced to do training anyway to stay "competitive"

Hos much does the cutoff matter when models can JFGI?

Re: The unbearable cheapness of open weight models

#147

Post by tliltocatl »

vrighter wrote: post #145
Tuna-Fish wrote: post #134

Earlier quoted context omitted.

> So why are they losing so much money? Mostly training. Claude didn't just get to be so good at coding by magic, it was suddenly so good because they did truly staggering amounts of RLHF and RLAIF on it. They are still doing that today, on any tasks they can figure out how to evaluate it on. This is capex for them. Their margins on inference are >90% today for tokens they sell (plans are hard to count, but still pro…

you forgot that to not have a knowledge cutoff and fall behing, you need to always be training new models. It matters jack shit if inference is cheap, if you are forced to do training anyway to stay "competitive"

Depends on the application, of course. For "google replacement" they are trying to sell it as - it's absolutely essential, but even then it works like crap. For coding… Maybe it's not so essential? Yea, it would not know about latest libraries, but maybe that's not much of a problem? Then, of course, it is a big question if LLM code generation is worth it at all.

Re: The unbearable cheapness of open weight models

#148

Post by overfeed »

mikae1 wrote: post #146
vrighter wrote: post #145

Earlier quoted context omitted.

you forgot that to not have a knowledge cutoff and fall behing, you need to always be training new models. It matters jack shit if inference is cheap, if you are forced to do training anyway to stay "competitive"

Hos much does the cutoff matter when models can JFGI?

The AI companies are attempting displace Google for that sweet ad revenue. Relying on another company's search index means they can squeeze you later for a bigger slice of the pie

Re: The unbearable cheapness of open weight models

#149

Post by dathanb82 »

Tuna-Fish wrote: post #135
uejfiweun wrote: post #133

Earlier quoted context omitted.

What are you even talking about? Everyone knows that Anthropic is drastically subsidizing their plans. It's actually the exact opposite of what you're talking about. The costs are extremely high and the prices are actually what's being subsidized and cheap right now.

This is an example of common knowledge that is wrong. People look at their cash burn, assume that they spend this to subsidize inference, and get bonkers answers. Inference is not their largest expense. Inference is cheap. Anthropic is only drastically subsidizing their plans if you count their training expenses as part of their costs.

If "inference is cheap," why is OpenAI spending a ton getting Broadcom to design custom AI chips that make inference cheaper? Reports suggest their custom silicon isn't all that good for training, it's all to make inference more efficient. That shouldn't be necessary if inference is already quite cheap.

Re: The unbearable cheapness of open weight models

#150

Post by xquce »

therobots927 wrote: post #140
Tuna-Fish wrote: post #135

Earlier quoted context omitted.

This is an example of common knowledge that is wrong. People look at their cash burn, assume that they spend this to subsidize inference, and get bonkers answers. Inference is not their largest expense. Inference is cheap. Anthropic is only drastically subsidizing their plans if you count their training expenses as part of their costs.

Are you an anthropic insider or something? Because if you are you should delete this comment. If you aren’t then you don’t know what the hell you’re talking about.

Surely the same can be said for the people saying the opposite?
Post reply on HN