261–270 of 588 posts
This is historic. For the first time, an open-weights LLM is right at the top. We won't be able to run this ourselves, but many providers can.
> For the first time, an open-weights LLM is right at the top. Hmm, not quite true, I think that honor, for better or worse, goes to OpenAI. When they released GPT2 (or GPT1 for that matter) is was quite literally the SOTA in the ecosystem when it was released.
Wait, so I can download it and run it locally now?? Wow... But it probably won't work on my computer, right?
If you have really high end hardware, you might be able to squeeze a heavily quantized version of Kimi-K3 onto your rig, but it will be too slow or too lobotomized to be useful.
This does put a near state-of-the-art open weights model within reach of what a small or medium business could afford if there's a case for local inference. It's probably not as good as Claude Fable or ChatGPT Sol. But if you're an organization that has a genuine need to run inference locally, this is a real possibility.
Is this for your homelab? Not in any practical sense.
Is this a possibility for organizations that can justify $1M or so on hardware for a near SOTA model they have full control over? Yeah, absolutely.
Earlier quoted context omitted.
A) is very doable with e.g. Amazon Bedrock. They'll give you HIPAA compliance, they even have a data center for US government classified data, they can give you European data sovereignty. And with OpenAI and Anthropic models to boot, you don't even have to settle for open weights. What kind of privacy needs do you really have beyond that?
It is not my use case but given recent political developments in international relations caused by the executive branch of the US government, off the top of my head, I could think of a lot of European or Canadian firms for which that would not be an option. No matter what they might promise about European sovereignty. For a good 'ol patriotic US domestic company? Sure.
Even for EU companies running open weights on EU stacks LLM inference on the GPU must process plaintext and I can't find any EU provider with NVIDIA H100/H200/Blackwell CC mode plus SEV-SNP or TDX, where you can cryptographically verify the workload ran somewhere the operator cannot inspect.
Personal compute is therefore the only option if you want personal autonomy privacy for IP &c. Maybe another option is to use cloud compute rented to fine tune a personal model that suits your own needs that would help bring the cost down, I don't know enough about this area to know if it kills the "intelligence" of those domains due to limited ?cross-verification within the LLM.
This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…
You're assuming inference providers are going to sell tokens at cost. You're also assuming that the inference providers have will optimized inference engine. I haven't seen that to be the case so far, to be honest. Take a look at GLM 5 vs GLM 5.2 pricing -- GLM 5.2 cost more despite being the same model. Take a look a look at DeepSeek, which hosts DS v4, profitably, yet others aren't able or willing to match the pric…
But I do think that the median price where this settles will tell us something about the floor at which it is profitable to serve this model.
> DeepSeek, which hosts DS v4, profitably
I specifically mentioned 3rd party providers, because there can be an argument that model creators themselves are subsidising tokens to gather training data for the next model. In fact, ds are public about their gathering of data (at least on openrouter they're marked as such). So that 0.x price point for dsv4-pro is likely subsidised.
Earlier quoted context omitted.
At this point in time what does meta get from releasing open weight models? Why devote the resources to it.
You can make the same argument for closed models. Why spend hundreds of billions training larger and larger models when you can just use Chinese models? Spend that money somewhere else further up the stack where there’s more value. Let China do the training since they’re so efficient at it.
Wait, so I can download it and run it locally now?? Wow... But it probably won't work on my computer, right?
The short answer is no, it won't work on your home computer. In it's current form it needs something like 594 GB of memory, far outside what you can reasonably run on normal consumer hardware in 2026. If you have really high end hardware, you might be able to squeeze a heavily quantized version of Kimi-K3 onto your rig, but it will be too slow or too lobotomized to be useful. This does put a near state-of-the-art ope…
This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…
I am curios what's the most profitable thing to "plant" (agriculture analogy) on the land (cards) that you have have: web hosting, vps, llms, image/video generation, etc
In my opinion, next step is to cut down on reasoning tokens while maintaining intelligence. The Chain of Thought and looping can still be an issue with these Chinese models. They in fact said K3 would improve in the area but it's still an issue that unfortunately harms the token cost wins a bit. OpenAI has been really impressive here, on the opposite end of this.
Earlier quoted context omitted.
There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socke…
Great, so the other member of the set matters for you more than cost. Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second?
Some people like doing things they want to do. Do I actually need to buy expensive pigments from europe to make paintings of flowers? My camera produces a much more accurate representation.