Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-weight AI is having its Kubernetes moment

tobi.knaup.me

Post reply on HN
Print view View original

171–180 of 346 posts

Re: Open-weight AI is having its Kubernetes moment

#171

Post by satvikpendem »

ozgung wrote: post #126

Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive at…

It's simple, the US government will put any Chinese open model companies on the entity list which blocks any company which does business with the US from also doing business with the Chinese companies. This creates a chilling effect where even if it may be harder to tell, no US company will be able to provide or use any overt Chinese open model and won't even risk trying to go around as the punishments for trying to evade the ban are severe.

Re: Open-weight AI is having its Kubernetes moment

#172

Post by munchler »

what-is-water wrote: post #133
munchler wrote: post #95

Earlier quoted context omitted.

I appreciate the effort and I'm in your target audience, but that document didn't help me. It seems to dive into the details of installing and running k8s without saying much about the purpose. From my very ignorant standpoint, K8s seems to be about running a "cluster", but I don't know why I would want to do that.

Kubernetes orchestrates your container workloads over a cluster, which consists of virtual/bare metal machines(nodes). This means you can tell the kubernetes API "I want to run a container workload" and it will be started on one of the nodes that form the cluster, unlike e.g. Docker, where a docker daemon belongs to a specific node. If you remove the node your workload is running on from the cluster the workload will…

Thank you for this explanation. It makes sense, but I don't really understand why it has become so popular.

Professionally, my experience is that certain software components need to run together on an individual machine (e.g. database server, app server, web server), and then those machines need to be networked in a certain way (e.g. web server talks to app server, which talks to database server), so I really need to care about the architecture of individual machines. You can then scale this out horizontally (e.g. add another web server) or vertically (e.g. upgrade your database server).

I'm old, so maybe I'm out of date, but having a cluster of "compute" that I can run arbitrary workloads on sounds neat, but is a capability that I've never needed.

Re: Open-weight AI is having its Kubernetes moment

#173

Post by hluska »

PunchyHamster wrote: post #140
Aurornis wrote: post #111

Earlier quoted context omitted.

Most new markets are funded by initial investment capital. Early entrants operate at a loss as they grow. This isn’t as unusual as some people are trying to make it sound. This has been happening since the dawn of finance. I thought this would be less foreign to everyone since we just went through this whole conversation for a decade with Uber and Lyft. Their demise was predicted from the start from everyone who thou…

Arguably we'd be much better off if none of those would be subsidized by investments, at least not to the "run unprofitable for decade+" level. Because that just absolutely murders any competition that manages to not get that level of free money. You're not pouring money in to make it happen at all at that point, you are pouring money in so nobody else can get the part of the pie. Which is great for investors, bad fo…

What would be the alternative? You’ve got the government funds absolutely everybody on one end of the scale. Where do we find a reasonable alternative?

Re: Open-weight AI is having its Kubernetes moment

#174

Post by mft_ »

hedora wrote: post #135
amazingamazing wrote: post #44

Sadly until china scales production of hardware it really isn’t economical to run this stuff yourself. It is good it exists though to put pressure against the labs. Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over.

Pre bubble prices (~= “we stop building data centers with subsidized credit / circular loans / hidden debt”), a 128GB halo strix ran for $1400, and 200-ish watts. Four of those in a cluster will run a 1T parameter frontier model: https://www.amd.com/en/developer/resources/technical-article... At 7 months of claude code subscription per node, the cluster pays for itself in 28 months. On a 5 year (60 month) depreciatio…

I’m happy to be proven wrong, but the limited examples I’ve seen of clustered Strix Halos are quite slow running large models (ie models too large to fit into the ram of a single machine) due to the slow networking between each one?

Re: Open-weight AI is having its Kubernetes moment

#175

Post by applicative »

cheriot wrote: post #73

Open-weight and OSS are wildly different and the article makes a poor comparison. What's the incentive for the Chinese labs to continue releasing weights 5 years from now? It's not a stable equilibrium and cannot last. - The lab spending large sums on research and training does not get the inference revenue to fund those efforts. - Unlike OSS where a single volunteer can keep a project going, training costs run into…

China has no end of money to support these companies. The reason this equilibrium is unstable is that the autonomous agentic coding aspect of the models has been so successfully improved that it will soon be a threat to China state security.

Re: Open-weight AI is having its Kubernetes moment

#176

Post by holoduke »

rullopat wrote: post #149
ozgung wrote: post #126

Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive at…

You need to ask what happened in Tienanmen square

All censorship whether it's tienanmen, sex nudity, violence etc are happening in a very thin layer of the model. It's easily bypassed. I use all open models without any censorship. I can easily ask how to make a nuclear bomb or how to build my own meth lab at home with a super details step by step list.

Re: Open-weight AI is having its Kubernetes moment

#178

Post by nyrikki »

Foobar8568 wrote: post #94
nyrikki wrote: post #50

Earlier quoted context omitted.

I don’t know if others would find this useful, but previous did have custom harnesses etc.. but tools have improved so much that I drastically simplified. That said, even the foundational models fail at the hard parts of my code so I use it opportunistically. I have reduced down to just using zed, will three locally hosted models. Qwen 3.6 27b on 1x3090 llama.cpp with 128k context ~50tps Qwen 3.6 35B-A3B on 1x titan…

You wouldn't get 100 tps on a Qwen 3.6 35b with a 5060 (or two) when a 5090 can barely reach that.

Depends on quant size etc... Qwen3.6 35B-A3B Q4_K_XL a multiple 5060ti + tensor split mode + MPT will hit ~100/TPS without problem, and I have personally hit ~190/tps on a single 5090 on a friends machine getting them setup up. If you use Q6_K etc... it slows down, what quant were you using?

Quantization + KV cache paging + speculative decoding (MPT or draft) is a fairly good mixture here.

Some examples as I don't have access to run tests on a 5060ti right now:

     https://njannasch.dev/blog/gemma-4-mtp-vs-qwen-speculative-decoding-5060ti/#vs-qwen-36-mtp

     https://www.reddit.com/r/LocalLLM/comments/1umw7vj/dual_5060_ti_16_gb_llm_inference_performance/
And here are some logs on unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL with the 2x 1080ti + 1x titan from above:

     27.21.533.298 I slot print_timing: id  0 | task 7843 | n_decoded =   1780, tg =  62.15 t/s
     27.24.537.173 I slot print_timing: id  0 | task 7843 | n_decoded =   1965, tg =  62.10 t/s
     27.27.541.142 I slot print_timing: id  0 | task 7843 | n_decoded =   2152, tg =  62.11 t/s
Q4_K_XL is a slight, acceptable degradation IMHO for performance like that.

Re: Open-weight AI is having its Kubernetes moment

#179

Post by hluska »

segmondy wrote: post #124
firasd wrote: post #2

One of the strangest things in the AI industry is 'tokenomics'. It's not very clear why using GPT-4 in early 2023 was so expensive and then six months later 20 bucks could get you a fair amount of GPT-4 inference. This pattern has continued across various labs/providers for years--there is a continuous see-saw of pricing that doesn't seem related to anything. So what open weight models do is at least provide a baseli…

What is strange about GPT-4 being expensive in 2023? Supply and demand. Which other model choices did we have? Prices are related to supply and demand. We see it play out with the introduction of capable open weight models or even other closed cloud models.

That’s a slightly naive view on pricing. That equilibrium point doesn’t just magically appear - it’s found through price testing.

Re: Open-weight AI is having its Kubernetes moment

#180

Post by hluska »

firasd wrote: post #141
segmondy wrote: post #124

Earlier quoted context omitted.

What is strange about GPT-4 being expensive in 2023? Supply and demand. Which other model choices did we have? Prices are related to supply and demand. We see it play out with the introduction of capable open weight models or even other closed cloud models.

Not really though right? As of early June 2026, Opus 4.8 in fast mode cost $50/M output tokens and Opus 4.6 & 4.7 cost $150/M output tokens in fast mode How can supply and demand explain the price drop? Was it cheaper to serve Opus 4.8? Is the demand for the newer Opus lower than for the older Opus? These are just fixed prices that seem picked out of thin air

They would have been picked out of thin air. That’s the joy of innovation - you have to randomly throw prices against the wall and see what sticks. The point where it sticks might be equilibrium or it may be an inefficient market… and nobody will know which one until it’s too late.
Post reply on HN