Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-weight AI is having its Kubernetes moment

tobi.knaup.me

Post reply on HN
Print view View original

331–340 of 346 posts

Re: Open-weight AI is having its Kubernetes moment

#331

Post by andriy_koval »

antonvs wrote: post #330
andriy_koval wrote: post #260

Earlier quoted context omitted.

It happened that I am working on OSS LLM -> finetuning -> benchmark with 100k tests pipeline, and unless I do some data augmentation, result is 100% deterministic. I think you likely right, that some parts of stack could induce some marginal float point error, but converged model can mitigate it, and on some principal set of knowledge can give deterministic result with high probability. Which leads me to believe if y…

If you're running on a single machine with a single GPU, then you may get deterministic results, although it still depends a lot on the details. For example if you're using Pytorch, you need to enable deterministic algorithms and may need to configure some other things as well. However, running in production at any sort of scale often involves multiple machines and multiple GPUs, and at that point, determinism can be…

No, I am not running on single GPU, but even then, nothing prevents to use single GPU for models verification.

Re: Open-weight AI is having its Kubernetes moment

#332

Post by tracerbulletx »

clhodapp wrote: post #196
andriy_koval wrote: post #192

Earlier quoted context omitted.

someone can run tests and see that models output exactly the same results, and then you are open to criminal investigation.

Models don't even agree with themselves in terms of returning identical results

Yes they do. Sampling is the only pseudo-random part. Models return a deterministic distribution of output tokens for a given input of tokens.

Re: Open-weight AI is having its Kubernetes moment

#333

Post by TGower »

nyrikki wrote: post #50
thih9 wrote: post #16

Is anyone using open weight models for agentic coding? What is your stack (harness, model) and how much do you pay per month? How would you compare your experience to a typical subsidized plan like Claude Code + Pro plan? I’m asking because i keep hearing that open weight models are cheap and efficient - is that really the case in practice?

I don’t know if others would find this useful, but previous did have custom harnesses etc.. but tools have improved so much that I drastically simplified. That said, even the foundational models fail at the hard parts of my code so I use it opportunistically. I have reduced down to just using zed, will three locally hosted models. Qwen 3.6 27b on 1x3090 llama.cpp with 128k context ~50tps Qwen 3.6 35B-A3B on 1x titan…

Are you saying that cloud models are not verifying the draft model predictions? The way draft models are used in something like llama.cpp results in exactly zero degredation of output quality, with the larger model verifying each draft model token and discarding it if it does not match.

Re: Open-weight AI is having its Kubernetes moment

#334

Post by WhiteOwlLion »

ozgung wrote: post #126

Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive at…

Just ask the model about Tianamen Square and you know if it’s a Chinese model or not.

Re: Open-weight AI is having its Kubernetes moment

#335

Post by [deleted] »

cheriot wrote: post #312
applicative wrote: post #175

Earlier quoted context omitted.

China has no end of money to support these companies. The reason this equilibrium is unstable is that the autonomous agentic coding aspect of the models has been so successfully improved that it will soon be a threat to China state security.

Like they did with solar panels, I suspect the goal is to dominate an industry and not need to subsidize it forever.

[deleted]

Re: Open-weight AI is having its Kubernetes moment

#336

Post by satvikpendem »

DoctorOetker wrote: post #327
satvikpendem wrote: post #325

Earlier quoted context omitted.

Or the US mandates only blessed models and thus disallows any other company from offering any other model.

That is still vulnerable: US-based get-rich-quick startup licenses a blessed model, or orders a few Gigatokens from another licensed /blessed model provider, at the same time it provides "blessed model" inference on its platform, but actually most of the inference is doing cheap foreign model inferences, the blessed model tokens were just bought to pretend serving the expensive blessed model. That is lucrative and no…

No, startups will not be allowed to provide inference anymore, it'll be only the big companies that personally have a relationship with and can follow the rules of the government, such as Anthropic, OpenAI, Google, Microsoft, and Amazon. That is the real risk to all this talk of regulation.

Re: Open-weight AI is having its Kubernetes moment

#337

Post by HDBaseT »

thih9 wrote: post #16

Is anyone using open weight models for agentic coding? What is your stack (harness, model) and how much do you pay per month? How would you compare your experience to a typical subsidized plan like Claude Code + Pro plan? I’m asking because i keep hearing that open weight models are cheap and efficient - is that really the case in practice?

At least with the Kimi Code plan, its limits are pretty abysmal compared to ChatGPT/Claude plans.

Re: Open-weight AI is having its Kubernetes moment

#338

Post by HDBaseT »

regexorcist wrote: post #320
applicative wrote: post #170

Everyone just keeps assuming, as if it were the law of gravity, that China will continue in perpetuity to deliver the weights of its 'frontier' models to Hugging Face. Its Mythos moment is a few months away and there is plenty of reporting suggesting their response will be similar, which is anyway obvious. It baffles me that anyone can seriously believe that China is going to put its Mythos successor on Hugging Face…

> there is plenty of reporting suggesting their response will be similar There is? Xi Jinping himself openly talked very recently about China's commitment to open weights models.

Xi Jinping realizes that open weight models accelerate open development.

There is more players in China than in the US. If everyone keeps learning off each other, eventually one of the China labs will overtake the US, but AI models aren't static (anything but), and the US will leap frog in a weeks time. Therefore the constant need for open weights remains.

Post reply on HN