Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-weight AI is having its Kubernetes moment

tobi.knaup.me

Post reply on HN
Print view View original

271–280 of 346 posts

Re: Open-weight AI is having its Kubernetes moment

#271

Post by georgeburdell »

m11a wrote: post #217
georgeburdell wrote: post #160

Earlier quoted context omitted.

No, it’s consistent with what China is doing in other markets, which is dumping product to drive others out of business. I had a shower thought on how to counteract this, specifically related to the AI dumping. If China is losing substantial money on every token, why wouldn’t an adversary try to maliciously increase consumption? This strategy is not really viable against physical goods dumping because demand is finit…

Because it's not losing money on each token? Aside from most global people using American inference providers to run the models, I suspect the cloud inference products of the Chinese labs are profitable, at least on the inference costs (ie: not including model training, salaries, etc).

There is no way Deepseek is making money even on inference

Re: Open-weight AI is having its Kubernetes moment

#272

Post by fragmede »

satvikpendem wrote: post #235
EMIRELADERO wrote: post #183

Earlier quoted context omitted.

Hosting and providing Chinese models wouldn't be the same as doing business with the sanctioned entities though, you don't interact with them in any capacity if you only use the weights and don't sign any contracts.

They can easily write the law such that your first part becomes illegal too.

So they get posted to hugging-piratebay-faces.ru or whatever by a mysterious Twitter account. 100% totally above board companies won't touch it, but the number of companies that wouldn't exist were it not for hacked copies of Microsoft office and Photoshop and shared logins would surprise you.

Re: Open-weight AI is having its Kubernetes moment

#273

Post by fragmede »

rvz wrote: post #190
ozgung wrote: post #126

Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive at…

> Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. It is not possible to 100% ban open weight models getting released in the same way you cannot stop leaks. Just ask Meta with the original Llama leak.

You can stop leaks. If no one wants to leak it, it doesn't get leaked. Mythos.gguf would be an awesome torrent to appear but it hasn't, for lots of reasons.

Re: Open-weight AI is having its Kubernetes moment

#275

Post by sjsdaiuasgdia »

appplication wrote: post #197
hackeraccount wrote: post #188

Earlier quoted context omitted.

No. There are questions you can ask but that's not it. Don't be political in a way that's toxic to half the country - be political in a way that's toxic to the entire country. I'll leave what those lines of inquiry would be as an open exercise.

It’s funny how asking “who won the 2012 election” and “who won the 2016 election” are not political but suddenly “who won the 2020 election” is. I think that should tell you whoever takes an easily verifiable fact and argues that it is “political” is a raging idiot. That said, I truly don’t mean that disparagingly. I just mean literally it’s right up there with flat earthers. There is a very low bar for critical thou…

>but there was very clearly only one person who sat in office for the four following years. It is not disputable.

Careful now, this path allows the weasel option of "Joe Biden was certified as the winner of the 2020 election" and similar "Biden didn't win, but he was installed" bs.

Re: Open-weight AI is having its Kubernetes moment

#276

Post by arczyx »

georgeburdell wrote: post #271
m11a wrote: post #217

Earlier quoted context omitted.

Because it's not losing money on each token? Aside from most global people using American inference providers to run the models, I suspect the cloud inference products of the Chinese labs are profitable, at least on the inference costs (ie: not including model training, salaries, etc).

There is no way Deepseek is making money even on inference

Pretty sure they are making money since on OpenRouter, there are other providers for DeepSeek V4 flash that are charging even less than DeepSeek themselves (eg DeepInfra and Digital Ocean).

https://openrouter.ai/compare/deepseek/deepseek-v4-flash/ten...

Re: Open-weight AI is having its Kubernetes moment

#277

Post by DrewADesign »

eszed wrote: post #154
DrewADesign wrote: post #81

Earlier quoted context omitted.

I think the argument is that artificial comes in with IP law, which some people feel is superfluous. I do think that corporate price gouging is a huge problem that does need to be addressed. But especially with smaller business types — creatives, et al— I still haven’t gotten any grownup answers about what would compel people to get professionally good at something and innovate in the complete absence of copyright: t…

I'm with you on all of that. There is, nevertheless, a strong argument that IP protection (particularly for creative / "culturally significant" works) is too long . Twenty years - interestingly enough, the original time-period in the US - of protection seems like a better (for society) deal than life of the author plus seventy. I think, in fact, most artists would agree: if you went back in time and asked a playwrite…

I’ve never met anyone that didn’t think copyrights were way too long in the US, and I’ve got a very large sample size of artists and attorneys. The only people that support such things are executives or counsel for large IP-holding entertainment companies.

I have, however, met a ton of very well-paid tech workers that were extremely against copyright, entirely, especially where it came to paying artists, such as musicians, for their labor. Pretty ironic because the market for software development labor market would probably land somewhere between graphic designers and company IT worker if the commercial software business had no IP protection.

Re: Open-weight AI is having its Kubernetes moment

#278

Post by munchler »

Krei-se wrote: post #246
munchler wrote: post #172

Earlier quoted context omitted.

Thank you for this explanation. It makes sense, but I don't really understand why it has become so popular. Professionally, my experience is that certain software components need to run together on an individual machine (e.g. database server, app server, web server), and then those machines need to be networked in a certain way (e.g. web server talks to app server, which talks to database server), so I really need to…

Your host with the webserver has no idea its just a docker image. It talks to app.domain.tld and gets a reply. app.domain.tld asks db.domain.tld with an SQL query and gets a reply. But it can be one of any deployed docker image on any bare metal host - which one is db. and app. etc. is decided by Kubernetes. In the background kubernetes routes all these docker images with each other without you having to think about…

Thank you for this. I probably don't have higher standards, but I do have way fewer than a million users!

Re: Open-weight AI is having its Kubernetes moment

#279

Post by potwinkle »

drnick1 wrote: post #223
stefan_ wrote: post #195

Earlier quoted context omitted.

"We have been having extensive discussions around open source strategy. [..] one thing we'd like to do soon is to create a language model with the approximate capability of GPT-3 that can run locally [..]. In general, we think this helps discourage others from releasing similarly-powerful models, and makes it harder for new efforts to get funded." - Sam Altman emails OpenAI board, 2.5 years after GPT-3 Chinese releas…

> Chinese release open models to drive the state of the art Are you sure it isn't just another form of Chinese industrial policy? China does not have frontier labs, but through distillation and their own work they can get pretty close. It's not enough to be competitive with Anthropic and OpenAI, but there is still money to be made by selling compute (software as a service), and in any case it's better than being left…

There are coding tasks where Kimi K3 outperforms Fable (haven't done much comparing between it and Sol) and the Chinese labs have access to their own synthetic datasets, along with frontier research. We're leaving behind the days where Chinese models are distilled Claude, but I hope Anthropic/OpenAI can continue to accelerate.

Re: Open-weight AI is having its Kubernetes moment

#280

Post by hedora »

downrightmike wrote: post #269
mft_ wrote: post #174

Earlier quoted context omitted.

I’m happy to be proven wrong, but the limited examples I’ve seen of clustered Strix Halos are quite slow running large models (ie models too large to fit into the ram of a single machine) due to the slow networking between each one?

That is already being addressed with GPUs and special PCIe daughter cards that link them together faster than the PCIe on the MOBO could. SLI but better.

Yeah; the benchmarks + setup guide I linked show that it takes a lot of careful setup to get them to work at all, and then you only get 100's of tokens per second per cluster.

That's why I pointed out the next generation is coming soon. Also, the AMD docs aren't using quantization (as far as I can tell, I only skimmed), which gives a speedup roughly linear in the compression ratio. Algorithms for that continue to improve, so expect a lossless factor of 2-8x on DRAM and throughput, at least.

Post reply on HN