Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-weight AI is having its Kubernetes moment

tobi.knaup.me

Post reply on HN
Print view View original

181–190 of 346 posts

Re: Open-weight AI is having its Kubernetes moment

#181

Post by Y_Y »

thih9 wrote: post #16

Is anyone using open weight models for agentic coding? What is your stack (harness, model) and how much do you pay per month? How would you compare your experience to a typical subsidized plan like Claude Code + Pro plan? I’m asking because i keep hearing that open weight models are cheap and efficient - is that really the case in practice?

GLM 5.2 awq4 via Opencode, it was better than corporate's fave Sonnet 4.8 and a but worse than Opus 4.8. overall quite capable of a lot of what the dev team needed. Not as good as Sonnet 5 but also never runs out of tokens.

The flip side is that it takes four H200s to run, and that will only let you cache context for maybe three users.

Fingers crossed our Blackwells show up and Kimi 3 really releases weights, because at some point devs are spending a significant portion of their salary on tokens and it's somehow cheaper to buy these ridiculous DGX servers and rack and run them.

Re: Open-weight AI is having its Kubernetes moment

#182

Post by amarant »

satvikpendem wrote: post #171
ozgung wrote: post #126

Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive at…

It's simple, the US government will put any Chinese open model companies on the entity list which blocks any company which does business with the US from also doing business with the Chinese companies. This creates a chilling effect where even if it may be harder to tell, no US company will be able to provide or use any overt Chinese open model and won't even risk trying to go around as the punishments for trying to…

But that's just the thing with open weights: you're not doing any business with company that made the model. They might publish the weights to a, say, European host, and then you download the model from Europe and and run it on your servers in America, and suddenly it's very hard to tell where the model was originally created.

Re: Open-weight AI is having its Kubernetes moment

#183

Post by EMIRELADERO »

satvikpendem wrote: post #171
ozgung wrote: post #126

Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive at…

It's simple, the US government will put any Chinese open model companies on the entity list which blocks any company which does business with the US from also doing business with the Chinese companies. This creates a chilling effect where even if it may be harder to tell, no US company will be able to provide or use any overt Chinese open model and won't even risk trying to go around as the punishments for trying to…

Hosting and providing Chinese models wouldn't be the same as doing business with the sanctioned entities though, you don't interact with them in any capacity if you only use the weights and don't sign any contracts.

Re: Open-weight AI is having its Kubernetes moment

#184

Post by Danox »

esseph wrote: post #119
amazingamazing wrote: post #44

Sadly until china scales production of hardware it really isn’t economical to run this stuff yourself. It is good it exists though to put pressure against the labs. Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over.

> Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over. I don't see how these are related. The accuracy and capabilities of your model are directly related to its size. You need a lot of memory for that. It will be decades before we get enough useful memory in a phone form factor at a price point people can aff…

If it wasn’t for the current memory fiasco, Apple would be even closer to providing a solution on the desktop that is affordable for most (Hacker News participants) who want to run a large model locally at reasonable cost, which will happen within the next 3 to 5 years easy with the way software and hardware are progressing, the main-frame future for most people that use computers isn’t coming back so OpenAI and Anthropic and other frontier developers are going to be disappointed and will have to shift gears ala’ Meta, Microsoft?

The United States if it dares to (I think they will try) but isn’t going to be able to stuff AI models back into the bottle open source is the future and when it comes to AI models yes you’ll be able to customize it to your specifications locally but the genie is out of the bottle. The bull out of the barn and is running down the road.

If United States insist on trying to lock the doors, censor, sanction, the rest of the world will just design and engineer around the United States. Trying to put up a wall, will damaged the United States more particularly with the current performance of Taco. None of the other countries are going to follow the United States not with the current administration they will hedge their bets.

Apple, is using Google now but that will change because the world is probably going down the open path, it’s looking like there was no real rush and no reason to spend so much money on something that’s going to be a commodity in the end, the only hold up is hardware and if it wasn’t for this current memory fiasco, many more people would have access to the hardware that they need to run models locally.

Re: Open-weight AI is having its Kubernetes moment

#185

Post by moffkalast »

amarant wrote: post #182
satvikpendem wrote: post #171

Earlier quoted context omitted.

It's simple, the US government will put any Chinese open model companies on the entity list which blocks any company which does business with the US from also doing business with the Chinese companies. This creates a chilling effect where even if it may be harder to tell, no US company will be able to provide or use any overt Chinese open model and won't even risk trying to go around as the punishments for trying to…

But that's just the thing with open weights: you're not doing any business with company that made the model. They might publish the weights to a, say, European host, and then you download the model from Europe and and run it on your servers in America, and suddenly it's very hard to tell where the model was originally created.

Yep, add a few blank layers, fine tune it a tiny bit and the weight checksums nor parameter counts won't match with anything, while the model will be practically the exact same. Time and time again random startups have tried passing established open models as their own.

"You made this? I made this."

Of course a conspicuous architecture would still give it away.

Re: Open-weight AI is having its Kubernetes moment

#186

Post by ButlerianJihad »

holoduke wrote: post #176
rullopat wrote: post #149

Earlier quoted context omitted.

You need to ask what happened in Tienanmen square

All censorship whether it's tienanmen, sex nudity, violence etc are happening in a very thin layer of the model. It's easily bypassed. I use all open models without any censorship. I can easily ask how to make a nuclear bomb or how to build my own meth lab at home with a super details step by step list.

Have you followed the steps, though?

Last year I asked some ordinary commercial LLM if it could draw me the schematics of a device like Little Boy from WWII. And it complied very nicely. Got a very detailed schematic graphic. Suitable for framing. Perhaps not quite suitable for reconstruction.

Re: Open-weight AI is having its Kubernetes moment

#187

Post by nojs »

Y_Y wrote: post #181
thih9 wrote: post #16

Is anyone using open weight models for agentic coding? What is your stack (harness, model) and how much do you pay per month? How would you compare your experience to a typical subsidized plan like Claude Code + Pro plan? I’m asking because i keep hearing that open weight models are cheap and efficient - is that really the case in practice?

GLM 5.2 awq4 via Opencode, it was better than corporate's fave Sonnet 4.8 and a but worse than Opus 4.8. overall quite capable of a lot of what the dev team needed. Not as good as Sonnet 5 but also never runs out of tokens. The flip side is that it takes four H200s to run, and that will only let you cache context for maybe three users. Fingers crossed our Blackwells show up and Kimi 3 really releases weights, because…

What hardware are you expecting to run K3 on?

Re: Open-weight AI is having its Kubernetes moment

#188

Post by hackeraccount »

theshrike79 wrote: post #152
rullopat wrote: post #149

Earlier quoted context omitted.

You need to ask what happened in Tienanmen square

Would ”who won the 2020 election” be a similar canary for American models?

No. There are questions you can ask but that's not it. Don't be political in a way that's toxic to half the country - be political in a way that's toxic to the entire country. I'll leave what those lines of inquiry would be as an open exercise.

Re: Open-weight AI is having its Kubernetes moment

#189

Post by honkycat »

singingtoday wrote: post #61
honkycat wrote: post #20

Earlier quoted context omitted.

Recently my company bought another company, and we kept zero of the original engineers, we just had to run the ghost ship. We walked in, and it was fine. Because it was all kubernetes and laid out like every other app for the most part. The kube hate is just sad at this point. You need to know like 15 concepts that are all applied in the same way. It mostly just works.

I ported my company over to k8s to solve a concurrency and scaling issues. What 15 concepts? You're making me worry that I missed something. It was straight forward: pods, nodes, hw type, lifecycle, deployment. They run almost the same docker as the old ec2s used. What did I miss? Is there something important I need to read?

Lol no you got em

I would say:

- deployments - pods -services - ingress - namespaces - cert-manager - external DNS - external secrets - configmaps - hpa - docker - volumes/pvc

That's the basics

Re: Open-weight AI is having its Kubernetes moment

#190

Post by rvz »

ozgung wrote: post #126

Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive at…

> Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them.

It is not possible to 100% ban open weight models getting released in the same way you cannot stop leaks.

Just ask Meta with the original Llama leak.

Post reply on HN