Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-weight AI is having its Kubernetes moment

tobi.knaup.me

Post reply on HN
Print view View original

121–130 of 346 posts

Re: Open-weight AI is having its Kubernetes moment

#121

Post by aleph_minus_one »

cheriot wrote: post #110
aleph_minus_one wrote: post #87

Earlier quoted context omitted.

> - Unlike OSS where a single volunteer can keep a project going, training costs run into the $billions. Just some thought: Wouldn't it make sense to build some kind of volunteer computing project to train the next-generation LLM by volunteers, similar to the BOINC [1] projects or Folding@home [2]? N.B.: BOINC was particularly famous for SETI@home (completed), Einstein@Home, Rosetta@home and PrimeGrid. I still rememb…

I’ll be impressed if somebody can make that work considering the vastly larger compute required.

> I’ll be impressed if somebody can make that work considering the vastly larger compute required.

I think you underestimate the computational ressources that the mentioned (and similar-kinded) scientific projects needed. Also consider how much computational ressources people invested into cryptocurrency mining.

No, I think the reasons are different:

- Many companies that train AI model use training data which must not be distributed for copyright reasons (and using it is a legal gray zone)x.

- Also consider that the amount of training data is insane. Scientific projects (and cryptocurrency mining, too) have the property that typically the amount of data (storage requirements) is small (or at least the computation can be partitioned so that each sub-task needs little data), but the required computing ressources are insane.

- AI companies consider a huge part of their training data as their "secret sauce" (they often even paid lots of money to generate it, for example by paying world-renowned experts for writing an answer for some important question).

Thus: Yes, the required computation ressources are huge, but this is a problem for which I consider it to be plausible that it can be solved. The real problems are in my opinion different.

Re: Open-weight AI is having its Kubernetes moment

#122

Post by broodbucket »

amazingamazing wrote: post #103
qiine wrote: post #96

Earlier quoted context omitted.

qwen3.6 27B q5, llama.cpp, RTX 3090, pi, cost: electricity bill

Rex 3090 isn’t free. Even if you already owned it, it wasn’t free. That’s years of a $20 subscription

It's an asset, though, and bizarrely it's one that's been appreciating the past 5 years

Re: Open-weight AI is having its Kubernetes moment

#123

Post by thisoneisreal »

sschueller wrote: post #72
amazingamazing wrote: post #44

Sadly until china scales production of hardware it really isn’t economical to run this stuff yourself. It is good it exists though to put pressure against the labs. Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over.

Model-on-Chip is coming. GPU are for general computing but have a huge bottle neck for doing model inference. Even not being able to significantly update a model that is burned on a chip the performance gains are immense. You also don't need the latest chip fabs to make them drastically reducing the cost.

One thought I had is that you could use FPGAs to get hardware performance but maintain the ability to dynamically update. I don't know enough about hardware to consider trying such a thing but I'm curious if that could be made practical and economical somehow.

Re: Open-weight AI is having its Kubernetes moment

#124

Post by segmondy »

firasd wrote: post #2

One of the strangest things in the AI industry is 'tokenomics'. It's not very clear why using GPT-4 in early 2023 was so expensive and then six months later 20 bucks could get you a fair amount of GPT-4 inference. This pattern has continued across various labs/providers for years--there is a continuous see-saw of pricing that doesn't seem related to anything. So what open weight models do is at least provide a baseli…

What is strange about GPT-4 being expensive in 2023? Supply and demand. Which other model choices did we have? Prices are related to supply and demand. We see it play out with the introduction of capable open weight models or even other closed cloud models.

Re: Open-weight AI is having its Kubernetes moment

#125

Post by segmondy »

amazingamazing wrote: post #44

Sadly until china scales production of hardware it really isn’t economical to run this stuff yourself. It is good it exists though to put pressure against the labs. Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over.

I don't know what you mean by "economical", but it has been "economical" to run this stuff yourself for the last 3 years.

1. You must be willing to be resourceful. 2. Be willing to learn, do the hard things. 3. Accept the tradeoffs.

Re: Open-weight AI is having its Kubernetes moment

#126

Post by ozgung »

Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive attempt to ban them by origin.

So, any solution to this “problem” must include ALL open-weight models. As far as I understand this is exactly what they intend to do. Axios article linked in the post mentions that. As in this quote:

“The source described leading AI labs or their allies approaching the administration every 3-5 months with an idea to ban open-source models.”

It doesn’t say “Chinese” open-source models. Because they already know that it’s not feasible. Any regulation must cover all the models.

Now there are solutions for that latter problem. But they are all ugly and restrictive. Making a DRM-like license protection system mandatory can be a solution. If a company wants to run an open model in their own servers, they can only use approved and certified pure “American” models. This of course creates a monopoly for the big labs who are authorized to train and distribute such “open” models. A company can fine-tune the model for its own needs but of course can’t distribute the derivative model.

I’m sure there are other solutions but all of them would be equally ugly. Also these regulations can’t be enforced to other countries easily so only Americans will be restricted.

Re: Open-weight AI is having its Kubernetes moment

#128

Post by johnvanommen »

tyfon wrote: post #108
thih9 wrote: post #16

Is anyone using open weight models for agentic coding? What is your stack (harness, model) and how much do you pay per month? How would you compare your experience to a typical subsidized plan like Claude Code + Pro plan? I’m asking because i keep hearing that open weight models are cheap and efficient - is that really the case in practice?

I'm using qwen 3.6 35B unsloth 4 bit with my 5950x (128 gb memory) and a 3060 12 gb gpu with a self made harness. At 10k context I get about 40 tps generation and 500 tps prefill. At 100k context I get about 25 tps generation and 400 tps prefill. It works, but I often use gpt or claude to make a detailed enumerated plan of what I want to do first, then have qwen follow it. I'm not sure if it is economical or not, but…

> The biggest benefit for me is that it's all done locally, and I know the harness is not uploading anything or sending telemetry to someone else.

Are there any articles you’d recommend for this?

I have Qwen running on an HP Z8. Very nice platform.

I have mine in a sandbox, due to privacy fears.

Your solution sounds more elegant.

Re: Open-weight AI is having its Kubernetes moment

#129

Post by DennisP »

andsoitis wrote: post #102
mjhay wrote: post #70

Earlier quoted context omitted.

Drug companies spend more on marketing than R&D

So?

So pharmaceutical companies spend far less on marketing outside the US, partly because every other country besides New Zealand makes those incessant drug ads illegal, and partly because governments negotiate prices and keep profit margins down. If the argument is that R&D costs are what make drugs expensive, then we could easily eliminate an even greater expense by just copying what other developed nations do.

Re: Open-weight AI is having its Kubernetes moment

#130

Post by baby_souffle »

sschueller wrote: post #72
amazingamazing wrote: post #44

Sadly until china scales production of hardware it really isn’t economical to run this stuff yourself. It is good it exists though to put pressure against the labs. Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over.

Model-on-Chip is coming. GPU are for general computing but have a huge bottle neck for doing model inference. Even not being able to significantly update a model that is burned on a chip the performance gains are immense. You also don't need the latest chip fabs to make them drastically reducing the cost.

I don't think asics specific to a specific model or even model family are likely to be commodity hardware anytime soon.

It's extremely expensive to build that and you'll be at least two major model generations behind before you even get your first wafers back. By the time you got your production run ready to go and packaged for market nobody's going to care.

Once we end up going something like 24 months between major advances and capabilities for these models then I can start to see asics for a model being possible.

Post reply on HN