Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-weight AI is having its Kubernetes moment

tobi.knaup.me

Post reply on HN
Print view View original

101–110 of 346 posts

Re: Open-weight AI is having its Kubernetes moment

#101

Post by aghilmort »

thewebguyd wrote: post #28
firasd wrote: post #2

One of the strangest things in the AI industry is 'tokenomics'. It's not very clear why using GPT-4 in early 2023 was so expensive and then six months later 20 bucks could get you a fair amount of GPT-4 inference. This pattern has continued across various labs/providers for years--there is a continuous see-saw of pricing that doesn't seem related to anything. So what open weight models do is at least provide a baseli…

> if you really want Kimi K2 instead of K3 you can still use it. I think this is a very important aspect, especially after the huge GPT-4o backlash when GTP-5 came out. Each model has certain quirks, and areas where the previous model might be better for some use cases than the latest and greatest, and the labs so far seem to have no desire to offer some kind of "LTS" release.

LTS is really useful framing even if increasingly distilled or slower on older hardware etc vs disappearing model acts

Re: Open-weight AI is having its Kubernetes moment

#102

Post by andsoitis »

mjhay wrote: post #70
andsoitis wrote: post #53

Earlier quoted context omitted.

Drug companies have a portfolio of compounds they research. Most don’t pay off, so R&D costs make their way into the pricing of those superstar and other drugs that do work. Also, timelines are pretty long.

Drug companies spend more on marketing than R&D

So?

Re: Open-weight AI is having its Kubernetes moment

#103

Post by amazingamazing »

qiine wrote: post #96
thih9 wrote: post #16

Is anyone using open weight models for agentic coding? What is your stack (harness, model) and how much do you pay per month? How would you compare your experience to a typical subsidized plan like Claude Code + Pro plan? I’m asking because i keep hearing that open weight models are cheap and efficient - is that really the case in practice?

qwen3.6 27B q5, llama.cpp, RTX 3090, pi, cost: electricity bill

Rex 3090 isn’t free. Even if you already owned it, it wasn’t free. That’s years of a $20 subscription

Re: Open-weight AI is having its Kubernetes moment

#104

Post by bee_rider »

jmyeet wrote: post #32
firasd wrote: post #2

One of the strangest things in the AI industry is 'tokenomics'. It's not very clear why using GPT-4 in early 2023 was so expensive and then six months later 20 bucks could get you a fair amount of GPT-4 inference. This pattern has continued across various labs/providers for years--there is a continuous see-saw of pricing that doesn't seem related to anything. So what open weight models do is at least provide a baseli…

Yeah I've been thinking about this and the analogy I came up with is that tokens are basically equivalent to an in-game currency in -free to play" mobile games. You're trading actual money for some notional "curency" or coins that can only be used for one thing but, unlike mobile game coins, you don't know how many coins something costs before you use them. It's kinda weird.

I think it is worse actually. Tokens in a game are usually just purchased for enjoyment in the game. They are purchased as a part of your entertainment budget, not expected to be useful in any way.

LLMs are fundamentally tools intended to be useful. But LLM vendors don’t understand their systems well enough to actually price the product people are trying to buy (for example, the actual product of a coding model is the code that it produces, not the tokens, which are just an internal mechanical process involved in the creation of the code). Token based pricing is that lack of understanding leaking out of the organization that ought to be responsible for it, and being dropped on the user.

Imagine if we made cars like this! You’d go to the car dealer and ask for a car. They’d bring you a pile of parts, charge you for them, and try to put them together in front of you. You’d go back and forth for a bit, rephrase where you want the steering wheel, etc. Some of the parts wouldn’t fit but you’d be invited to pay for replacements as well. In the end you’d either have a car or not, that’s your problem.

Re: Open-weight AI is having its Kubernetes moment

#105

Post by eigenspace »

jumpkick wrote: post #92
eigenspace wrote: post #45

Earlier quoted context omitted.

The weights themselves aren't stolen. The claim is that Chinese companies are using VPNs and proxies to buy massive amounts of Claude Pro and Codex subscription accounts, and then selling usage on those subscriptions as cheap white-label LLM API usage. While selling that LLM API usage, they then capture all the prompts, outputs, and intermediate thinking the LLM does, and then sell those logs to the companies making…

The parent I replied to said frontier weights were “being stolen” which is not the literal case. I think precision is important here.

I agree. That's why I said that the weights were not being stolen, and then explained what is actually being done.

Re: Open-weight AI is having its Kubernetes moment

#106

Post by mlyle »

bobthebob wrote: post #97
mlyle wrote: post #84

Earlier quoted context omitted.

> Because you need those calculations to "pump" nuclear weapons. They are needed to calculate both the geometry to make fusion bombs possible at all and to calculate the effect of a given geometry. They are the reason US/Russia/China have the biggest and strongest weapons known to humanity. We have a couple new nuclear weapon designs, but not really going for bigger or stronger. Just packaging. We built the big power…

High end compute also existed in the 60’s. The military has already mastered fusion (power), and likely has mastered gravity in some form in secret. They aren’t using mainstream compute for these discoveries

> High end compute also existed in the 60’s.

Believe me, I know-- my dad did some work on the 360/44 and other large systems.

High end late 1960s compute -- of the sort used to go to the moon or to design big nuclear weapons -- was roughly 486DX4-100 class. Not individual computers; the total computing at DOE or NASA. Of course, it would be hard to replace either with a single 486 because of availability, usage at different geographic locations, etc.

You can assume a single large AMD Threadripper machine ($25k?) outclasses late-1960s DoE by roughly a factor of 50,000. And that assumes you didn't bother to put a GPU in it.

> The military has already mastered fusion (power), and likely has mastered gravity in some form in secret. They aren’t using mainstream compute for these discoveries

K

Re: Open-weight AI is having its Kubernetes moment

#108

Post by tyfon »

thih9 wrote: post #16

Is anyone using open weight models for agentic coding? What is your stack (harness, model) and how much do you pay per month? How would you compare your experience to a typical subsidized plan like Claude Code + Pro plan? I’m asking because i keep hearing that open weight models are cheap and efficient - is that really the case in practice?

I'm using qwen 3.6 35B unsloth 4 bit with my 5950x (128 gb memory) and a 3060 12 gb gpu with a self made harness.

At 10k context I get about 40 tps generation and 500 tps prefill. At 100k context I get about 25 tps generation and 400 tps prefill.

It works, but I often use gpt or claude to make a detailed enumerated plan of what I want to do first, then have qwen follow it.

I'm not sure if it is economical or not, but I have solar on the roof so the power use is not really an issue and I already have the hardware.

The biggest benefit for me is that it's all done locally, and I know the harness is not uploading anything or sending telemetry to someone else.

Re: Open-weight AI is having its Kubernetes moment

#109

Post by [deleted] »

Scene_Cast2 wrote: post #46
thih9 wrote: post #16

Is anyone using open weight models for agentic coding? What is your stack (harness, model) and how much do you pay per month? How would you compare your experience to a typical subsidized plan like Claude Code + Pro plan? I’m asking because i keep hearing that open weight models are cheap and efficient - is that really the case in practice?

I'm using Kimi K3 + OpenCode. I pay their API pricing, costs about $5 / hour (and chews through ~10 million tokens / hour) during continuous use when I have one or two sessions running and doing their thing. Can't comment on how it compares to plans (I really don't like the limitations and general shenanigans I see around plans, so I've never tried them). It is notably slower than Fable / Opus / Gemini, but also vast…

[deleted]

Re: Open-weight AI is having its Kubernetes moment

#110

Post by cheriot »

aleph_minus_one wrote: post #87
cheriot wrote: post #73

Open-weight and OSS are wildly different and the article makes a poor comparison. What's the incentive for the Chinese labs to continue releasing weights 5 years from now? It's not a stable equilibrium and cannot last. - The lab spending large sums on research and training does not get the inference revenue to fund those efforts. - Unlike OSS where a single volunteer can keep a project going, training costs run into…

> - Unlike OSS where a single volunteer can keep a project going, training costs run into the $billions. Just some thought: Wouldn't it make sense to build some kind of volunteer computing project to train the next-generation LLM by volunteers, similar to the BOINC [1] projects or Folding@home [2]? N.B.: BOINC was particularly famous for SETI@home (completed), Einstein@Home, Rosetta@home and PrimeGrid. I still rememb…

I’ll be impressed if somebody can make that work considering the vastly larger compute required.
Post reply on HN