Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The unbearable cheapness of open weight models

jamesoclaire.com

Post reply on HN
Print view View original

151–160 of 195 posts

Re: The unbearable cheapness of open weight models

#151

Post by petra »

dathanb82 wrote: post #149
Tuna-Fish wrote: post #135

Earlier quoted context omitted.

This is an example of common knowledge that is wrong. People look at their cash burn, assume that they spend this to subsidize inference, and get bonkers answers. Inference is not their largest expense. Inference is cheap. Anthropic is only drastically subsidizing their plans if you count their training expenses as part of their costs.

If "inference is cheap," why is OpenAI spending a ton getting Broadcom to design custom AI chips that make inference cheaper? Reports suggest their custom silicon isn't all that good for training, it's all to make inference more efficient. That shouldn't be necessary if inference is already quite cheap.

A large part of the market will be ad based. For that, having the lowest cost inference is useful.

Also for agent doing r&d, cheaper tokens allows doing more, which is always good.

Re: The unbearable cheapness of open weight models

#152

Post by dada216 »

hakfoo wrote: post #141
manwithopinions wrote: post #124

Earlier quoted context omitted.

So why are they losing so much money? Money is made on the subset of inference that is charged at cost + margin via their APIs. API usage is so high because customers are still finding their feet, trying to understand how to measure the value they get from their spend, erring on the side of spend. Yes, in a world of unmeasured value and tokenmaxxing, inference is profitable on SOTA models because all capacity is bein…

> Outside of coding, almost every business case for AI doesn’t need above human intelligence, it doesn’t even need human intelligence, or half a human intelligence, a business can extract a lot of value from a machine that has a fraction of a human’s intelligence. What the business world actually needed isn't intelligence, it's VBA with a bit of polish on it. Yeah, people want tools to distill reports, and puff nonse…

I have deployed very successful LLM based software that reads sales people emails and inserts orders, or stuff akin to orders, in the rest of the systems. Can you write me a regex that parses a messy human email thread and produces a clean JSON with all the order details? It's been working for a year with less than 3% error rate, better than what the humans themselves were doing.

Re: The unbearable cheapness of open weight models

#153

Post by dada216 »

clickety_clack wrote: post #76
fredley wrote: post #23

Earlier quoted context omitted.

3) Buy all the RAM, increasing the barrier to entry to push back the tide a bit, in time for a juicy IPO.

4) Make it illegal to use anything but regulated models.

Then they will leave the huge advantage in cost to the competition, I mean their customers competitors. Hard to fathom how US companies will not want to use the cheaper option when EU and Asian companies can.

Re: The unbearable cheapness of open weight models

#154

Post by ben_w »

jgord wrote: post #126
ben_w wrote: post #68

Earlier quoted context omitted.

> You can watch Scott Manley's video on it, where he does some rough calculations and explains the overall architecture. I'm currently writing a blog post, and there's one big thing everyone, including Scott Manley, missed. Once I realised it, I wondered what took me so long to spot this issue.

care to share the one glaring obstacle ? slightly related .. I saw a talk on DCs in space, and it said median Earth orbit had a latency of 500ms .. but back of envelope seems to be : 15,000km above Earth would have around 100ms latency, comparable to internet ping times. Not an expert, feel free to weigh in.

> care to share the one glaring obstacle ?

I'm still working on the blog, but as a quickie: it's the lesson of the Datasaurus dozen, that sometimes you need to look at the actual distribution rather than statistics.

Here's what the safety exclusion zone around a million of them in orbit looks like, if arranged something like the current plan: https://raw.githubusercontent.com/BenWheatley/blog/refs/head...

There's no (safe) gaps. Plenty of physical space, but the safety margin eats it all up. Nothing else is allowed to use those orbital shells or anything between them.

Also, this is what happens if you put them all in a single orbit at the same altitude:

https://raw.githubusercontent.com/BenWheatley/blog/refs/head...

> slightly related .. I saw a talk on DCs in space, and it said median Earth orbit had a latency of 500ms .. but back of envelope seems to be : 15,000km above Earth would have around 100ms latency, comparable to internet ping times.

500ms means ~150,000 km travel distance; for that distance as round-trip time from origin to destination and back again means the one-way distance is 75,000 km, so if it's via a single satellite bounce then the average distance to the satellite would be 37,500 km: [You]-37.5Mm-[Satellite]-37.5Mm-[Them]-37.5Mm-[Satellite]-37.5Mm-[You].

I think they must be assuming all comms are via geostationary satellites. In some talks, this is what the speaker actually meant, though they may not have been clear about it; other times, there's talks from people who copied the former but perhaps didn't understand.

For DCs in space, even in GEO, it would be half the distance because you're communicating with the satellite itself not with someone else somewhere else on the ground.

Re: The unbearable cheapness of open weight models

#155

Post by amanaplanacanal »

rectang wrote: post #121
clickety_clack wrote: post #76

Earlier quoted context omitted.

4) Make it illegal to use anything but regulated models.

License the training corpus and encourage copyright suits against outputs from models trained on unlicensed corpora.

This won't work if the courts decide that training is fair use, which certainly seems the direction they are going.

Re: The unbearable cheapness of open weight models

#156

Post by amanaplanacanal »

ben_w wrote: post #154
jgord wrote: post #126

Earlier quoted context omitted.

care to share the one glaring obstacle ? slightly related .. I saw a talk on DCs in space, and it said median Earth orbit had a latency of 500ms .. but back of envelope seems to be : 15,000km above Earth would have around 100ms latency, comparable to internet ping times. Not an expert, feel free to weigh in.

> care to share the one glaring obstacle ? I'm still working on the blog, but as a quickie: it's the lesson of the Datasaurus dozen, that sometimes you need to look at the actual distribution rather than statistics. Here's what the safety exclusion zone around a million of them in orbit looks like, if arranged something like the current plan: https://raw.githubusercontent.com/BenWheatley/blog/refs/head... There's no…

My gut says another obstacle is maintenance. How long can a datacenter on the ground run without maintenance? How will this be affordable in orbit?

Re: The unbearable cheapness of open weight models

#157

Post by amanaplanacanal »

isoprophlex wrote: post #62

Aren't these open models so cheap because they're (partially) chinese gov. sponsored, and because they're stealing and redistributing the IP that comes in?

Whose IP do you think they are stealing? According to US courts, training is fair use. And even if it wasn't, they are distilling output from other models, which isn't copyrightable, again according to US courts.

Re: The unbearable cheapness of open weight models

#158

Post by amanaplanacanal »

snootypoot wrote: post #77

i agree with his statement that the big companies and the string pullers in government are inching toward banning open models.allowing the plebs unrestricted access to things seems against the wishes of the "you will own nothing and be happy" / "you will rent everything on the cloud and subscribe to your appliances" crowd such as blackrock and so on. anyone who disagrees is not seeing the forest, only the trees.

I don't see how they could ban them in the US. Code is speech, and the first amendment still mostly holds. They might try, but I don't see the courts upholding it.

Re: The unbearable cheapness of open weight models

#159

Post by ben_w »

amanaplanacanal wrote: post #156
ben_w wrote: post #154

Earlier quoted context omitted.

> care to share the one glaring obstacle ? I'm still working on the blog, but as a quickie: it's the lesson of the Datasaurus dozen, that sometimes you need to look at the actual distribution rather than statistics. Here's what the safety exclusion zone around a million of them in orbit looks like, if arranged something like the current plan: https://raw.githubusercontent.com/BenWheatley/blog/refs/head... There's no…

My gut says another obstacle is maintenance. How long can a datacenter on the ground run without maintenance? How will this be affordable in orbit?

People already talk about that, so I wouldn't be adding much new. That said, had already put in a bit about cost of launching.

TL;DR: Alphabet researchers (and Alphabet owns more of SpaceX than the entire IPO so if anything they're biased to optimism), recon it will take SpaceX launching about 370,000 tons to orbit before they've even figured out how to get the costs down to the point it makes sense to put these in orbit.

Re: The unbearable cheapness of open weight models

#160

Post by therobots927 »

xquce wrote: post #150
therobots927 wrote: post #140

Earlier quoted context omitted.

Are you an anthropic insider or something? Because if you are you should delete this comment. If you aren’t then you don’t know what the hell you’re talking about.

Surely the same can be said for the people saying the opposite?

I didn’t make a claim. The parent explicitly said it was a misconception that inference is not profitable.

No one knows if it’s profitable or not so we’re left to speculate.

Post reply on HN