Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If this is true, the hyperscalers are toast

klementoninvesting.substack.com

Post reply on HN
Print view View original

61–70 of 108 posts

Re: If this is true, the hyperscalers are toast

#61

Post by readthenotes1 »

philipallstar wrote: post #22

This logic seems mad. If people only need SLMs then hyperscalers can also centrally host higher-efficiency models, and still gain efficiencies of scale and convenience over hosting locally.

That sounds a little bit like the guy who said that we don't need a computer in the home.

There may be a great deal of advantage to be able to run a small language model on something I have in my hand, disconnected.

Although mainframes have their use, the pendulum of centralized to distributed has gone back and forth and there are benefits to be gleaned from either model, sometimes at the same time.

Re: If this is true, the hyperscalers are toast

#62

Post by genxy »

philipallstar wrote: post #22

This logic seems mad. If people only need SLMs then hyperscalers can also centrally host higher-efficiency models, and still gain efficiencies of scale and convenience over hosting locally.

Hyperscalers don't run computing at some multiple more efficient than on prem. The only way hyperscalers can compete is if they own the sand to cycles supply chain and ensure that raw compute is out priced in the market (ram,flash,compute). Ram and flash were an easy target because they are a commodity in name only.

Re: If this is true, the hyperscalers are toast

#63

Post by slfnflctd »

amelius wrote: post #36
fph wrote: post #31

Earlier quoted context omitted.

But do you really need a model that has the complete Duran Duran discography memorized and preloaded in RAM at all time?

That's a different question. Probably not. But: 1. Training a large model with lots of information, then stripping the "useless" information from that model to obtain a small model => nobody has shown this. 2. Training a small model, letting it use a database tool so it scores the same as a large model without database => nobody has shown this.

This really elegantly sums up two of the most major areas of research in the field right now. I expect people are going to hammer at both problems for a while. It will be interesting to see how much useful stuff is built as a result.

Re: If this is true, the hyperscalers are toast

#65

Post by rotis »

fph wrote: post #31
amelius wrote: post #26

Earlier quoted context omitted.

The logic seems mad to me because SLMs can simply not hold as much information as an LLM. Maybe if you combine an SLM with a database (as a tool) then it could work, but someone should first prove that.

But do you really need a model that has the complete Duran Duran discography memorized and preloaded in RAM at all time?

Sure. I don't give a damn about them. But Phil Collins man. I love reviewing his body of work before I get worked up. Obvously we cannot cut him, because I need him. So how do you decide what to leave out?

Re: If this is true, the hyperscalers are toast

#66

Post by [deleted] »

regularfry wrote: post #32
philipallstar wrote: post #22

This logic seems mad. If people only need SLMs then hyperscalers can also centrally host higher-efficiency models, and still gain efficiencies of scale and convenience over hosting locally.

The valuations of the hyperscalars won't sustain just being more efficient than something you can run locally. There's a market there, but it's for margin on a commodity. They're priced for oligopoly on unique, premium products.

[deleted]

Re: If this is true, the hyperscalers are toast

#67

Post by Jamesbeam »

I think it’s a bit more complicated.

There is a likely US scenario and a rest of the world scenario. It will be interesting to see if China acts on the overextension of the US Military in the Middle East. Taiwan will be a big play for both and crucial to the hyperscalers.

But since the US is dabbling in piracy again and telling people what they can do and not do with their shit, it’s not too far-fetched that everyone that is not a global superpower is at risk of getting bombed to smithereens if they are a danger to US AI supremacy.

This is such a crazy timeline, predicting even like a single year ahead feels like looking into a medieval glass ball.

But we are humans, I am confident we will find a way to fuck this up royally for everyone. Brace for impact.

Re: If this is true, the hyperscalers are toast

#68

Post by classified »

slfnflctd wrote: post #60
regularfry wrote: post #32

Earlier quoted context omitted.

The valuations of the hyperscalars won't sustain just being more efficient than something you can run locally. There's a market there, but it's for margin on a commodity. They're priced for oligopoly on unique, premium products.

Yes, it's all about the amount of money invested right now. Justifying that was always going to be tricky, and it still is. The compute capacity, however, is here to stay. There will be a seemingly endless line of people queued up to buy it at pennies on the dollar if this whole thing blows up financially. And they will use it!

> And they will use it!

Just out of curiosity: Use it for what?

Re: If this is true, the hyperscalers are toast

#69

Post by js8 »

embedding-shape wrote: post #39
fph wrote: post #31

Earlier quoted context omitted.

But do you really need a model that has the complete Duran Duran discography memorized and preloaded in RAM at all time?

I mean maybe yes? The hypothesis from the early GPT days was (and in a small way still remains): "If we just chuck more data into the training, does it get better at X, even if the data was seemingly unrelated to X?", and the workings of LLMs seem to kind be pointing in that direction, although with some ceiling. But seemingly models good at programming for example, would get worse at programming if you removed every…

"But seemingly models good at programming for example, would get worse at programming if you removed everything not-programming. Train a model solely on syntax, and it'll be worse than a general purpose LLM on syntax, in general at least."

That might be because emergence of capabilities to reason about programs requires abstractions (such as fuzzy and modal logic) that are rarely present in software sources. That doesn't mean the reasoning model itself has to be large; neither does it have to emerge from the ML training on large language corpus, we might construct it by different means.

Re: If this is true, the hyperscalers are toast

#70

Post by Majromax »

genxy wrote: post #62
philipallstar wrote: post #22

This logic seems mad. If people only need SLMs then hyperscalers can also centrally host higher-efficiency models, and still gain efficiencies of scale and convenience over hosting locally.

Hyperscalers don't run computing at some multiple more efficient than on prem. The only way hyperscalers can compete is if they own the sand to cycles supply chain and ensure that raw compute is out priced in the market (ram,flash,compute). Ram and flash were an easy target because they are a commodity in name only.

> Hyperscalers don't run computing at some multiple more efficient than on prem.

I'd disagree here. I see two avenues for an efficiency multiple, albeit a single-digit multiple:

* Client aggregation allows a hyperscaler to average out demand spikes from uncorrelated clients, reducing the peak:average demand ratio and allowing better budgeting of compute.

* Dynamic batching allows typical requests to run in batches of more-than-1 and/or overlap, offering better internal compute utilization ratios (e.g. interleaving output and input streams). The small limit of on-device LLMs will run with batch sizes of one with strong memory bandwidth bottlenecks.

For an example of these factors in action, see the API cost differential between batch, standard, and 'fast' processing. OpenAI prices these tiers at a 1:2:4 ratio.

Post reply on HN