Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The End of Moore's Law for AI? Gemini Flash Offers a Warning

sutro.sh

Post reply on HN
Print view View original

71–78 of 78 posts

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#71

Post by NetRunnerSu »

x-complexity wrote: post #68

The article assumes that there will be no architectural improvements / migrations in the future, & that Sparse MoE will always stay. Not a great foundation to build upon. Personally, I'm rooting for RWKV / Mamba2 to pull through, somehow. There's been some work done to increase their reasoning depths, but transformers still beat them without much effort. https://x.com/ZeyuanAllenZhu/status/1918684269251371164

In fact, what you need is a dynamic sparse live hyperfragmented Transformer MoE, rather than a product like RNN that is destined to be backward...

In terms of microbiology, the architecture of Transformer is more in line with the highly interconnected global receptive field of neurons

https://github.com/dmf-archive/PILF

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#72

Post by gchamonlive »

> This is the first time a major provider has backtracked on the price of an established model. While it may seem like a simple adjustment, we believe this signals a turning point. The industry is no longer on an endless downward slide of cost. Instead, we’ve hit a fundamental soft floor on the cost of intelligence, given the current state of hardware and software.

That is assuming pricing and price drops only occurred because of cost reductions caused by technical advancements. While that certainly played a role in it, that disregards the role investiment money takes.

Maybe we've hit a wall in the "Moore's law for AI", or maybe it's just harder to justify these massive investments while all you have to show for are marginal improvements to the eyes of these investors, which are becoming increasingly anxious to have their money back.

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#73

Post by guluarte »

simonw wrote: post #12
FirmwareBurner wrote: post #8

Earlier quoted context omitted.

Aren't all LLMs loosing money at this point?

I don't believe that's true on inference - I think most if not all of the major providers are selling inference at a (likely very small) margin over what it costs to serve them (hardware + energy). They likely lose money when you take into account the capital cost of training the model itself, but that cost is at least fixed: once it's trained you can serve traffic from it for as long as you chose to keep the model r…

Some companies like Google, Facebook, Microsoft, and OpenAI are definitely losing money providing free inference to millions of users daily. Companies where most users are using their API, like Anthropic, are probably seeing good margins since most of their users are paying users.

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#75

Post by int_19h »

fnord123 wrote: post #43
furyofantares wrote: post #30

Earlier quoted context omitted.

This is extremely theorycrafted but I see this as an excellent thing driving AI forward, not holding it back. I suspect a large part of the reason we've had many decades of exponential improvements in compute is the general purpose nature of computers. It's a narrow set of technologies that are universally applicable and each time they get better/cheaper they find more demand, so we've put an exponentially increasing…

They aren't arguing against LLMs They are arguing against their toaster's LLM to make the perfect toast from being trained on the tax policies of the Chang Dynasty.

Thing is, we keep finding out again and again that having a very broad training mix in the baseline model makes it better across the board, including in those specialized tasks when you fine-tune it.

As I understand it, the general ability to reason is what the models get out of "being trained on the tax policies of the Chang Dynasty", and we haven't really figured out a better way to do so than to throw most everything at them. And even if all you do is make toast, you still need some intelligence.

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#76

Post by krackers »

Wowfunhappy wrote: post #50
wat10000 wrote: post #40

Earlier quoted context omitted.

That would be strange. There's no hidden memory or data channel, the "thinking" output is all the model receives afterwards. If it's all nonsense, then nonsense is all it gets. I wouldn't be completely surprised if a context with a bunch of apparent nonsense still helps somehow, LLMs are weird, but it would be odd.

Wasn't there some study that just telling the LLM to write a bunch of periods first improves responses?

There are several such papers, off the top of my head one is https://arxiv.org/abs/2404.15758

It's a bit more subtle though, if I understand correctly this only works for parallelizable problems. Which makes intuitive sense since the model cannot pass information along with each dot. So in that sense COT can be seen as some form of sampling, which also tracks with findings that COT doesn't boost the "raw intelligence" but rather uncovers latent intelligence, converting pass@k to maj@k. Antirez touches upon this in [1].

On the other hand, I think problems with serial dependencies require "real" COT since the model needs to track the results of subproblems. There's also some studies which show a meta-structure to the COT itself though, e.g. if you look at DeepSeek there are clear patterns of backtracking and such that are slightly more advanced than naive repeated samplings. https://arxiv.org/abs/2506.19143

[1] https://news.ycombinator.com/item?id=44288049

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#77

Post by krackers »

krackers wrote: post #76
Wowfunhappy wrote: post #50

Earlier quoted context omitted.

Wasn't there some study that just telling the LLM to write a bunch of periods first improves responses?

There are several such papers, off the top of my head one is https://arxiv.org/abs/2404.15758 It's a bit more subtle though, if I understand correctly this only works for parallelizable problems. Which makes intuitive sense since the model cannot pass information along with each dot. So in that sense COT can be seen as some form of sampling, which also tracks with findings that COT doesn't boost the "raw intelligence…

Although thinking a bit more, even constrained to only output dots, there can still some amount of information passing between each token, namely in the hidden states. The attention block N layers deep will compute attention scores off of the residual stream for previous inputs at that layer, so some information can be passed along this way.

It's not very efficient though, because for token i layer N can only receive as input layer N-1 for tokens i-1, i-2... So information is sort of passed along diagonally. If handwavily the embedding represents some "partial result" then it can be passed along diagonally from (N-1, i-1) to (N, i) to have the COT for token i+1 continue to work on it. So this way even though the total circuit depth is still bounded by # of layers, it's clearly "more powerful" than just naively going from layer 1...n, because during the other steps you can maybe work on something else.

But it's still not as powerful as allowing the results at layer n to be fed back in, which effectively unrolls the depth. This maybe intuitively justifies the results in the paper (I think it also has some connection to communication complexity).

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#78

Post by fnord123 »

int_19h wrote: post #75
fnord123 wrote: post #43

Earlier quoted context omitted.

They aren't arguing against LLMs They are arguing against their toaster's LLM to make the perfect toast from being trained on the tax policies of the Chang Dynasty.

Thing is, we keep finding out again and again that having a very broad training mix in the baseline model makes it better across the board, including in those specialized tasks when you fine-tune it. As I understand it, the general ability to reason is what the models get out of "being trained on the tax policies of the Chang Dynasty", and we haven't really figured out a better way to do so than to throw most everyth…

> And even if all you do is make toast, you still need some intelligence.

No you don't. That was the point of the example.

Post reply on HN