Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The End of Moore's Law for AI? Gemini Flash Offers a Warning

sutro.sh

Post reply on HN
Print view View original

41–50 of 78 posts

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#41

Post by mathiaspoint »

wat10000 wrote: post #40
thomashop wrote: post #39

Earlier quoted context omitted.

I have the impression that the thinking helps even if the actual content of the thinking output is nonsense. It awards more cycles to the model to think about the problem.

That would be strange. There's no hidden memory or data channel, the "thinking" output is all the model receives afterwards. If it's all nonsense, then nonsense is all it gets. I wouldn't be completely surprised if a context with a bunch of apparent nonsense still helps somehow, LLMs are weird, but it would be odd.

Eh. The embeddings themselves could act like hidden layer activations and encode some useful information.

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#42

Post by yorwba »

wat10000 wrote: post #40
thomashop wrote: post #39

Earlier quoted context omitted.

I have the impression that the thinking helps even if the actual content of the thinking output is nonsense. It awards more cycles to the model to think about the problem.

That would be strange. There's no hidden memory or data channel, the "thinking" output is all the model receives afterwards. If it's all nonsense, then nonsense is all it gets. I wouldn't be completely surprised if a context with a bunch of apparent nonsense still helps somehow, LLMs are weird, but it would be odd.

Attention operates entirely on hidden memory, in the sense that it usually isn't exposed to the end user. An attention head on one thinking token can attend to one thing and the same attention head on the next thinking token can attend to something entirely different, and the next layer can combine the two values, maybe on the second thinking token, maybe much later. So even nonsense filler can create space for intermediate computation to happen.

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#43

Post by fnord123 »

furyofantares wrote: post #30
fusionadvocate wrote: post #26

What is holding back AI is this business necessity that models must perform everything. Nobody can push for a smaller model that learns a few simple tasks and then build upon that, similar to the best known intelligent machine: the human. If these corporations had to build a car they would make the largest possible engine, because "MORE ENGINE MORE SPEED", just like they think that bigger models means bigger intellig…

This is extremely theorycrafted but I see this as an excellent thing driving AI forward, not holding it back. I suspect a large part of the reason we've had many decades of exponential improvements in compute is the general purpose nature of computers. It's a narrow set of technologies that are universally applicable and each time they get better/cheaper they find more demand, so we've put an exponentially increasing…

They aren't arguing against LLMs They are arguing against their toaster's LLM to make the perfect toast from being trained on the tax policies of the Chang Dynasty.

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#44

Post by barrkel »

wat10000 wrote: post #40
thomashop wrote: post #39

Earlier quoted context omitted.

I have the impression that the thinking helps even if the actual content of the thinking output is nonsense. It awards more cycles to the model to think about the problem.

That would be strange. There's no hidden memory or data channel, the "thinking" output is all the model receives afterwards. If it's all nonsense, then nonsense is all it gets. I wouldn't be completely surprised if a context with a bunch of apparent nonsense still helps somehow, LLMs are weird, but it would be odd.

This isn't quite right. Even when an LLM generates meaningless tokens, its internal state continues to evolve. Each new token triggers a fresh pass through the network, with attention over the KV cache, allowing the model to refine its contextual representation. The specific tokens may be gibberish, but the underlying computation can still reflect ongoing "thinking".

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#45

Post by trhway »

cmogni1 wrote: post #2

The article does a great job of highlighting the core disconnect in the LLM API economy: linear pricing for a service with non-linear, quadratic compute costs. The traffic analogy is an excellent framing. One addition: the O(n^2) compute cost is most acute during the one-time prefill of the input prompt. I think the real bottleneck, however, is the KV cache during the decode phase. For each new token generated, the m…

That obviously should and will be fixed architecturally.

>For each new token generated, the model must access the intermediate state of all previous tokens.

Not all the previous tokens are equal, not all deserve the same attention so to speak. The farther the tokens, the more opportunity for many of them to be pruned and/or collapsed with other similarly distant and lesser meaningful tokens in a given context. So instead of O(n^2) it would be more like O(nlog(n))

I mean, you'd expect that for example "knowlegde worker" models (vs. say "poetry" models) would posses some perturbative stability wrt. changes to/pruning of the remote previous tokens, at least to those tokens which are less meaningful in the current context.

Personally, i feel the situation is good - performance engineering work again becomes somewhat valuable as we're reaching N where O(n^2) forces management to throw some money at engineers instead of at the hardware :)

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#46

Post by [deleted] »

gpm wrote: post #25
georgeburdell wrote: post #20

Is there math backing up the “quadratic” statement with LLM input size? At least in the traffic analogy, I imagine it’s exponential, but for small amounts exceeding some critical threshold, a quadratic term is sufficient

Every token has to calculate attention for every previous token, that is that attention takes O(sum_i=0^n i) work, sum_i=0^n i = n(n-1)/2, so that first expression is equivalent to O(n^2). I'm not sure where you're getting an exponential from.

[deleted]

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#48

Post by llm_nerd »

Basing anything on Google's pricing is folly. Quite recently Google offered several of their preview models at a price of $0.00.

Because they were the underdog. Everyone was talking about ChatGPT, or maybe Anthropic. Then Deepseek. Google were the afterthought that was renowned for that ridiculous image generator that envisioned 17th century European scientists as full-headdress North American natives.

There has been absolute 180 since then, and Google now has the ability to set their pricing similar to the others. Indeed, Google's pricing still has a pretty large discount over similarly capable model levels, even after they raised prices.

The warning is that there is no free lunch, and when someone is basically subsidizing usage to get noticed, they don't have to do that once their offering is good.

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#49

Post by mpalmer »

Is this overthinking it? Google had a huge incentive to outprice Anthropic and OAI to join the "conversation". I was certainly attracted to the low price initially, but I'm staying because it's still affordable and I still think the Gemini 2.5 options are the best simple mix of models available.

Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning

#50

Post by Wowfunhappy »

wat10000 wrote: post #40
thomashop wrote: post #39

Earlier quoted context omitted.

I have the impression that the thinking helps even if the actual content of the thinking output is nonsense. It awards more cycles to the model to think about the problem.

That would be strange. There's no hidden memory or data channel, the "thinking" output is all the model receives afterwards. If it's all nonsense, then nonsense is all it gets. I wouldn't be completely surprised if a context with a bunch of apparent nonsense still helps somehow, LLMs are weird, but it would be odd.

Wasn't there some study that just telling the LLM to write a bunch of periods first improves responses?
Post reply on HN