Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia's Risky Business

stratechery.com

Post reply on HN
Print view View original

111–120 of 185 posts

Re: Nvidia's Risky Business

#111

Post by ryukoposting »

mattnewton wrote: post #33
onlyrealcuzzo wrote: post #21

Earlier quoted context omitted.

What makes this insanely hard to predict is that the compute needed for the same quality output has roughly gone down 90% every 18 months for ~5 years. 1) We don't know how long that trend will continue, but you do know where to look for when it may end (if smaller sized models continue to compress the knowledge effectively of larger models). 2) We don't know when the appetite for higher cost models might go down and…

I think efficiency is unlikely to result in lower demand for compute, instead more useful compute per watt increases the value of that compute; and we are not going to run out of economically useful things to do with it anytime soon on the demand side. The harder thing to forecast for me is if we hit a wall on increasing efficiency, either on the model weights side or silicon side, with current approaches. If we have…

> efficiency is unlikely to result in lower demand for compute, instead more useful compute per watt increases the value of that compute

I buy that. Jevon's Paradox, sure.

> and we are not going to run out of economically useful things to do with it anytime soon on the demand side

This I don't buy. Not fully, at least. Whether or not there's demand for LLMs in some particular field is one thing, whether or not there is a sustainable business model to be built out of that demand is another thing entirely.

There is a staggering amount of money pouring into startups looking for novel use cases for LLM-based agents. As usual, 99% of them will fail, but those other 1% are going to have to look harder and harder to find a novel use case that can actually be served profitably.

First of all, there's only so many places where a chatbot is going to sell. But, that also seems to be the only interface anyone can come up with that allows a user to steer an agent through a long-running task reliably. I'd love to be proven wrong here.

Also, if current trends plateau and large datacenters are still needed for complex tasks, that would stimy growth of LLM usage across entire industries.

But, if present trends continue, then local inference will become feasible for most tasks. That would lower the barrier to entry across tons of heavily-regulated and/or cost-sensitive industries. But, widespread local inference will almost certainly come with a painful market correction centered around hyperscalers, which would itself dry up the pool for ventures into new markets.

Re: Nvidia's Risky Business

#112

Post by CorrectHorseBat »

tomaskafka wrote: post #96
YuechenLi wrote: post #68

Nvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute…

I had a hard time understanding why didn’t AMD make a better developer experience for this two years ago, and am now even more baffled that even with all the LLMs they still don’t seem to have moved a single inch, despite this probably being a tens of billions dollars worth feature.

Hardware companies are notoriously bad at software. The software they use sucks, the languages they use suck, the internal tooling software they write sucks.

They don't know what good developer experience is, how do you expect them to deliver it to other people?

Re: Nvidia's Risky Business

#113

Post by HDThoreaun »

bigyabai wrote: post #51
synergy20 wrote: post #42

Earlier quoted context omitted.

that undercuts their core business, so it will be a defensive play at most to fend off mac and amd's local inference offerings

I don't know how people can say this with a straight face. Nvidia was selling desktop-grade ARM SOCs before Apple Silicon was ever announced, specifically for edge robotics, computer vision and ML. The absolute fastest desktop Mac GPUs cannot beat an Nvidia laptop GPU in prefill or inference speeds. Apple Silicon is a non-entity for professional datacenter deployment and arguably unusable for frontier models at agent…

Every time nvidia takes its fab time and uses it to build anything other than datacenter chips it is losing money due to the massive markups the datacenter products have. Expanding their consumer offering means the datacenter backlog is going down which is very bad for their margins. Consumers will never pay 10-100x what it costs to fab something like datacenter users will.

Re: Nvidia's Risky Business

#114

Post by bigyabai »

coredog64 wrote: post #105
bigyabai wrote: post #71

Earlier quoted context omitted.

But people have been saying this about CUDA for twenty years , and we are not any closer to a replacement GPGPU paradigm today. The root comment in this thread was about Nvidia hedging their bet on lost AI market share. They recognize that a reduced pace in training and inference competition will undercut their business, but CUDA isn't a one-trick pony for LLMs alone. TPUs are - you can't even reuse the same architec…

> you can't even reuse the same architecture for training and inference AWS begs to differ. They originally split between `Trainium` and `Inferentia` but now support both with `Trainium`

I stand corrected, only Google's post-Ironwood TPUs have the split as well.

Nonetheless, TPU architectures are still a systolic array, and have their own limitations for scalability and flexibility. CUDA is no silver bullet, but it satisfies the demands of the edge and research customers very well.

Re: Nvidia's Risky Business

#115

Post by HDThoreaun »

Dardalus wrote: post #57

Tend to agree with Ben's thesis RE Demis and DeepMind not really being focused on the agentic coding race. That being said, it remains to be seen whether Sergey and Koray can inspire the foot soldiers in the same way that Sama and Dario do. I'm not too optimistic, and that's to say nothing of the fact that Google cannot possibly hope to compete with these other companies on potential employee upside.

I think the linked semi-analysis piece is about right. Google is more focused on protecting its 4.2 trillion dollar golden goose than pushing cutting edge which leads to a bureaucratic nightmare that researchers simply dont need to engage with when openAI/anthropic are offering even more money. Why waste time dealing with bureaucracy at google when you can be top dog at openAI or anthropic?

Re: Nvidia's Risky Business

#116

Post by schopra909 »

YuechenLi wrote: post #68

Nvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute…

I’m not entirely sure if local development will lead to Nvidia’s supremacy being challenged.

I think a simple reason why it’s been hard to unseat in Nvidia is first mover advantage. A lot more water has flown through Nvidia pipes than TPUs or AMDs chips for that matter.

TPUs and AMD chips aren’t priced cheaper than NVIDIA (at least for my purposes training models). So there hasn’t been an impetus for me to venture there and use those chips.

Anecdotally, folks I know who have tried using TPUs and AMD chips have hit more issues with the underlying drivers than with NVIDIA chips. That costs time and money to fix.

Eventually the other chips will go through enough iterations and stability will be reached

Re: Nvidia's Risky Business

#117

Post by compiler-guy »

mandeepj wrote: post #110
elictronic wrote: post #103

Earlier quoted context omitted.

Retraining a large high paid user base is often a non-starter. To put this in perspective, Boeing’s eventual retraining costs for all the pilots for the 737Max was around 5 billion dollars. Looking at software more specifically the Linux foundation reported based on software dev salaries in 2008 it would be 1.4 billion to only write the Linux kernel. Up until about 2023 there wasn’t enough money involved to have any…

Yet, Microsoft pulled it off with a new .NET Framework, and Apple with its new iOS SDK. There are many more examples besides those two.

Both of those were a vendor X electing to stop updating framework A in favor of framework B. That’s a high cost for vendor X’s users, but if their business depends on vendor X, they have no choice. Maybe they can switch to vendor Y at that point, but now you are switching both vendors and frameworks.

Replacing CUDA with another framework has much lower motivation. That advantages of the new framework must cover the switching costs and the risk of such a switch. All while CUDA continues to evolve and allow access to additional features.

Apple and Microsoft had something of a captive userbase. New vendor on the block trying to replace CUDA does not.

Re: Nvidia's Risky Business

#118

Post by bigyabai »

HDThoreaun wrote: post #113
bigyabai wrote: post #51

Earlier quoted context omitted.

I don't know how people can say this with a straight face. Nvidia was selling desktop-grade ARM SOCs before Apple Silicon was ever announced, specifically for edge robotics, computer vision and ML. The absolute fastest desktop Mac GPUs cannot beat an Nvidia laptop GPU in prefill or inference speeds. Apple Silicon is a non-entity for professional datacenter deployment and arguably unusable for frontier models at agent…

Every time nvidia takes its fab time and uses it to build anything other than datacenter chips it is losing money due to the massive markups the datacenter products have. Expanding their consumer offering means the datacenter backlog is going down which is very bad for their margins. Consumers will never pay 10-100x what it costs to fab something like datacenter users will.

In many cases it isn't Nvidia paying for the fab time. For instance, the Nintendo Switch 2 is basically a pure-play design and support product for Nvidia, while Nintendo negotiates with Samsung for the actual SOC prices. Their IP philosophy is closer to AMD's than Apple's, Nvidia has long roped in 3rd party manufacturers to mark up, integrate and sell their hardware.

Re: Nvidia's Risky Business

#119

Post by [deleted] »

georgemcbay wrote: post #49
2OEH8eoCRo0 wrote: post #43

Earlier quoted context omitted.

What's the danger? They slide back down to being just a gaming graphics card company with a $10 share price?

> What's the danger? They slide back down to being just a gaming graphics card company with a $10 share price? Nvidia dropping from being a $5 trillion company to a $242 billion company would be 1929 levels of bad. Global economy end of days stuff, especially since Nvidia can't crash that hard without a lot of other stuff crashing with it.

[deleted]

Re: Nvidia's Risky Business

#120

Post by acdha »

bigyabai wrote: post #71
coredog64 wrote: post #56

Earlier quoted context omitted.

> The more money Nvidia makes, the more motivated their competitors will be to get a piece of that market and the more customers will be looking for alternatives like the push into TPUs which the article discussed. Your margin is my opportunity - Jeff Bezos

But people have been saying this about CUDA for twenty years , and we are not any closer to a replacement GPGPU paradigm today. The root comment in this thread was about Nvidia hedging their bet on lost AI market share. They recognize that a reduced pace in training and inference competition will undercut their business, but CUDA isn't a one-trick pony for LLMs alone. TPUs are - you can't even reuse the same architec…

> But people have been saying this about CUDA for twenty years, and we are not any closer to a replacement GPGPU paradigm today.

How much money was in it for the first decade or so? I think AMD was asleep at the switch but e.g. Apple just did their own thing for the parts which they prioritized.

My understanding is also that Anthropic and OpenAI have also worked to decouple themselves so I think it’s likely that the CUDA moat is going to be less of a barrier than it used to be from the perspective of guaranteeing Nvidia profits.

Post reply on HN