Earlier quoted context omitted.
Nvidia sell iot boards with unified architecture. Would not be shocked if they launch pc/laptop/server boards at some point.
Unified memory DGX Spark and RTX Spark laptops are already a thing :)
As is AMD Strix Point
Nvidia's Risky Business
81–90 of 185 posts
Re: Nvidia's Risky Business
#82Earlier quoted context omitted.
I think efficiency is unlikely to result in lower demand for compute, instead more useful compute per watt increases the value of that compute; and we are not going to run out of economically useful things to do with it anytime soon on the demand side. The harder thing to forecast for me is if we hit a wall on increasing efficiency, either on the model weights side or silicon side, with current approaches. If we have…
When efficiency reaches the point where local models on consumer hardware are good enough, demand for cloud tokens could rapidly shrink.
It could be quite a while before we reach that point though. 5+ years easily.
I've been keenly interested in the ability to run local models, but the hardware is just not there. Consumer RAM speeds and capacity will have to significantly increase before local models will be able to perform as well as even the lowest end GPT-5.6 Luna model.
This is on the backdrop of RAM becoming prohibitively expensive. And without the speed and quantity of RAM, it becomes impossible to generate tokens at interactive speeds, regardless of model. There is a fundamental dependency between calculating all of the active params with the given RAM speed.
Even with a model that has been quantized all the way down to Q4, the DGX/RTX Spark chip with 128GB of RAM can only generate ~18 tokens/sec for a MoE model with only 30B active parameters. There haven't been any broadly useful models below 30B active parameters. And that is for a $5000+ piece of hardware that will be one of the best for running on-device models.
I really want to buy instead of rent my AI, but the economics are truly terrible.
Re: Nvidia's Risky Business
#83For awhile I've found two things hard to square, that the hardware and software making up current gen AI will bring us to a socioeconomic singularity, and the reality the thing they're mostly trying to emulate is a few pounds of meat and fat running on tens of watts equivalent. On one hand the current AIs are obviously super human in some tasks, get completely dunked on in others by far simpler organisms. My cat can…
Another interesting discrepancy is that people think current GPUs are maybe capable of running AGI but they still can barely manage photorealistic rendering of a single room in realtime, or simulate something like a shirt thrown into a pile of laundry. They can generate a video of it based on millions of existing videos, but not do a real simulation of light and physics in realtime.
Re: Nvidia's Risky Business
#84Nvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute…
That's really interesting. I have no experience writing anything that involves GPUs/TPUs, but over the years I've consistently read that CUDA is the "real moat" of Nvidia, which I never totally believed, but the way you describe makes it seem like it's not actually a moat in the slightest. It just happens to be an ecosystem associated with hardware that is not only considered the gold standard but happens to be more open than potential competition. Could it be that Nvidia has been on top because none of the competition has actually tried kicking them where it hurts?
Re: Nvidia's Risky Business
#85IMO focusing on the hyperscalers is kind of misleading. Yes, for programmers and tech companies AI is kinda boring now, but AI integration in general is still kind of uncharted territory. There are so many small companies and individuals just getting started with AI today and I believe a large the customer base (and revenue) is still untapped. Hell, I’m discovering new use cases regularly still and the average misman…
This is what so many people on HN and the market are constantly missing. Jason Kottke almost didn't found his blog in 1998, famously quoted as saying: "I thought I was too late, that no one would be interested." Needless to say, the internet was a tiny joke in 1998 compared to what it is now. We are just barely scratching the surface of what's possible with AI, both in terms of the leading edge and in the 'torso' of…
I've always marveled at how one can pick any year since the internet went mainstream and in that year people thought, "Oh my goodness, the internet is amazing!"
Then, move five years forward from that year and look back. In every case, people think, "Hah! The internet was so simple then!"
In 2031, I suspect we'll say the same about 2026.
Re: Nvidia's Risky Business
#86Ben is wrong; demand for compute, aka revenue backlogs, is mythical and will collapse, simply because of two reasons : 1. Circular investment/spending. 2. Too much capital in the system, so returns cannot be hit regardless because the barrier is too high. (Evidence being every capital cycle in history)
I think more interesting take here would be WHEN this will happen. I don't think Ben, or anyone else, thinks we wont have some sort of correction or stabilization in supply/demand (he has said as much) But when will that occur? 2 months? 2 years? 20 years?
Re: Nvidia's Risky Business
#87Nvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute…
Now with LLMs why a programming framework is a moat?
Re: Nvidia's Risky Business
#88Earlier quoted context omitted.
What makes this insanely hard to predict is that the compute needed for the same quality output has roughly gone down 90% every 18 months for ~5 years. 1) We don't know how long that trend will continue, but you do know where to look for when it may end (if smaller sized models continue to compress the knowledge effectively of larger models). 2) We don't know when the appetite for higher cost models might go down and…
I think efficiency is unlikely to result in lower demand for compute, instead more useful compute per watt increases the value of that compute; and we are not going to run out of economically useful things to do with it anytime soon on the demand side. The harder thing to forecast for me is if we hit a wall on increasing efficiency, either on the model weights side or silicon side, with current approaches. If we have…
> think efficiency is unlikely to result in lower demand for compute
You can't save yourself rich.
Re: Nvidia's Risky Business
#89Ever free newsletter and talking head spouts narratives like this free. If you want something that quantifies and gives actionable information, you must do it yourself or pay for it. What are your below $2000/month sources for good analysis?
[deleted]
Re: Nvidia's Risky Business
#90Nvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute…
That's really interesting. I have no experience writing anything that involves GPUs/TPUs, but over the years I've consistently read that CUDA is the "real moat" of Nvidia, which I never totally believed, but the way you describe makes it seem like it's not actually a moat in the slightest. It just happens to be an ecosystem associated with hardware that is not only considered the gold standard but happens to be more…
Software has always been the moat but for some reason it's always hamstrung by upper management. The latest of the frenzies being replacing sane (or whatever we have) of development practices with AI-slop.
Management likes it because it removes software developers from the loop.