Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia's Risky Business

stratechery.com

Post reply on HN
Print view View original

141–150 of 185 posts

Re: Nvidia's Risky Business

#141

Post by mycall »

musebox35 wrote: post #80
YuechenLi wrote: post #68

Nvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute…

The biggest advantage of tpus is the high bandwidth fiber optic interconnect between them that allows distributed computing on pods with thousands of tpus and the co-design of cooling systems that go with their racks. I do not think that we will see personal tpus any time soon.

My Pixel 9 has a personal tpu

Re: Nvidia's Risky Business

#142

Post by cepp »

ravenstine wrote: post #84
YuechenLi wrote: post #68

Nvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute…

That's really interesting. I have no experience writing anything that involves GPUs/TPUs, but over the years I've consistently read that CUDA is the "real moat" of Nvidia, which I never totally believed, but the way you describe makes it seem like it's not actually a moat in the slightest. It just happens to be an ecosystem associated with hardware that is not only considered the gold standard but happens to be more…

One of the rarely-mentioned value adds Nvidia provides is nccl[1] which makes multi-node networking and topology essentially plug and play. The other players have since caught on [2][3] and are working hard to catch up but I'd say networking is a real moat.

[1] https://developer.nvidia.com/nccl [2] https://pytorch.org/blog/torchcomms/ [3] https://rocm.docs.amd.com/projects/rccl/en/latest/

Re: Nvidia's Risky Business

#143

Post by bigyabai »

acdha wrote: post #120
bigyabai wrote: post #71

Earlier quoted context omitted.

But people have been saying this about CUDA for twenty years , and we are not any closer to a replacement GPGPU paradigm today. The root comment in this thread was about Nvidia hedging their bet on lost AI market share. They recognize that a reduced pace in training and inference competition will undercut their business, but CUDA isn't a one-trick pony for LLMs alone. TPUs are - you can't even reuse the same architec…

> But people have been saying this about CUDA for twenty years, and we are not any closer to a replacement GPGPU paradigm today. How much money was in it for the first decade or so? I think AMD was asleep at the switch but e.g. Apple just did their own thing for the parts which they prioritized. My understanding is also that Anthropic and OpenAI have also worked to decouple themselves so I think it’s likely that the…

Money wasn't really the problem. Apple pulled OpenCL together pro-bono, and worked with Khronos to find willing industry stakeholders that would oppose Nvidia. OpenCL needed hardware standardization though, and nobody wanted to design or implement on a scalable GPGPU architecture like CUDA had. AMD and Apple both bet big on raster efficiency, which turned out to be a terrible play when Nvidia was already putting dedicated ray tracing and tensor hardware into their GPUs. They both bet the farm against each other, and only Nvidia won.

Once Apple fully left Khronos, AMD played the smartest card they had; they architecturally split RDNA and CDNA into separate product lines, so they could optimize them independently. This staunched the bleeding, and gave AMD a datacenter presence that Apple Silicon could only dream of. Still not a scalable architecture, but better than nothing.

Re: Nvidia's Risky Business

#144

Post by kevstev »

YuechenLi wrote: post #132
kevstev wrote: post #125

Earlier quoted context omitted.

I think OP is overstating it a bit tbh. So Nvidia has the market for the hardware, which helps, but there is literally no alternative to CUDA. Nvidia keeps it a scalpel for skilled users, its not super easy to use, but unlocks orders of more magnitudes of power for the use cases it excels at vs CPUs. I don't have access to anything like it in the Apple ecosystem. AMD has had years to try and counter it, but just has…

I think there is a misconception here: CUDA is not even close to being C for assembly used by Nvidia's GPUs, PTX is, and it uses JIT to compile to Nvidia's GPU assembly, SASS. It honestly easier just to have LLMs write PTX directly than to go through CUDA C/C++ at all. Again, "CUDA" isn't a programming language, it stands for Compute Unified Device Architecture; "C/C++ for CUDA" are the high-level languages that comp…

My point with the c comparison is that it often feels like a very thin layer that still requires you to know a lot about what's going on under the hood. It was more to give a view in what it's like to work with the lib/api.

Maybe I wasn't being precise enough with my language for this forum, and also my last hands on experience with it was roughly 6 years ago, maybe it's gotten better. But it was much less (and forgive the imprecision!) python/pytorch-like where you say hey take this big blob of data and just slice and dice it on your many cores, and more like ok, here is the data, let's cudamemcopy it in these size chunks over to the gpu itself, to be used by this block of threads and run these commands (kernel in cudaspeak) on it. Much more painstaking and micromanagey of the resources.

Pytorch IMHO feels like a proper abstracted API that hides the details and lets you just unleash the fury at the cost of some efficiency, while the cuda api itself, similar to working with C, forces you to really think about the low level details. I have a heavy backend and systems development background, and while it wasn't really intimidating to me, it was like wow you really have to have a deep working knowledge of how these things work and it felt like a step back in time IMHO.

I doubt that's going to satisfy you but I think it gives a clearer picture of what using cuda is like if you typically use higher level languages and haven't touched C since college.

Re: Nvidia's Risky Business

#146

Post by akoboldfrying »

YuechenLi wrote: post #68

Nvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute…

Interesting take on Google's TPUs. What I've previously heard (and still believe) is that Google's decision to only rent out, never sell, their TPUs is a deliberate and savvy strategy for bolstering GCP, which will work provided that TPUs are able to actually compete with other hardware (in practice meaning Nvidia). A few months ago there was some discussion on HN comparing them, and I think the verdict at the time was that their latest-gen TPUs win on compute-per-Joule for LLM-type workloads by quite a margin, which I think is huge for those who want to run LLMs at scale.

Re: Nvidia's Risky Business

#147

Post by thelastgallon »

I wonder why Google doesn't create an open-source CUDA alternative. Google released Kubernetes to stay relevant/competitive in the cloud wars, they were a distant third. They now have an opportunity to create an open source industry standard.

Or the companies spending trillions of dollars can do a Manhattan Project (Or X-Prize) and let a thousand startups work on it. One will succeed. Between Google, Amazon, FB, Microsoft, Apple, AMD, Qualcomm, Intel (and dozens of other companies) there is enough economic incentive to do it. Also, isn't this what AI is supposed to be extremely good at, CUDA experts can continue to write CUDA (without having to learn anything new), a translation layer will rewrite it. If software can be one-shot from markdown files, this can't be impossible.

Re: Nvidia's Risky Business

#148

Post by Altaba »

motoxpro wrote: post #137
Altaba wrote: post #106

Earlier quoted context omitted.

well its typically when it becomes clear that the private equity firms taking the risk decide they can not get the returns they need, forcing the backstoppers such as Nvidia to take that burden, and the whole ecosystem collapses.

I guess my point is that until that collapse, returns are very very very good. This can (and probably will) go on for a decent amount of time more, regardless of the inevitable things you point out. My guess is at least another 2 years, as most people don't use AI yet, or maybe more precisely, AI is not used in the underlying workflows (which are invisible to the consumer) that make up most people's jobs. Who knows i…

Yes, agreed, it is timing. I think a sign we are getting closer is the new equity issuances, which kind of are leveraging the current environment and the retail excitement.

Re: Nvidia's Risky Business

#149

Post by YuechenLi »

kevstev wrote: post #144
YuechenLi wrote: post #132

Earlier quoted context omitted.

I think there is a misconception here: CUDA is not even close to being C for assembly used by Nvidia's GPUs, PTX is, and it uses JIT to compile to Nvidia's GPU assembly, SASS. It honestly easier just to have LLMs write PTX directly than to go through CUDA C/C++ at all. Again, "CUDA" isn't a programming language, it stands for Compute Unified Device Architecture; "C/C++ for CUDA" are the high-level languages that comp…

My point with the c comparison is that it often feels like a very thin layer that still requires you to know a lot about what's going on under the hood. It was more to give a view in what it's like to work with the lib/api. Maybe I wasn't being precise enough with my language for this forum, and also my last hands on experience with it was roughly 6 years ago, maybe it's gotten better. But it was much less (and forgi…

I think you are pretty precise and your comment is appreciated, so I think I need to clarify my wording as well. I think a lot of CUDA C/C++'s difficulties are self-inflicted because they conflate two things into what they call "kernels": runtime and shaders. Fundamentally, GPU programming for compute is easier than either graphics programming as well as CPU programming because you are limited on what you can do on the GPU, memory allocation is static because the matrix size doesn't really change dynamically during runtime and branching behaviors are generally to be avoided for GPU compute.

It's pretty heretical for me to say this, but a lot of GPU compute complexity that Nvidia is doing in CUDA is unnecessary and is by the simple fact that to do anything meaningful you have to either use their library or handle allocation/scheduling yourself. Imagine if JavaScript required you to handroll part of the V8/Node's JIT compiler, allocator and scheduler yourself every time you just want to make a webpage, that is essentially what CUDA is doing.

The actual "program" that runs on the GPU, the compute shaders in PTX/SPIR-V, are very low level but pretty straight forward once you get down to it.

Re: Nvidia's Risky Business

#150

Post by gr_norm »

ijidak wrote: post #121
YuechenLi wrote: post #68

Nvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute…

Genuine question. Given that LLMs are supposed to allow us to rewrite anything, and I am an LLM believer, what I don't understand is: how does CUDA continue to be a moat in a world where LLMs can rewrite entire software development stacks? If NVIDIA is right about AI, isn't this same technology going to erode the software side of this same software moat?

Indeed. Now, given that CUDA is apparently not being usurped, update your priors.
Post reply on HN