Live data from Hacker News

How to Think About GPUs

jax-ml.github.io

101–110 of 127 posts

Re: How to Think About GPUs

#101
post #32

I find the piece, much like a lot of other documentation, "imprecise". Like most such efforts, it likely caters to a group of people expected to benefit from being explained what a GPU is, but it fumbles it terms, e.g. (the first image with burned-in text): > The "Warp Scheduler" is a SIMD vector unit like the TPU VPU with 32 lanes, called "CUDA Cores" It's not clear from the above what a "CUDA core" (singular) _is_…

My recursive brain got a chuckle out of wondering about "imprecise" being in quotes. I found the quotes made the meaning a touch...imprecise.

While I can understand the imprecise point, I found myself very impressed by the quality of the writing. I don't envy making digestible prose about the differences between GPUs and TPUs.

Re: How to Think About GPUs

#102

Why haven't Nvidia developed a TPU yet?

Meaning what? Something less flexible? Less CUDA cores and more Tensor Cores?

The majority of NVidia's profits (almost 90%) do come from data center, most of which is going to be neural net acceleration, and I'd have to assume that they have optimized their data center products to maximize performance for typical customer workloads.

I'm sure that Microsoft would provide feedback to Nvidia if they felt changes were needed to better compete with Google in the cloud compute market.

Re: How to Think About GPUs

#103
post #19

I find it very hard to justify investing time into learning something that's neither open source nor has multiple interchangeable vendors. Being good at using Nvidia chips sounds a lot like being an ABAP consultant or similar to me. I realize there's a lot of money to be made in the field right now, but IIUC historically this kind of thing has not been a great move.

You can write software for the hardware in a cross-compiled language like Triton. The hardware reality stays the same, a company like Cerebras might have the superior architecture, but you have server rooms filled with H100, A100, and MI300s whether you believe in the hardware or not.

Re: How to Think About GPUs

#104
post #32

I find the piece, much like a lot of other documentation, "imprecise". Like most such efforts, it likely caters to a group of people expected to benefit from being explained what a GPU is, but it fumbles it terms, e.g. (the first image with burned-in text): > The "Warp Scheduler" is a SIMD vector unit like the TPU VPU with 32 lanes, called "CUDA Cores" It's not clear from the above what a "CUDA core" (singular) _is_…

Shamelessly responding as the author. I (mostly) agree with you here. > please be surgically precise with your terms There's always a tension between precision in every explanation and the "moral" truth. I can say "a SIMD (Single Instruction Multiple Data) vector unit like the TPU VPU with 32 ALUs (SIMD lanes) which NVIDIA calls CUDA Cores", which starts to get unwieldy and even then leaves terms like vector units un…

I feel you handle this balance quite gracefully, to the point where I was impressed at your handling of the issue while reading and before checking the comments section. I don't know why the idea of something being called something by marketing or documentation (names which one must, strategically accept and internalize) but fundamentally and functionally being better described with other language isn't clearer (which is more useful, so also needed) to the grandparent poster. You want people to be aware of both and explain both without dwelling or getting caught on it, it struck me as an artful choice.

Re: How to Think About GPUs

#105
post #96

Earlier quoted context omitted.

I didnt say any of those things at all. Only that Apple not only might use CUDA internally but made a public release available too. CUDA seems to be a trigger word in this thread for some. https://9to5mac.com/2025/07/15/apples-machine-learning-frame...

> I didnt say any of those things at all. what does this sentence mean? > Apple is simply exporting to CUDA now.

My takeaway was definitely not that all of Apple is using only one framework.

Re: How to Think About GPUs

#106
post #32

I find the piece, much like a lot of other documentation, "imprecise". Like most such efforts, it likely caters to a group of people expected to benefit from being explained what a GPU is, but it fumbles it terms, e.g. (the first image with burned-in text): > The "Warp Scheduler" is a SIMD vector unit like the TPU VPU with 32 lanes, called "CUDA Cores" It's not clear from the above what a "CUDA core" (singular) _is_…

Shamelessly responding as the author. I (mostly) agree with you here. > please be surgically precise with your terms There's always a tension between precision in every explanation and the "moral" truth. I can say "a SIMD (Single Instruction Multiple Data) vector unit like the TPU VPU with 32 ALUs (SIMD lanes) which NVIDIA calls CUDA Cores", which starts to get unwieldy and even then leaves terms like vector units un…

I appreciate your response. I made a point of not revising my comment after posting it and finding in a subsequent parable the following, quoting:

> Each SM is broken up into 4 identical quadrants, which NVIDIA calls SM subpartitions, each containing a Tensor Core, 16k 32-bit registers, and a SIMD/SIMT vector arithmetic unit called a Warp Scheduler, whose lanes (ALUs) NVIDIA calls CUDA Cores.

And right after:

> CUDA Cores: each subpartition contains a set of ALUs called CUDA Cores that do SIMD/SIMT vector arithmetic.

So, to your defense and my shame -- you *did* do better than I was able to infer from first glance. And I can take absolutely no issue with a piece elaborating on originally "vague" sentence later on -- we need to read top to bottom, after all.

Much of the difficulty with laying out knowledge in written word is inherent constraints like choosing between deferring detail to "further down" at the expense of giving the "bird's eye view". I mean there is a reason writing is hard, technical writing perhaps more so, in a way. You're doing much better than a lot of other stuff I've had to learn with, so I can only thank you to have done as much as you already have.

To be more constructive still, I agree the border between clarity and utility isn't always clearly drawn. But I think you can think of it as a service to your readers -- go with precision I say -- if you really presuppose the reader should know SIMD, chances are they are able to grok a new definition like "SIMD lane" if you define it _once_ and _well_. You don't need to be "unwieldy" in repetition -- the first time may be hard but you only need to do it once.

I am rambling. I do believe there are worse and better ways to impart knowledge of the kind in writing, but I too obviously don't have the answers, so my criticism was in part inconstructive, just a sheer outcry of mild frustration once I started conflating things from the get go but before I decided to give it a more thorough read.

One last thing though: I always like when a follow-up article starts with a preamble along of "In the previous part of the series..." so new visitors can simultaneously become aware there's prior knowledge that may be assumed, _and_ navigate their way to desired point in the series, all the way to the start perhaps. That frees you from e.g. wanting to annotate abbreviations in every part, if you want to avoid doing that.

Re: How to Think About GPUs

#107

Earlier quoted context omitted.

So it's a "SIMD lane" that can itself perform actual SIMD instructions? I think you want a metaphor that doesn't also depend on its literal meaning.

> that can itself perform actual SIMD instructions? Mostly, no; it can't really perform actual SIMD instructions itself. If you look at the SASS (the assembly language used on NVIDIA GPUs) I don't believe you'll see anything like that. In high-level code, you do have expressions involving "vectorized types", which look like they would translate into SIMD instruction, but they 'serialize', at the single thread level.…

Please see https://docs.nvidia.com/cuda/parallel-thread-execution/index....

Re: How to Think About GPUs

#109

Earlier quoted context omitted.

> I didnt say any of those things at all. what does this sentence mean? > Apple is simply exporting to CUDA now.

My takeaway was definitely not that all of Apple is using only one framework.

then please enlighten me: what does the sentence mean?

Re: How to Think About GPUs

#110
post #32

I find the piece, much like a lot of other documentation, "imprecise". Like most such efforts, it likely caters to a group of people expected to benefit from being explained what a GPU is, but it fumbles it terms, e.g. (the first image with burned-in text): > The "Warp Scheduler" is a SIMD vector unit like the TPU VPU with 32 lanes, called "CUDA Cores" It's not clear from the above what a "CUDA core" (singular) _is_…

Shamelessly responding as the author. I (mostly) agree with you here. > please be surgically precise with your terms There's always a tension between precision in every explanation and the "moral" truth. I can say "a SIMD (Single Instruction Multiple Data) vector unit like the TPU VPU with 32 ALUs (SIMD lanes) which NVIDIA calls CUDA Cores", which starts to get unwieldy and even then leaves terms like vector units un…

I often put requirements at the top of article

> This article assumes you've read [this] and [this] and understand > [this topic] and [this topic too]

I'm not sure that's helpful, and, I don't put everything. Those links might also have further links saying you need X, Y, and Z. But at least there is a trail on where to start

Post reply on HN