Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Think About GPUs

jax-ml.github.io

Post reply on HN
Print view View original

91–100 of 127 posts

Re: How to Think About GPUs

#91

Post by einpoklum »

pklausler wrote: post #70
einpoklum wrote: post #40

Earlier quoted context omitted.

> It's not clear from the above what a "CUDA core" (singular) _is_ A CUDA core is basically a SIMD lane on an actual core on an NVIDIA GPUs. For a longer version of this answer: https://stackoverflow.com/a/48130362/1593077

So it's a "SIMD lane" that can itself perform actual SIMD instructions? I think you want a metaphor that doesn't also depend on its literal meaning.

> that can itself perform actual SIMD instructions?

Mostly, no; it can't really perform actual SIMD instructions itself. If you look at the SASS (the assembly language used on NVIDIA GPUs) I don't believe you'll see anything like that.

In high-level code, you do have expressions involving "vectorized types", which look like they would translate into SIMD instruction, but they 'serialize', at the single thread level.

There are exceptions to this though, like FP16 operations which might work on 2xFP16 32-bit registers, and other cases. But that is not the rule.

Re: How to Think About GPUs

#92

Post by Symmetry »

robbies wrote: post #84
hackrmn wrote: post #32

I find the piece, much like a lot of other documentation, "imprecise". Like most such efforts, it likely caters to a group of people expected to benefit from being explained what a GPU is, but it fumbles it terms, e.g. (the first image with burned-in text): > The "Warp Scheduler" is a SIMD vector unit like the TPU VPU with 32 lanes, called "CUDA Cores" It's not clear from the above what a "CUDA core" (singular) _is_…

I’m being earnest: what is an appropriate level of computer architecture knowledge? SIMD is 50 years old. From the resource intro: > Expected background: We’re going to assume you have a basic understanding of LLMs and the Transformer architecture but not necessarily how they operate at scale. I suppose this doesn’t require any knowledge about how computers work, but core CPU functionality seems…reasonable?

SIMD is quite old but the changes Nvidia made to call it SIMT and that they used as an excuse to call their vector lanes "cores" are quite a bit newer.

Re: How to Think About GPUs

#93

Post by pornel »

tormeh wrote: post #19

I find it very hard to justify investing time into learning something that's neither open source nor has multiple interchangeable vendors. Being good at using Nvidia chips sounds a lot like being an ABAP consultant or similar to me. I realize there's a lot of money to be made in the field right now, but IIUC historically this kind of thing has not been a great move.

There are two CUDAs – a hardware architecture, and a software stack for it.

The software is proprietary, and easy to ignore if you don't plan to write low-level optimizations for NVIDIA.

However, the hardware architecture is worth knowing. All GPUs work roughly the same way (especially on the compute side), and the CUDA architecture is still fundamentally the same as it was in 2007 (just with more of everything).

It dictates how shader languages and GPU abstractions work, regardless of whether you're using proprietary or open implementations. It's very helpful to understand peculiarities of thread scheduling, warps, different levels of private/shared memory, etc. There's a ridiculous amount of computing power available if you can make your algorithms fit the execution model.

Re: How to Think About GPUs

#94

Post by porridgeraisin »

adrian_b wrote: post #16
porridgeraisin wrote: post #4

Earlier quoted context omitted.

I was referring to this portion of TFA > CUDA cores are much more flexible than a TPU’s VPU: GPU CUDA cores use what is called a SIMT (Single Instruction Multiple Threads) programming model, compared to the TPU’s SIMD (Single Instruction Multiple Data) model.

This flexibility of CUDA is a software facility, which is independent of the hardware implementation. For any SIMD processor one can write a compiler that translates a program written for the SIMT programming model into SIMD instructions. For example, for the Intel/AMD CPUs with SSE4/AVX/AVX-512 ISAs, there exists a compiler of this kind (ispc: https://github.com/ispc/ispc ).

Thanks, I will look into that.

However, I'm still confused about the original statement. What I had thought was that

pre-volta GPUs, each thread in a warp has to execute in lock-step. Post-volta, they can all execute different instructions.

Obviously this is a surface level understanding. How do I reconcile this with what you wrote in the other comment and this one?

Re: How to Think About GPUs

#95

Post by j45 »

victor106 wrote: post #77
j45 wrote: post #66

Earlier quoted context omitted.

Tech is always like this. You move from one thing to the next. With your transferable skills, experience and thinking that is beyond one programming language. Even Apple is simply exporting to CUDA now.

> Even Apple is simply exporting to CUDA now. Really!!! Any resources you can share?

It’s one way but still something.

https://9to5mac.com/2025/07/15/apples-machine-learning-frame...

Re: How to Think About GPUs

#96

Post by j45 »

almostgotcaught wrote: post #82
j45 wrote: post #66

Earlier quoted context omitted.

Tech is always like this. You move from one thing to the next. With your transferable skills, experience and thinking that is beyond one programming language. Even Apple is simply exporting to CUDA now.

> Even Apple is simply exporting to CUDA now. This is like when journalists write clickbait article titles by omitting all qualifiers (eg "states banning fluoride" when it's only some states). One framework added a CUDA backend. You think all of Apple uses only one framework? Further what makes you think this even gets internal use?

I didnt say any of those things at all.

Only that Apple not only might use CUDA internally but made a public release available too.

CUDA seems to be a trigger word in this thread for some.

https://9to5mac.com/2025/07/15/apples-machine-learning-frame...

Re: How to Think About GPUs

#97

Post by NikolaNovak »

woooooo wrote: post #47
NikolaNovak wrote: post #45

Earlier quoted context omitted.

I mean, I'm in Toronto Canada, a fairly big city and market, and have an open seat for a couple of good senior Oracle DBAs pretty much constantly. The market may have reduced over decades but there's still more demand than supply. And the core DBA skills are transferable to other RDBMS as well. While I agree that some niche technologies are fleeting, it's perhaps not the best example :-)

That's actually interesting! My experience is different, especially compared to the late 90s and early 00s, most people avoid Oracle if they can. But yes, its always worth having someone who's job is to think about the database if it's your lynchpin.

I think it's famously said that 5% of IT is in the exciting new stuff that's on Hacker News front page, and 95% is in boring line-of-business, back office "enterprise" software that's as unglamorous as it is unavoidable :-). Even seemingly modern giants like Google or Amazon etc - check what their payroll and financial system is in the background.

And wait until I tell you about my Cobol open seats - on modern Linux on cloud VMs too! :-)

Re: How to Think About GPUs

#98

Post by almostgotcaught »

j45 wrote: post #96
almostgotcaught wrote: post #82

Earlier quoted context omitted.

> Even Apple is simply exporting to CUDA now. This is like when journalists write clickbait article titles by omitting all qualifiers (eg "states banning fluoride" when it's only some states). One framework added a CUDA backend. You think all of Apple uses only one framework? Further what makes you think this even gets internal use?

I didnt say any of those things at all. Only that Apple not only might use CUDA internally but made a public release available too. CUDA seems to be a trigger word in this thread for some. https://9to5mac.com/2025/07/15/apples-machine-learning-frame...

> I didnt say any of those things at all.

what does this sentence mean?

> Apple is simply exporting to CUDA now.

Re: How to Think About GPUs

#99

Post by ngcc_hk »

This is part 12 … the title seems to hint on how do one think about Gpu today … eg why llm comes about. Instead it is about cf with tpu? And then I note the part 12 … not sure what one should expect to jump in the middle of a whole series and what … well may stop and move on.

Re: How to Think About GPUs

#100

Post by tormeh »

saagarjha wrote: post #20
tormeh wrote: post #19

I find it very hard to justify investing time into learning something that's neither open source nor has multiple interchangeable vendors. Being good at using Nvidia chips sounds a lot like being an ABAP consultant or similar to me. I realize there's a lot of money to be made in the field right now, but IIUC historically this kind of thing has not been a great move.

Sure, but you can make money in the field and retire faster than it becomes irrelevant. FWIW none of the ideas here are novel or nontransferable–it's just the specific design that is proprietary. Understanding how to do an AllReduce has been of theoretical interest for decades and will probably remain worth doing far into the future.

Only in Silicon Valley. But if you can, definitely do.
Post reply on HN