Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Think About GPUs

jax-ml.github.io

Post reply on HN
Print view View original

61–70 of 127 posts

Re: How to Think About GPUs

#61

Post by KeplerBoy »

einpoklum wrote: post #43

We should remember that these structural diagrams are _not_ necessarily what NVIDIA actually has as hardware. They carefully avoid guaranteeing that any of the entities or blocks you see in the diagrams actually _exist_. It is still just a mental model NVIDIA offers for us to think about their GPUs, and more specifically the SMs, rather than a simplified circuit layout. For example, we don't know how many actual func…

Interesting perspective. Aren't SMs basically blocked while running tensor core operations, which might hint that it's the same FPUs doing the work after all?

Re: How to Think About GPUs

#62

Post by dahart »

gregorygoc wrote: post #9

It’s mind boggling why this resource has not been provided by NVIDIA yet. It reached the point that 3rd parties reverse engineer and summarize NV hardware to a point it becomes an actually useful mental model. What are the actual incentives at NVIDIA? If it’s all about marketing they’re doing great, but I have some doubts about engineering culture.

What makes you think that? It appears most of this material came straight out of NVIDIA documentation. What do you think is missing? I just checked and found the H100 diagram for example is copied (without being correctly attributed) from the H100 whitepaper: https://resources.nvidia.com/en-us-hopper-architecture/nvidi...

Much of the info on compute and bandwidth is from that and other architecture whitepapers, as well as the CUDA C++ programming guide, which covers a lot of what this article shares, in particular chapters 5, 6, and 7. https://docs.nvidia.com/cuda/cuda-c-programming-guide/

There’s plenty of value in third parties distilling and having short form versions, and of writing their own takes on this, but this article wouldn’t have been possible without NVIDIA’s docs, so the speculation, FUD and shade is perhaps unjustified.

Re: How to Think About GPUs

#63

Post by j45 »

tormeh wrote: post #19

I find it very hard to justify investing time into learning something that's neither open source nor has multiple interchangeable vendors. Being good at using Nvidia chips sounds a lot like being an ABAP consultant or similar to me. I realize there's a lot of money to be made in the field right now, but IIUC historically this kind of thing has not been a great move.

Nvidia also trotted along with a low share price for a long time financing and supporting what they believed in.

When cuda rose to prominence were there any viable alternatives?

Re: How to Think About GPUs

#64

Post by j45 »

Q6T46nT668w6i3m wrote: post #50
physicsguy wrote: post #28

Earlier quoted context omitted.

It really isn't that hard to pivot. It's worth saying that if you were already writing OpenMP and MPI code then learning CUDA wasn't particularly difficult to get started, and learning to write more performant CUDA code would also help you write faster CPU bound code. It's an evolution of existing models of compute, not a revolution.

I agree that “learning CUDA wasn’t particularly difficult to get started,” there are Grand Canyon sized chasms between CUDA and its alternatives when attempting to crank performance.

It will improve for sure but this shouldn’t be downplayed.

Re: How to Think About GPUs

#65

Post by j45 »

hackrmn wrote: post #34
tormeh wrote: post #19

I find it very hard to justify investing time into learning something that's neither open source nor has multiple interchangeable vendors. Being good at using Nvidia chips sounds a lot like being an ABAP consultant or similar to me. I realize there's a lot of money to be made in the field right now, but IIUC historically this kind of thing has not been a great move.

I grew up learning programming on a genuine IBM PC running MS-DOS, neither of which was FOSS but taught me plenty that I routinely rely on today in one form or another.

Very true.

Stories of exploring DOS often ended up at hex editing and assembly.

Best to learn with whatever options are accessible, plenty is transferable.

While there is an embarrassment of options to learn from or with today the greatest gaffe can be overlooking learning.

Re: How to Think About GPUs

#66

Post by j45 »

saagarjha wrote: post #20
tormeh wrote: post #19

I find it very hard to justify investing time into learning something that's neither open source nor has multiple interchangeable vendors. Being good at using Nvidia chips sounds a lot like being an ABAP consultant or similar to me. I realize there's a lot of money to be made in the field right now, but IIUC historically this kind of thing has not been a great move.

Sure, but you can make money in the field and retire faster than it becomes irrelevant. FWIW none of the ideas here are novel or nontransferable–it's just the specific design that is proprietary. Understanding how to do an AllReduce has been of theoretical interest for decades and will probably remain worth doing far into the future.

Tech is always like this.

You move from one thing to the next.

With your transferable skills, experience and thinking that is beyond one programming language.

Even Apple is simply exporting to CUDA now.

Re: How to Think About GPUs

#67

Post by j45 »

Philpax wrote: post #24
dotancohen wrote: post #22

Earlier quoted context omitted.

I've been hearing that for over a decade. I can't even name off hand any CUDA competitors, none of them are likely to gain enough traction to upset CUDA in the coming decade.

Hence the "if" :-) ROCm is getting some adoption, especially as some of the world's largest public supercomputers have AMD GPUs. Some of this is also being solved by working at a different abstraction layer; you can sometimes be ignorant to the hardware you're running on with PyTorch. It's still leaky, but it's something.

Waiting just adds more dust to the skills pile.

Programming languages are groups of syntax.

Re: How to Think About GPUs

#68

Post by j45 »

moralestapia wrote: post #39
tormeh wrote: post #19

I find it very hard to justify investing time into learning something that's neither open source nor has multiple interchangeable vendors. Being good at using Nvidia chips sounds a lot like being an ABAP consultant or similar to me. I realize there's a lot of money to be made in the field right now, but IIUC historically this kind of thing has not been a great move.

It's money. You would do it for money.

Generally, earning an honest living seems to be a requirement of this world that individual words and beliefs won’t change.

Work keeps us humble enough to be open to learn.

Re: How to Think About GPUs

#70

Post by pklausler »

einpoklum wrote: post #40
hackrmn wrote: post #32

I find the piece, much like a lot of other documentation, "imprecise". Like most such efforts, it likely caters to a group of people expected to benefit from being explained what a GPU is, but it fumbles it terms, e.g. (the first image with burned-in text): > The "Warp Scheduler" is a SIMD vector unit like the TPU VPU with 32 lanes, called "CUDA Cores" It's not clear from the above what a "CUDA core" (singular) _is_…

> It's not clear from the above what a "CUDA core" (singular) _is_ A CUDA core is basically a SIMD lane on an actual core on an NVIDIA GPUs. For a longer version of this answer: https://stackoverflow.com/a/48130362/1593077

So it's a "SIMD lane" that can itself perform actual SIMD instructions?

I think you want a metaphor that doesn't also depend on its literal meaning.

Post reply on HN