Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Think About GPUs

jax-ml.github.io

Post reply on HN
Print view View original

41–50 of 127 posts

Re: How to Think About GPUs

#41

Post by namibj »

tucnak wrote: post #17

This post is a great illustration why TPU's lend more nicely towards homogenous computing: yes, there's systolic array limitations (not good for sparsity) but all things considering, bandwidth doesn't change as your cluster ever so larger grows. It's a shame Google is not interested in selling this hardware: if they were available, it would open the door to compute-in-network capabilities far beyond what's currently…

Do TPUs allow having a variable array dimension at somewhat inner nesting level of the loop structure yet? Like, where you load expensive (bandwidth-heavy) data in from HBM, process a variable-length array with this, then stow away/accumulate into a fixed-size vector?

Last I looked they would require the host to synthesize a suitable instruction stream for this on-the-fly with no existing tooling to do so efficiently.

An example where this would be relevant would be LLM inference prefill stage with (activated) MoA expert count on the order of — to a small integer smaller than — the prompt length, where you'd want to only load needed experts and only load each one at most once per layer.

Re: How to Think About GPUs

#42

Post by woooooo »

sigbottle wrote: post #36
woooooo wrote: post #29

Earlier quoted context omitted.

It's a useful bit of caution to remember transferrable fundamentals, I remember when Oracle wizards were in high demand.

There are tons of ML compilers right now, FlashAttention brought back the cache-aware model to parallel programming, Moore's law hit is limit and heterogeneous hardware is taking taking off. Just some fundamentals I can think of off the top of my head. I'm surprised people saying that the lower level systems/hardware stuff are untransferable. These things are used everywhere . If anything, it's the AI itself that's p…

Im actually doing a ton of research in the area myself, the caution was against becoming an Nvidia expert narrowly rather than a general low level programmer with Nvidia skills included.

Re: How to Think About GPUs

#43

Post by einpoklum »

We should remember that these structural diagrams are _not_ necessarily what NVIDIA actually has as hardware. They carefully avoid guaranteeing that any of the entities or blocks you see in the diagrams actually _exist_. It is still just a mental model NVIDIA offers for us to think about their GPUs, and more specifically the SMs, rather than a simplified circuit layout.

For example, we don't know how many actual functional units an SM has; we don't know if the "tensor core" even _exists_ as a piece of hardware, or whether there's just some kind of orchestration of other functional units; and IIRC we don't know what exactly happens at the sub-warp level w.r.t. issuing and such.

Re: How to Think About GPUs

#44

Post by sdenton4 »

dotancohen wrote: post #22
Philpax wrote: post #21

Earlier quoted context omitted.

There's more in common with other GPU architectures than there are differences, so a CUDA consultant should be able to pivot if/when the other players are a going concern. It's more about the mindset than the specifics.

I've been hearing that for over a decade. I can't even name off hand any CUDA competitors, none of them are likely to gain enough traction to upset CUDA in the coming decade.

...Well, the article compares GPUs to tpus, made by a competitor you probably know the name of...

Re: How to Think About GPUs

#45

Post by NikolaNovak »

woooooo wrote: post #29
qwertox wrote: post #26

Earlier quoted context omitted.

It's a valid point of view, but I don't see the value in sharing it. There are enough people for who it's worth it, even if just for tinkering, and I'm sure you are aware of that. It reads a bit like "You shouldn't use it because..." Learning about Nvidia GPUs will teach you a lot about other GPUs as well, and there are a lot of tutorials about the former, so why not use it if it interests you?

It's a useful bit of caution to remember transferrable fundamentals, I remember when Oracle wizards were in high demand.

I mean, I'm in Toronto Canada, a fairly big city and market, and have an open seat for a couple of good senior Oracle DBAs pretty much constantly. The market may have reduced over decades but there's still more demand than supply. And the core DBA skills are transferable to other RDBMS as well. While I agree that some niche technologies are fleeting, it's perhaps not the best example :-)

Re: How to Think About GPUs

#46

Post by evertedsphere »

hackrmn wrote: post #32

I find the piece, much like a lot of other documentation, "imprecise". Like most such efforts, it likely caters to a group of people expected to benefit from being explained what a GPU is, but it fumbles it terms, e.g. (the first image with burned-in text): > The "Warp Scheduler" is a SIMD vector unit like the TPU VPU with 32 lanes, called "CUDA Cores" It's not clear from the above what a "CUDA core" (singular) _is_…

https://cloud.google.com/tpu/docs/system-architecture-tpu-vm

should have most of it

Re: How to Think About GPUs

#47

Post by woooooo »

NikolaNovak wrote: post #45
woooooo wrote: post #29

Earlier quoted context omitted.

It's a useful bit of caution to remember transferrable fundamentals, I remember when Oracle wizards were in high demand.

I mean, I'm in Toronto Canada, a fairly big city and market, and have an open seat for a couple of good senior Oracle DBAs pretty much constantly. The market may have reduced over decades but there's still more demand than supply. And the core DBA skills are transferable to other RDBMS as well. While I agree that some niche technologies are fleeting, it's perhaps not the best example :-)

That's actually interesting! My experience is different, especially compared to the late 90s and early 00s, most people avoid Oracle if they can. But yes, its always worth having someone who's job is to think about the database if it's your lynchpin.

Re: How to Think About GPUs

#48

Post by radarsat1 »

tomhow wrote: post #12

Discussion of original series: How to scale your model: A systems view of LLMs on TPUs - https://news.ycombinator.com/item?id=42936910 - Feb 2025 (30 comments)

A comment from there:

> There are plans to release a PDF version; need to fix some formatting issues + convert the animated diagrams into static images.

I don't see anything on the page about it, has there been an update on this? I'd love to put this on my e-reader.

Re: How to Think About GPUs

#49

Post by Q6T46nT668w6i3m »

Philpax wrote: post #24
dotancohen wrote: post #22

Earlier quoted context omitted.

I've been hearing that for over a decade. I can't even name off hand any CUDA competitors, none of them are likely to gain enough traction to upset CUDA in the coming decade.

Hence the "if" :-) ROCm is getting some adoption, especially as some of the world's largest public supercomputers have AMD GPUs. Some of this is also being solved by working at a different abstraction layer; you can sometimes be ignorant to the hardware you're running on with PyTorch. It's still leaky, but it's something.

Look at the state of PyTorch’s CI pipelines and you’ll immediately see that ROCm is a nightmare. Especially nowadays when TPU and MPS, while missing features, rarely create cascading failures throughout the stack.

Re: How to Think About GPUs

#50

Post by Q6T46nT668w6i3m »

physicsguy wrote: post #28
tormeh wrote: post #19

I find it very hard to justify investing time into learning something that's neither open source nor has multiple interchangeable vendors. Being good at using Nvidia chips sounds a lot like being an ABAP consultant or similar to me. I realize there's a lot of money to be made in the field right now, but IIUC historically this kind of thing has not been a great move.

It really isn't that hard to pivot. It's worth saying that if you were already writing OpenMP and MPI code then learning CUDA wasn't particularly difficult to get started, and learning to write more performant CUDA code would also help you write faster CPU bound code. It's an evolution of existing models of compute, not a revolution.

I agree that “learning CUDA wasn’t particularly difficult to get started,” there are Grand Canyon sized chasms between CUDA and its alternatives when attempting to crank performance.
Post reply on HN