Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every developer should know about GPU computing

codeconfessions.substack.com

Post reply on HN
Print view View original

81–90 of 186 posts

Re: What every developer should know about GPU computing

#81

Post by crooked-v »

salawat wrote: post #70
smokel wrote: post #61

Earlier quoted context omitted.

That's a refreshing take. I tried real hard to understand what some continental philosophers, such as Latour, Deleuze, and Žižek are on about, giving some of their texts quite some benefit of the doubt. After about ten years of doing so, I am more and more returning to my previous opinion that some of these just like to talk, and even though they have lots to say, they say so much, that I still have to do all the act…

The Ur subject of philosophy is not the writing and reading of philosophy, but to exercise the mind in coming to grips with that which we have no conception of yet. Language is it's medium, and the IO is those texts. However the nugget at the center is figuring out that magic blackbox that lets you take all the dead ends in those texts and divine a reasonable guiding principle or recognition of an end to the space of…

The Tao Te Ching opens with almost exactly that subject... though, of course, it also makes you think about it a while first to understand what it's getting at.

Re: What every developer should know about GPU computing

#82

Post by godelski »

I thought I'd share something with my experience with HPC that applies to many areas, especially in the rise of GPUs.

The main bottleneck isn't compute, it is memory. If you go to talks you're gonna see lots of figures like this one[0] (typically also showing disk speeds, which are crazy small).

Compute is increasing so fast that at this point we finish our operations long faster than it takes to save those simulations or even create the visualizations and put on disk. There's a lot of research going into this, with a lot of things like in situ computing (asynchronous operations, often pushing to a different machine, but needing many things like flash buffers. See ADIOS[1] as an example software).

What I'm getting at here is that we're at a point where we have to think about that IO bottleneck, even for non-high performance systems. I work in ML now, which we typically think of as compute bound, but being in the generative space there are still many things where the IO bottlenecks. This can be loading batches into memory, writing results to disk, or communication between distributed processes. It's one beg reason we typically want to maximize memory usage (large batches).

There's a lot of low hanging fruit in these areas that aren't going to be generally publishable works but are going to have lots of high impact. Just look at things like LLaMA CPP[2], where in the process they've really decreased the compute time and memory load. There's also projects like TinyLLaMa[3] who are exploring training a 1B model and doing so on limited compute, and are getting pretty good results. But I'll tell you from personal experience, small models and limited compute experience doesn't make for good papers (my most cited work did this and has never been published, gotten many rejections for not competing with models 100x it's size, but is also quite popular in the general scientific community who work with limited compute). Wfiw, companies that are working on applications do value these things, but it is also noise in the community that's hard to parse. Idk how we can do better as a community to not get trapped in these hype cycles, because real engineering has a lot of these aspects too, and they should be (but aren't) really good areas for academics to be working in. Scale isn't everything in research, and there's a lot of different problems out there that are extremely important but many are blind to.

And one final comment, there's lots of code that is used over and over that are not remotely optimized and can be >100x faster. Just gotta slow down and write good code. The move fast and break things method is great for getting moving but the debt compounds. It's just debt is less visible, but there's so much money being wasted from writing bad code (and LLMs are only going to amplify this. They were trained on bad code after all). And no, pytorch isn't going to optimize everything for you automagically (no one can. Optimization is often situationally dependent). You're still gonna need to understand the distributed package and it is worth learning MPI.

[0] https://drivenets.com/wp-content/uploads/2023/05/blog-networ...

[1] https://github.com/ornladios/ADIOS2

[2] https://github.com/ggerganov/llama.cpp

[3] https://github.com/jzhang38/TinyLlama

Re: What every developer should know about GPU computing

#83

Post by einpoklum »

> Most programmers have an intimate understanding of CPUs

Not at all. Very few programmers have an intimate understanding of CPUs. I'm not one of them, for example. And that's without mentioning how most programmers actually write Excel scripts, and most of the rest write interpreted code like Javascript or Python which is somewhat removed from the CPU.

> because they grow up writing code for the CPU

There is not really such thing as "The CPU". The design space of CPUs is large enough for it to contain quite a variation of beasts. For example,

> CPUs were designed to execute sequential instructions

Originally, yes, but - one cannot really make this argument about CPUs since Dennard scaling hit the wall, and scale-out began in earnest.

-----

While the mental model is indeed different, hardware-wise, you should think of GPUs as CPUs, except that...

* You only ever use SIMD registers: 32 lanes x 32bit

* Large register file (up to 256 SIMD registers)

* No branch prediction

* Predicated execution supported, but no if-then-else (sort of).

* Very little stack to speak of

* ~10x L1 cache latency

* A piece of L1 "cache" which doesn't cache anything and you can just write to it like regular memory.

* ~100x L2 cache latency, no coherence, and it's smaller

* ~100x main memory latency (and no L3)

* A bunch of special registers for implementing geometric thread & lane indexing.

* ~10x memory bandwidth

* ~5x-10x cores than on a CPU

(numbers are for NVIDIA cards, AMD are slightly different; and of course individual cards differ etc.)

-----

Now take all of those hardware differences, and conceive a programming model. What you get is the fanciful tale of a "not-like-a-CPU" processor.

Programming for the CPU, we often do the opposite, i.e. ignore the cores we have, ignore things like latency hiding, careful management of what's in registers and what's in the cache, and just write naive serial programs.

Re: What every developer should know about GPU computing

#84

Post by crooked-v »

galangalalgol wrote: post #79
anonporridge wrote: post #49

Earlier quoted context omitted.

Definitely not true about most programmers, but maybe the author meant CS educated engineers. Going through a formal CS program will give you an intimate understanding of CPUs, especially when compared to the very light coverage of GPUs.

Are most programmers self trained now? I can see if someone self trains for fe or even full stack with an eye on compensation, they wouldn't understand program counters and the like. But so many people seem motivated by video games to get into the industry that I'd expect them to be reading about things like the fast inverse square root or similar.

A lot of game dev now happens in high-level engines, often with their own extra layer of scripting engine. Even working on the nitty-gritty of spatial logic rather than using engine-based colliders/detection would be pretty rare.

Re: What every developer should know about GPU computing

#85

Post by [deleted] »

salawat wrote: post #70
smokel wrote: post #61

Earlier quoted context omitted.

That's a refreshing take. I tried real hard to understand what some continental philosophers, such as Latour, Deleuze, and Žižek are on about, giving some of their texts quite some benefit of the doubt. After about ten years of doing so, I am more and more returning to my previous opinion that some of these just like to talk, and even though they have lots to say, they say so much, that I still have to do all the act…

The Ur subject of philosophy is not the writing and reading of philosophy, but to exercise the mind in coming to grips with that which we have no conception of yet. Language is it's medium, and the IO is those texts. However the nugget at the center is figuring out that magic blackbox that lets you take all the dead ends in those texts and divine a reasonable guiding principle or recognition of an end to the space of…

[deleted]

Re: What every developer should know about GPU computing

#86

Post by rational_indian »

qwertox wrote: post #80

Let's assume I have an array of 10.000 lat/lng-pairs. I want to compute the length of the track. I duplicate the array and remove the first item in the duplicated array, append the last entry of the original array to the duplicate in order for them to be equal in length. Then I use a vectorized haversine algorithm on these arrays to obtain a third one with the distances between each "row" of the two arrays. With NumP…

You need to try it and see if it is any faster. NVIDIA has a drop-in replacement for numpy: https://developer.nvidia.com/cunumeric.

Re: What every developer should know about GPU computing

#87

Post by dist-epoch »

qwertox wrote: post #80

Let's assume I have an array of 10.000 lat/lng-pairs. I want to compute the length of the track. I duplicate the array and remove the first item in the duplicated array, append the last entry of the original array to the duplicate in order for them to be equal in length. Then I use a vectorized haversine algorithm on these arrays to obtain a third one with the distances between each "row" of the two arrays. With NumP…

> Would it be worth it if one has to consider that the data needs to be moved to the GPU and the result be retrieved

Depends on your batch size. If the computation on the CPU is less than lets say 200 ms it's probably not worth it.

Also consider that integrated GPUs don't have separate memory, I'm not sure, but they might not have a high cost of moving data to memory.

Re: What every developer should know about GPU computing

#88

Post by tlb »

qwertox wrote: post #80

Let's assume I have an array of 10.000 lat/lng-pairs. I want to compute the length of the track. I duplicate the array and remove the first item in the duplicated array, append the last entry of the original array to the duplicate in order for them to be equal in length. Then I use a vectorized haversine algorithm on these arrays to obtain a third one with the distances between each "row" of the two arrays. With NumP…

Probably not. The computation is only a few trig instructions per array element, so most of the time is moving data on either CPU or GPU.

Re: What every developer should know about GPU computing

#89

Post by dataflow »

qwertox wrote: post #80

Let's assume I have an array of 10.000 lat/lng-pairs. I want to compute the length of the track. I duplicate the array and remove the first item in the duplicated array, append the last entry of the original array to the duplicate in order for them to be equal in length. Then I use a vectorized haversine algorithm on these arrays to obtain a third one with the distances between each "row" of the two arrays. With NumP…

Not the answer to your question, but:

> I duplicate the array and remove the first item in the duplicated array, append the last entry of the original array to the duplicate in order for them to be equal in length.

I assume/hope this is only what you're doing logically, not physically?

Otherwise you might as well just compute the length using n - 1 points from the existing array, then do the remaining portion manually and add it to the existing sum. That would avoid the copying of the whole array.

Post reply on HN