Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Every Developer Should Know About GPU Computing (2023)

blog.codingconfessions.com

Post reply on HN
Print view View original

11–20 of 46 posts

Re: What Every Developer Should Know About GPU Computing (2023)

#11

Post by jms55 »

winwang wrote: post #4

Makes me consider writing a post on misconceptions of GPU computing, such as requiring the problem to be fully data-parallel.

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc.

You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet.

Another kind of misconception: data transfer is a _really_ overlooked issue. People think "oh this is a parallel problem, I can have the GPU do it" and completely discount the cost to send the data to the GPU, and then get it back. If you want to write 20mb of data to a buffer, that's not just a memcpy, all that data has to go over the PCIe buss to the GPU (which again, is a completely separate device unless you're using an iGPU), and that's going to be expensive (in real time contexts). Similarly if you want to read a whole large buffer of results back from the GPU, that's going to take some time.

Re: What Every Developer Should Know About GPU Computing (2023)

#12

Post by physicsguy »

jms55 wrote: post #11
winwang wrote: post #4

Makes me consider writing a post on misconceptions of GPU computing, such as requiring the problem to be fully data-parallel.

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…

I found that, I recently worked on a project where there was a Python library using pymc3, scikit-learn and pytensor that is used for time sensitive calculations. The performance wasn't very good and someone was pushing for us to spend a lot of time moving it to GPU, but as I had to point out, that means almost a total rewrite, not simply switching to the GPU enabled version(s) of those libraries, where they even exist.

Re: What Every Developer Should Know About GPU Computing (2023)

#13

Post by wiz21c »

jms55 wrote: post #11
winwang wrote: post #4

Makes me consider writing a post on misconceptions of GPU computing, such as requiring the problem to be fully data-parallel.

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…

Another one I had when starting was to underestimate the power of the GPU. I really needed to increase the size of my toy problems to actually see the benefits of using a GPU.

For data transfers, my experience is that you rather quickly hit that bottleneck, and it's a tough one. And it's not proportional to the number of transferred bits: transferring one byte naively can be extremely costly (like half the performance, I'm not joking)

Re: What Every Developer Should Know About GPU Computing (2023)

#14

Post by ImHereToVote »

esperent wrote: post #8

Unrelated but I absolutely love this reply from the previous time this was posted and someone complained about the line "most programmmers ...": > Try this on: "A non-trivial number of Computer Scientists, Computer Engineers, Electrical Engineers, and hobbyists have ..." > Took some philosophy courses for fun in college. I developed a reading skill there that lets me forgive certain statements by improving them inste…

That's beautiful

Re: What Every Developer Should Know About GPU Computing (2023)

#15

Post by dmichulke »

esperent wrote: post #8

Unrelated but I absolutely love this reply from the previous time this was posted and someone complained about the line "most programmmers ...": > Try this on: "A non-trivial number of Computer Scientists, Computer Engineers, Electrical Engineers, and hobbyists have ..." > Took some philosophy courses for fun in college. I developed a reading skill there that lets me forgive certain statements by improving them inste…

I know this process as steelmanning.

https://umbrex.com/resources/tools-for-thinking/what-is-stee...

If you see people doing this, befriend them because it means they're valuing knowledge higher than their ego.

Re: What Every Developer Should Know About GPU Computing (2023)

#16

Post by ElFitz »

jms55 wrote: post #11
winwang wrote: post #4

Makes me consider writing a post on misconceptions of GPU computing, such as requiring the problem to be fully data-parallel.

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…

> Another kind of misconception: data transfer is a _really_ overlooked issue. […] If you want to write 20mb of data to a buffer, that's not just a memcpy, all that data has to go over the PCIe buss to the GPU […], and that's going to be expensive (in real time contexts). Similarly if you want to read a whole large buffer of results back from the GPU, that's going to take some time.

Does having a unified memory, like Apple’s M-series chips, help with that?

Re: What Every Developer Should Know About GPU Computing (2023)

#17

Post by aquariusDue »

dmichulke wrote: post #15
esperent wrote: post #8

Unrelated but I absolutely love this reply from the previous time this was posted and someone complained about the line "most programmmers ...": > Try this on: "A non-trivial number of Computer Scientists, Computer Engineers, Electrical Engineers, and hobbyists have ..." > Took some philosophy courses for fun in college. I developed a reading skill there that lets me forgive certain statements by improving them inste…

I know this process as steelmanning. https://umbrex.com/resources/tools-for-thinking/what-is-stee... If you see people doing this, befriend them because it means they're valuing knowledge higher than their ego.

Kinda reminds me of Rogerian rhetoric: https://en.wikipedia.org/wiki/Rogerian_argument

I suppose it always depends on the goals of the conversation and the participants' greater view of the world. Your advice also seems similar to the sage wisdom along the lines of "It's the mark of an educated mind to entertain a thought without accepting it" but goes one step further.

I generally agree with you, it truly is rare to find people who can put aside their ego for the pursuit of a higher (or common) goal.

Re: What Every Developer Should Know About GPU Computing (2023)

#18

Post by Hunpeter »

penr0se wrote: post #7
cgreerrun wrote: post #2

This video is a great explainer too: How do Graphics Cards Work? Exploring GPU Architecture ( https://youtu.be/h9Z4oGN89MU?si=EPPO0kny-gN0zLeC )

Wow, this is one of the best videos I've ever watched. Thanks for sharing

I've been binging Branch Education the last week or so, and I concur that the videos are exceptionally well made. Some commenters noticed one or two mistakes in some of them, but nothing major.

Re: What Every Developer Should Know About GPU Computing (2023)

#19

Post by amelius »

esperent wrote: post #8

Unrelated but I absolutely love this reply from the previous time this was posted and someone complained about the line "most programmmers ...": > Try this on: "A non-trivial number of Computer Scientists, Computer Engineers, Electrical Engineers, and hobbyists have ..." > Took some philosophy courses for fun in college. I developed a reading skill there that lets me forgive certain statements by improving them inste…

Seems in-line with this HN guideline:

> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

Re: What Every Developer Should Know About GPU Computing (2023)

#20

Post by trq01758 »

jms55 wrote: post #11
winwang wrote: post #4

Makes me consider writing a post on misconceptions of GPU computing, such as requiring the problem to be fully data-parallel.

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…

And if one is using iGPU, one might think I'll have a great bandwidth, but reality is that DDR memory for CPU is optimized for low latency not bandwidth and they'll probably have a 64 bit channel (or 2x32 bits) from a single DDR module or 128 bit in dual channel configuration, while something like RTX 4090 will have onboard graphics-DDR GDDR memory on 384 bit channel very much optimized for bandwidth and not latency pushing according to specs a terabyte per second. Apple really needed their memory architecture - having a high memory bandwidth for onboard GPU to have reasonable performance.
Post reply on HN