Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Every Developer Should Know About GPU Computing (2023)

blog.codingconfessions.com

Post reply on HN
Print view View original

21–30 of 46 posts

Re: What Every Developer Should Know About GPU Computing (2023)

#21

Post by amelius »

jms55 wrote: post #11
winwang wrote: post #4

Makes me consider writing a post on misconceptions of GPU computing, such as requiring the problem to be fully data-parallel.

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…

What is holding manufacturers back to create a bus for the GPU that is as fast as main memory?

Re: What Every Developer Should Know About GPU Computing (2023)

#22

Post by sokoloff »

amelius wrote: post #21
jms55 wrote: post #11

Earlier quoted context omitted.

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…

What is holding manufacturers back to create a bus for the GPU that is as fast as main memory?

PCIe x16 and DDR bandwidth are in the same order of magnitude already (depending on the exact version of each), but around a factor of 16 slower than internal GPU memory.

Re: What Every Developer Should Know About GPU Computing (2023)

#23

Post by zokier »

jms55 wrote: post #11
winwang wrote: post #4

Makes me consider writing a post on misconceptions of GPU computing, such as requiring the problem to be fully data-parallel.

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…

While I do understand conceptually that GPU is basically its own computer, I struggle to understand how this works in terms of operating systems and multitasking. Fundamentally managing resources between tasks is one of the core functions of operating systems, and stuff like CPU schedulers and virtual memory are fairly well understood. But how are the resources on GPUs managed? If I have n processes doing GPU compute (/graphics), how are the limited GPU time and memory allocated between them? Can you set priorities and limits like you can with other resources?

I feel these sort of "operational" questions are often neglected in the discussions, but considering how GPU is increasingly being used in wide range of applications (both for graphics and compute) I think it's becoming relevant to think how they play together.

Re: What Every Developer Should Know About GPU Computing (2023)

#24

Post by b3orn »

ElFitz wrote: post #16
jms55 wrote: post #11

Earlier quoted context omitted.

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…

> Another kind of misconception: data transfer is a _really_ overlooked issue. […] If you want to write 20mb of data to a buffer, that's not just a memcpy, all that data has to go over the PCIe buss to the GPU […], and that's going to be expensive (in real time contexts). Similarly if you want to read a whole large buffer of results back from the GPU, that's going to take some time. Does having a unified memory, like…

In theory yes, because you wouldn't need to copy the data, in practice it depends on the API and you might end up copying data from RAM to RAM. If the API doesn't allow you to simply pass an address to the GPU then you need to allocate memory on the GPU and copy your data to that memory, even if it's unified memory.

Re: What Every Developer Should Know About GPU Computing (2023)

#27

Post by ferbivore »

cgreerrun wrote: post #2

This video is a great explainer too: How do Graphics Cards Work? Exploring GPU Architecture ( https://youtu.be/h9Z4oGN89MU?si=EPPO0kny-gN0zLeC )

These videos always give me an unpleasant feeling that I don't know how to express in a useful way. Almost everything in this one is oversimplified, misleading, or wrong, yet I feel like any attempt to argue for this would come off as pedantic; any individual complaint could be countered by either "it's technically true" or "it's just entertainment".

Like, the "cowboy hat" example is wrong on multiple levels - GPUs are not SIMD machines, model-to-world translation doesn't work like that in practice - but you can maybe excuse it as a useful educational lie, and he does kind-of explain SIMT later, so is objecting to it valid? Or: the video claims to be about "GPUs", but is in fact exclusively about Nvidia's architecture and the GA102 in particular; is this a valid complaint, or is the lie excusable because it's just a YouTube video? Or: it overemphasizes the memory chips because of who's sponsoring it; does this compromise the message? Or: it plays fast-and-loose with die shots and floorplans; is a viewer expected to understand that it's impossible to tell where the FMA units really are? Or: it spends a lot of time on relatively unimportant topics while neglecting things like instruction dispatch, registers, dedicated graphics hardware, etc.; but is it really fair to complain, considering the target audience doesn't seem to be programmers? And so on.

Did you actually get anything out of this video? Any new knowledge? The article seems like a much more useful intro, even if it is also specific to Nvidia and CUDA.

Re: What Every Developer Should Know About GPU Computing (2023)

#28

Post by schmidtleonard »

Miniminix wrote: post #6

If you are not writing the GPU kernel, just use a high level language which wraps up the CUDA, Metal, or whatever. https://julialang.org https://juliagpu.org

The big problem I've had historically with non-native CUDA wrappers is that they always seem to omit or bug some feature that is critical for my application, and the amount of debugging pain and implementation or bugfix work to get around this problem exceeds the effort "savings" of a high level interface by an order of magnitude or three.

Re: What Every Developer Should Know About GPU Computing (2023)

#29

Post by Kon-Peki »

zokier wrote: post #23
jms55 wrote: post #11

Earlier quoted context omitted.

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…

While I do understand conceptually that GPU is basically its own computer, I struggle to understand how this works in terms of operating systems and multitasking. Fundamentally managing resources between tasks is one of the core functions of operating systems, and stuff like CPU schedulers and virtual memory are fairly well understood. But how are the resources on GPUs managed? If I have n processes doing GPU compute…

The CUDA driver handles multitasking issues between processes, as each process has no knowledge of any other process.

But within your own application, yes, you can create multiple streams and assign each a priority.

If you are in Apple land, you can set the priority of your Metal command queues. Even with the integrated GPU, it is logically a separate device and you have limited control over it. And the comment about having its own compiler allows for some weird behavior. For example, it is possible to use an iPad to do Metal GPU programming using the Swift Playgrounds app: Use the standard boilerplate Swift code that sets an app up for executing a Metal shader. But instead of telling it to run a compiled kernel from the kernel library that would have been created during the application compile step on a desktop PC, you pass it Metal C++ source code as a plain old Swift string. The Metal runtime will compile and run it! You get no warnings about errors in your code; it either works or it doesn't, but alas, it shows that the compiler is indeed somewhere else :)

Re: What Every Developer Should Know About GPU Computing (2023)

#30

Post by adrian_b »

zokier wrote: post #23
jms55 wrote: post #11

Earlier quoted context omitted.

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…

While I do understand conceptually that GPU is basically its own computer, I struggle to understand how this works in terms of operating systems and multitasking. Fundamentally managing resources between tasks is one of the core functions of operating systems, and stuff like CPU schedulers and virtual memory are fairly well understood. But how are the resources on GPUs managed? If I have n processes doing GPU compute…

Scheduling and allocation are done by the GPU driver, e.g. the CUDA runtime, with some hardware/firmware assistance from the GPU, which also contains one or more microcontrollers which may perform some of the tasks required for this.
Post reply on HN