GPU are optimized for number crunching. Do they get used at all for string processing? I ask because I develop data wrangling software and most of it is string processing (joins, concatenations, aggregations, filtering etc), rather than numerical.
What Every Developer Should Know About GPU Computing (2023)
31–40 of 46 posts
Re: What Every Developer Should Know About GPU Computing (2023)
#32Earlier quoted context omitted.
> Another kind of misconception: data transfer is a _really_ overlooked issue. […] If you want to write 20mb of data to a buffer, that's not just a memcpy, all that data has to go over the PCIe buss to the GPU […], and that's going to be expensive (in real time contexts). Similarly if you want to read a whole large buffer of results back from the GPU, that's going to take some time. Does having a unified memory, like…
In theory yes, because you wouldn't need to copy the data, in practice it depends on the API and you might end up copying data from RAM to RAM. If the API doesn't allow you to simply pass an address to the GPU then you need to allocate memory on the GPU and copy your data to that memory, even if it's unified memory.
For Apple specifically, you have to act as if you do not have unified memory because Apple still supports discrete GPUs in Metal and also Swift is reference counted - the CPU portion of the app has no idea if the GPU portion is still using something (remember that the CPU and GPU are logically different devices even when they are on the same die).
When you are running your code on an M- or A-series processor, most of that stuff probably ends up as no-ops. But worse case is that you copy from RAM to RAM, which is extraordinarily faster than pushing anything across the PCIe bus.
Re: What Every Developer Should Know About GPU Computing (2023)
#33This video is a great explainer too: How do Graphics Cards Work? Exploring GPU Architecture ( https://youtu.be/h9Z4oGN89MU?si=EPPO0kny-gN0zLeC )
These videos always give me an unpleasant feeling that I don't know how to express in a useful way. Almost everything in this one is oversimplified, misleading, or wrong, yet I feel like any attempt to argue for this would come off as pedantic; any individual complaint could be countered by either "it's technically true" or "it's just entertainment". Like, the "cowboy hat" example is wrong on multiple levels - GPUs a…
Honestly, I think Branch Education does a solid job with topics like this. Yes, it glosses over details and sometimes simplifies to the point of oversights, but that’s often necessary in educational content to avoid getting bogged down. It’s a balance: if you dive too deep, you risk losing the main points.
Branch Education is designed to introduce complex concepts, often for high schoolers or newcomers to the subject. Even my first grader finds it interesting because it’s visually engaging and conveys a general understanding, even if much of the terminology goes over their head. Their video on how computer chips are made, for example, managed to hold the whole family’s attention. That is hard to do for most of the nerdy shit I watch on YouTube!
It’s not meant to be a deep dive—Ben Eater is better suited for that. His work on instruction counters, registers, and the intricacies of “how CPUs work” is incredible, but it’s for a different audience. You need a fair amount of computer science and electrical engineering knowledge to get the most out of his content. Good luck getting my family to watch him breadboard an entire graphics system; it’s fascinating but requires a serious commitment to appreciate fully.
Re: What Every Developer Should Know About GPU Computing (2023)
#34[deleted]
Re: What Every Developer Should Know About GPU Computing (2023)
#35Makes me consider writing a post on misconceptions of GPU computing, such as requiring the problem to be fully data-parallel.
In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…
When will “they” invent a better bus to go between the system memory used by the CPU and the GPU? It seems like that is a pretty major bottleneck.
It took a while before SSD’s stopped using SATA despite SATA being a huge bottleneck. Now it uses something else whose name I cannot recall. Surely there is work to do something like that for the GPU.
Because while it’s been a long while since I’ve built a computer I do know that the video card has always been the peripheral that pushed the limits of whatever interconnect bus existed at the time. There was all kinds of hacks for it like AGP, Video Local Bus, and then even PCI express to a large degree.
Re: What Every Developer Should Know About GPU Computing (2023)
#36Earlier quoted context omitted.
What is holding manufacturers back to create a bus for the GPU that is as fast as main memory?
PCIe x16 and DDR bandwidth are in the same order of magnitude already (depending on the exact version of each), but around a factor of 16 slower than internal GPU memory.
So the system memory is already slower than the memory on the GPU card?
I suppose next you’ll also say the GPU memory is designed to be accessed in a much more “parallel fat pipe” way that can shove gobs of data across the bus all at once vs the CPU which doesn’t have that requirement?
I mean the whole idea is “single instruction multiple data” and GPU takes that to the extreme… so yeah I guess the data pipeline has to be ultra-wide to shove shit across all the little cores as quickly as possible.
Re: What Every Developer Should Know About GPU Computing (2023)
#37Earlier quoted context omitted.
In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…
While I do understand conceptually that GPU is basically its own computer, I struggle to understand how this works in terms of operating systems and multitasking. Fundamentally managing resources between tasks is one of the core functions of operating systems, and stuff like CPU schedulers and virtual memory are fairly well understood. But how are the resources on GPUs managed? If I have n processes doing GPU compute…
The driver handles time slicing between processes, mapping virtual memory to real memory, etc.
You're right that this is an actual consideration for programs. I don't know about CUDA/GPUGPU stuff, but for games, you need to manage residency of your resources, essentially telling the driver which resources (buffers/textures) are critical and can't be unloaded to make space for other apps, and which aren't critical.
Re: What Every Developer Should Know About GPU Computing (2023)
#38Earlier quoted context omitted.
In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data…
And if one is using iGPU, one might think I'll have a great bandwidth, but reality is that DDR memory for CPU is optimized for low latency not bandwidth and they'll probably have a 64 bit channel (or 2x32 bits) from a single DDR module or 128 bit in dual channel configuration, while something like RTX 4090 will have onboard graphics-DDR GDDR memory on 384 bit channel very much optimized for bandwidth and not latency…
Yep, this is another great callout. Desktop GPUs are (in my experience) often heavily memory limited, and that's with their big high bandwidth memory chips. The latency is a problem, but latency hiding means overall throughput is good and it works out in practice.
iGPUs have less latency, but also much less bandwidth. So all those global memory fetches in modern GPU algorithms become much slower, when looking at a birds-eye level of overall throughput across the dispatch. It's why things like SSAO are way more expensive on iGPUs, despite needing to operate at a lower resolution.
Re: What Every Developer Should Know About GPU Computing (2023)
#39GPU are optimized for number crunching. Do they get used at all for string processing? I ask because I develop data wrangling software and most of it is string processing (joins, concatenations, aggregations, filtering etc), rather than numerical.
Do you have millions of strings that need to be manipulated in the same way at the same time?
Re: What Every Developer Should Know About GPU Computing (2023)
#40This video is a great explainer too: How do Graphics Cards Work? Exploring GPU Architecture ( https://youtu.be/h9Z4oGN89MU?si=EPPO0kny-gN0zLeC )
These videos always give me an unpleasant feeling that I don't know how to express in a useful way. Almost everything in this one is oversimplified, misleading, or wrong, yet I feel like any attempt to argue for this would come off as pedantic; any individual complaint could be countered by either "it's technically true" or "it's just entertainment". Like, the "cowboy hat" example is wrong on multiple levels - GPUs a…
The only time I really feel like I understand something is by building something with it. So actually writing a CUDA kernel to do grayscale conversion and then tweaking the code.
BUT... both the video and the article are useful before you do that. They both allow you to build a mental model of how GPUs work that you can test later on.