Popping the GPU Bubble
moondream.ai
Popping the GPU Bubble
1–10 of 53 posts
Re: Popping the GPU Bubble
#2I really appreciate this type of articles. I feel like a lot of knowledge in LLM training and inference is locked inside the heads of practitioners. Similar to compiler engineers before.
To work in LLM training/inference you’re expected to know this stuff but to know this stuff you need to be working in the space.
Re: Popping the GPU Bubble
#3> you find that the GPU often sits idle, not for lack of work, but because the CPU hasn't told it what to do next yet. This phenomenon is called a GPU bubble.
This is true, but I've never heard anyone refer to this as a GPU bubble before.
I think most people hear "GPU bubble" and think of a financial bubble of some kind.
Re: Popping the GPU Bubble
#4Different bubble than the one I was hoping for.
This appears to be different than the recent "Speculative Pipeline Decoding" paper: https://arxiv.org/abs/2605.30852
Re: Popping the GPU Bubble
#5> you find that the GPU often sits idle, not for lack of work, but because the CPU hasn't told it what to do next yet. This phenomenon is called a GPU bubble. This is true, but I've never heard anyone refer to this as a GPU bubble before. I think most people hear "GPU bubble" and think of a financial bubble of some kind.
Yes, the title seems off - I also thought I am going to be reading about the AI/pricing bubble.
Re: Popping the GPU Bubble
#6> you find that the GPU often sits idle, not for lack of work, but because the CPU hasn't told it what to do next yet. This phenomenon is called a GPU bubble. This is true, but I've never heard anyone refer to this as a GPU bubble before. I think most people hear "GPU bubble" and think of a financial bubble of some kind.
The term I would use would be “underutilised”
Re: Popping the GPU Bubble
#7> you find that the GPU often sits idle, not for lack of work, but because the CPU hasn't told it what to do next yet. This phenomenon is called a GPU bubble. This is true, but I've never heard anyone refer to this as a GPU bubble before. I think most people hear "GPU bubble" and think of a financial bubble of some kind.
The term I would use would be “underutilised”
"stall" is the best term I can think of as in "pipeline stall".
Better term, anyone?
Re: Popping the GPU Bubble
#8> you find that the GPU often sits idle, not for lack of work, but because the CPU hasn't told it what to do next yet. This phenomenon is called a GPU bubble. This is true, but I've never heard anyone refer to this as a GPU bubble before. I think most people hear "GPU bubble" and think of a financial bubble of some kind.
It's very common to call it a GPU bubble in gamedev, though not strictly for CPU induced bubbles.
Re: Popping the GPU Bubble
#9> you find that the GPU often sits idle, not for lack of work, but because the CPU hasn't told it what to do next yet. This phenomenon is called a GPU bubble. This is true, but I've never heard anyone refer to this as a GPU bubble before. I think most people hear "GPU bubble" and think of a financial bubble of some kind.
It appears to be a real term? https://docs.vulkan.org/tutorial/latest/Synchronization/Asyn...
Very odd, but perhaps more familiar to graphics programmers? I will say I'd probably call it a stall, which is exactly what the Vulkan docs call it moments later, so :shrug:
Re: Popping the GPU Bubble
#10I really appreciate this type of articles. I feel like a lot of knowledge in LLM training and inference is locked inside the heads of practitioners. Similar to compiler engineers before. To work in LLM training/inference you’re expected to know this stuff but to know this stuff you need to be working in the space.
Gentle reminder that while most money is spent on LLM inference, the vast majority of useful AI use is in fact not LLMs. Also, more and more work is poured into making small models. One thing I like about the whole export controls saga is that people are finding creative ways to squeeze performance out of these devices as witnessed in this post. But, if you then look at solutions like vLLM, vLLM will just fill whatever VRAM is available, no matter the context size, or the model size. So then you have two things to worry about:
First, where do you know exactly what the optimal VRAM assignment per model, per context size is, which seems to be currently based purely on experience and second how do you make sure that only that amount is available to your infra/containers, which is being handled by DRA and stuff like https://project-hami.io
While only tangentially related to the blog post here. The title is picked in such a way that I couldn't help, but put the shameless plug here. When he wrote popping the bubble, I thought we're talking about devices and reducing NVIDIA dependency, but this seems very focused on Cuda.
Disclaimer: I work with Dynamia.ai, the founders of which created HAMi.