Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Popping the GPU Bubble

moondream.ai

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Popping the GPU Bubble

#41

Post by augment_me »

radq wrote: post #25
augment_me wrote: post #23

As someone who works in the field, the blog is nice but it has a lot of CODEX fingerprints on it, and it's also very specific to the size of the model in question in a way that is not explicit from the blog until the very last section. In general, for some reason CODEX loves CUDA-streams, it's the first optimization it goes for every time when writing GPU kernels. However in many cases this is not a bottleneck, it ha…

Appreciate you saying the blog was nice. Not sure what you mean by "CODEX fingerprints", but I'll engage with the other points. We work on small models, and our customers want real-time inference on modern GPUs. The sub-title says "near-realtime VLM inference". 20-30ms forward passes are a non-starter for these workloads. If you scroll down to the section titled "A cost model for the bubble", you will find both bench…

My comment is aimed to highlight that the "GPU Bubble" is frames as a general solution when it's not, its a specific bottleneck based on your model size. Your dont mention your model size anywhere, the reader has to infer it from the runtimes, and if they dont know the average forward pass of a model, well too bad, they will leave without understanding the actual trade-off.

The benchmarks you point to in the section titled "A cost model for the bubble" dont include any CPU overheads or the T_block-T_pipe you mention, they just give the improvement %.

In general, you answers here in the thread read as defensive and unhumble. They leave a sour taste of your company, you should consider how you engage with your audience.

Re: Popping the GPU Bubble

#42

Post by tjoekbezoer »

radq wrote: post #27
tjoekbezoer wrote: post #24

Regarding the critique on the title: perhaps an analogy can be made to propeller cavitation on ships. Water influx rate, propeller design and operational parameters all influence the detrimental effect of water bubbles forming — deteriorating the system's efficiency. The GPU would be the propeller, the influx is the work, and the operational parameters is what this article's about.

I'm disappointed with the commentary here. "GPU bubble" is an industry standard term, and literally how I would describe this to my colleagues in the industry. Look for example at the second slide here https://media.steampowered.com/apps/valve/2015/Alex_Vlachos_...

Just trying to be helpful by making an adequately coined term more palatable to a critical audience, thereby expediting the end of a fruitless discussion on an otherwise excellent article. Compliments.

Re: Popping the GPU Bubble

#43

Post by knollimar »

amelius wrote: post #40
aleph_minus_one wrote: post #38

Earlier quoted context omitted.

In data center operations, GPUs have some specific lifetime. Because datacenter GPUs are currently so expensive and hard to get, they don't get dropped on the market at some point (even if a better replacement has arrived), but used as long as possible. Even if the AI companies decide to use their own ASICs, they will rather slowly, but continuously introduce them, while removing GPUs that have reached their end of l…

Yes, short term this is right. But at some point PyTorch will have a model.toVHDL() method, and we'll have a PCBWAY-style website for tapeout of the circuit. Nvidia's future looks less bright than they think and their GPU market will certainly pop.

Doesn't that assume that VHDL is trivial? I feel like there are tons of performance tradeoffs or hardware designers wouldn't have jobs

Re: Popping the GPU Bubble

#44

Post by dragonwriter »

amelius wrote: post #40
aleph_minus_one wrote: post #38

Earlier quoted context omitted.

In data center operations, GPUs have some specific lifetime. Because datacenter GPUs are currently so expensive and hard to get, they don't get dropped on the market at some point (even if a better replacement has arrived), but used as long as possible. Even if the AI companies decide to use their own ASICs, they will rather slowly, but continuously introduce them, while removing GPUs that have reached their end of l…

Yes, short term this is right. But at some point PyTorch will have a model.toVHDL() method, and we'll have a PCBWAY-style website for tapeout of the circuit. Nvidia's future looks less bright than they think and their GPU market will certainly pop.

I can't imagine that model lifetimes will ever justify using model-specific ASICS for public serving (maybe something like serving fixed certified AI models in a vehicle or robot) over more generic GPUs/NPUs until after the AI bubble pops.

Re: Popping the GPU Bubble

#45

Post by ksbd-pls-finish »

fragmede wrote: post #34
radq wrote: post #26

Earlier quoted context omitted.

This is what people in the field call it. I'm sorry you're offended.

You. You are people in the field. You can choose to name it anything else in the article that you just wrote. "We call it the GPU-CPU pipeline stall, but others might call it the GPU bubble."

The term is much older than the current GPU craze though. Ypu're trying to regulate how experts in a field communicate, which is... Weird.

Re: Popping the GPU Bubble

#46

Post by brrrrrm »

nl wrote: post #3

> you find that the GPU often sits idle, not for lack of work, but because the CPU hasn't told it what to do next yet. This phenomenon is called a GPU bubble. This is true, but I've never heard anyone refer to this as a GPU bubble before. I think most people hear "GPU bubble" and think of a financial bubble of some kind.

it's very much an in-domain term for folks in machine learning. heavily used when pipeline parallelism caught on in training https://alband.github.io/doc_view/pipeline.html

Re: Popping the GPU Bubble

#47

Post by ralferoo »

SCdF wrote: post #9
nl wrote: post #3

> you find that the GPU often sits idle, not for lack of work, but because the CPU hasn't told it what to do next yet. This phenomenon is called a GPU bubble. This is true, but I've never heard anyone refer to this as a GPU bubble before. I think most people hear "GPU bubble" and think of a financial bubble of some kind.

It appears to be a real term? https://docs.vulkan.org/tutorial/latest/Synchronization/Asyn... Very odd, but perhaps more familiar to graphics programmers? I will say I'd probably call it a stall, which is exactly what the Vulkan docs call it moments later, so :shrug:

I'm a rendering engineer and have used this term frequently.

It's actually a very common technique in rendering to not always be able to easily fill in the gaps, that we frequently deliberately introduce an extra frame of latency, so that the GPU is rendering jobs for the latter half of rendering passes for frame N+1 and the early half of rendering passes of frame N+2 while frame N is visible. This still means that a frame takes the same total GPU time to render, but means that the gaps between jobs on a single frame can be usefully filled with something else from the other.

Re: Popping the GPU Bubble

#49

Post by aleph_minus_one »

dragonwriter wrote: post #44
amelius wrote: post #40

Earlier quoted context omitted.

Yes, short term this is right. But at some point PyTorch will have a model.toVHDL() method, and we'll have a PCBWAY-style website for tapeout of the circuit. Nvidia's future looks less bright than they think and their GPU market will certainly pop.

I can't imagine that model lifetimes will ever justify using model-specific ASICS for public serving (maybe something like serving fixed certified AI models in a vehicle or robot) over more generic GPUs/NPUs until after the AI bubble pops.

Be aware that currently the hardware costs and electric bill are two huge problems of modern LLMs.

If such AI models will deliver on their qualitative promises, and just the huge cost is the burden to overcome, custom ASIC might be a part of the solution.

If, on the other hand, AI models will still be unsuitable for many applications because of their qualitative issues, it is a much harder and different problem to solve - in this case, the AI bubble will plausibly burst.

Re: Popping the GPU Bubble

#50

Post by amelius »

knollimar wrote: post #43
amelius wrote: post #40

Earlier quoted context omitted.

Yes, short term this is right. But at some point PyTorch will have a model.toVHDL() method, and we'll have a PCBWAY-style website for tapeout of the circuit. Nvidia's future looks less bright than they think and their GPU market will certainly pop.

Doesn't that assume that VHDL is trivial? I feel like there are tons of performance tradeoffs or hardware designers wouldn't have jobs

No it does not assume that. Some very smart people will write that model.toVHDL() function. And keep in mind that a DL model is only a very small subset of what you can use VHDL for, and most models will have a very similar implementation in hardware from a conceptual point of view.

And don't take it too literally, VHDL could be replaced by other hardware design languages, maybe even at lower abstraction levels.

Post reply on HN