Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

vettedconsumer.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

#41

Post by searealist »

danbruc wrote: post #39

Why would a RTX 5090 with 32 GB not be able to deal with a 40 GB model? Is there anything preventing me from swapping the weights that do not fit into VRAM in and out of RAM? PCIe 5.0 x16 should max out around 64 GB/s, so slower than the unified memory machine, but at least it should be possible.

There are two phases to LLMs:

1) prefill

2) decode

For prefill, you are compute bound, and it is trivial to batch multiple input tokens together. When using cpu offload, software like llama.cpp will batch weight uploads with tokens that need those weights and perform work on the GPU. It works very well. With a large batch size and pcie5 you can get prefill speeds close to having all weights on the GPU.

For decode, you are bandwidth bound, and it is difficult to batch multiple output tokens together. There is no benefit to sending your weights to the GPU because even if it internally has insane bandwidth, you are still bottlenecked by system RAM (and adding a pcie5 upload would bottleneck it further). This is the number people usually talk about when they say they are getting a certain tk/s.

Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

#43

Post by trinix912 »

1000100_1000101 wrote: post #37
craftkiller wrote: post #6

Earlier quoted context omitted.

I was under the impression that when you're streaming the weights from disk because the full model won't fit in memory, that it is solely reading from the SSD, not writing, so it wouldn't be causing wear on your SSD.

You'd need your OS to support, and be configured to use, a disk mounting option that disables file access timestamps, otherwise reads ARE writes.

...and disable swap.

Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

#44

Post by NortySpock »

"integrated graphics processor, using system memory" had its name dragged through the mud for decades.

So we had to rebadge it to "unified memory".

Curious if we'll ever see some old integrated graphics processor "hacked" to manage to handle 128 GB of allocated system RAM and be able to serve diffusion-LLMs at a decent rate on "old" hardware...

Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

#45

Post by reinitctxoffset »

danbruc wrote: post #39

Why would a RTX 5090 with 32 GB not be able to deal with a 40 GB model? Is there anything preventing me from swapping the weights that do not fit into VRAM in and out of RAM? PCIe 5.0 x16 should max out around 64 GB/s, so slower than the unified memory machine, but at least it should be possible.

5090 can do all but lossless NVFP4 (OMMA) and NVIDIA does fairly good quants of most anything popular. This isn't quite a 4x reduction from what you see on the label because they're a little conservative with the QKV projections (IMHO) but it's on the order of that. So a dense model at 50-70B parameters is the sweet spot. It's a great card for strong dense models.

In principle you could have bidirectional PCIe x16 pipelining at it would move the roofline a little with fast DDR5, I think llama.cpp has a flag for it.

Or go rent a B200 on vast.ai for 4 bucks an hour or thereabouts, a single heavy Opus session for a couple hours is like a week of any model on vast or RunPods.

NVIDIA publishes something called NGC containers that generally work out of the box. I started running Qwen3.6-NVFP4-MTP locally and then I'll put something heavy on Baseten if I'm lazy or Vast if I want a good deal.

Opus (and maybe now 5.6) are still the strongest for like, the really delicate shit, kernel modules or something, but that's on pace to cross over this year, and the overtraining and misalignment are getting so bad when they phase 4.6 out I'm pulling my plan. I don't pay to get gaslit about Constitutional AI.

It's time to have an exit strategy.

Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

#46

Post by hn_c »

searealist wrote: post #41
danbruc wrote: post #39

Why would a RTX 5090 with 32 GB not be able to deal with a 40 GB model? Is there anything preventing me from swapping the weights that do not fit into VRAM in and out of RAM? PCIe 5.0 x16 should max out around 64 GB/s, so slower than the unified memory machine, but at least it should be possible.

There are two phases to LLMs: 1) prefill 2) decode For prefill, you are compute bound, and it is trivial to batch multiple input tokens together. When using cpu offload, software like llama.cpp will batch weight uploads with tokens that need those weights and perform work on the GPU. It works very well. With a large batch size and pcie5 you can get prefill speeds close to having all weights on the GPU. For decode, yo…

> For decode, (...) it is difficult to batch multiple output tokens together.

I think it's the other way around? The GPU has to stream gigabytes of active layer weights to compute the next token, so having a batch of next-token predictions sitting there on the GPU goingh through the layers makes better use of the bandwidth.

At least that's what I observed on a Strix Halo, batching 4 predictions yields like 2-3x the total tps.

Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

#47

Post by reinitctxoffset »

throwa356262 wrote: post #8

"Can't" is not really correct. Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.

This is a very understandable misconception that I wouldn't blame anyone for having but MoE is actually terrible for inference in most any local LLM / home lab scenario. MoE is popular because it's cheap to train, but because most modern routing needs the previous layer's activations (except at the very beginning) it winds up being just this side of impossible to pipeline / prefetch without all the experts resident. Plus the grouped GEMM kernels have terrible support on any card in most people's house, it's just really unwieldy.

Dense models are very straightforward to share/pipeline because you know all the shapes and geometry up front, that's the inference friendly option.

MoE sells a lot of HBMe3.

Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

#48

Post by OutOfHere »

giantrobot wrote: post #14
craftkiller wrote: post #6

Earlier quoted context omitted.

I was under the impression that when you're streaming the weights from disk because the full model won't fit in memory, that it is solely reading from the SSD, not writing, so it wouldn't be causing wear on your SSD.

It is and it doesn't. You only get into disk writes if the system starts paging out to disk.

No one can reliably track uses of "it", and definitely not multiple uses, so please take your time to qualify everything explicitly. Currently I have no idea what each "it" is referring to.

Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

#49

Post by searealist »

hn_c wrote: post #46
searealist wrote: post #41

Earlier quoted context omitted.

There are two phases to LLMs: 1) prefill 2) decode For prefill, you are compute bound, and it is trivial to batch multiple input tokens together. When using cpu offload, software like llama.cpp will batch weight uploads with tokens that need those weights and perform work on the GPU. It works very well. With a large batch size and pcie5 you can get prefill speeds close to having all weights on the GPU. For decode, yo…

> For decode, (...) it is difficult to batch multiple output tokens together. I think it's the other way around? The GPU has to stream gigabytes of active layer weights to compute the next token, so having a batch of next-token predictions sitting there on the GPU goingh through the layers makes better use of the bandwidth. At least that's what I observed on a Strix Halo, batching 4 predictions yields like 2-3x the t…

MTP can give you small batches, but it is still WAY smaller than the batches you can get with prefill, which is limited only by the number of input tokens you have (but has diminishing returns on performance).

But:

1) It still makes no sense to upload the weights to the GPU with MTP as you are still bottlenecked by the weight upload.

2) I'm not sure MTP helps much with MoE models.

Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

#50

Post by [deleted] »

searealist wrote: post #41
danbruc wrote: post #39

Why would a RTX 5090 with 32 GB not be able to deal with a 40 GB model? Is there anything preventing me from swapping the weights that do not fit into VRAM in and out of RAM? PCIe 5.0 x16 should max out around 64 GB/s, so slower than the unified memory machine, but at least it should be possible.

There are two phases to LLMs: 1) prefill 2) decode For prefill, you are compute bound, and it is trivial to batch multiple input tokens together. When using cpu offload, software like llama.cpp will batch weight uploads with tokens that need those weights and perform work on the GPU. It works very well. With a large batch size and pcie5 you can get prefill speeds close to having all weights on the GPU. For decode, yo…

[deleted]
Post reply on HN