Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running DeepSeek R1 Models Locally on NPU

blogs.windows.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: Running DeepSeek R1 Models Locally on NPU

#11

Post by justincormack »

tamlin wrote: post #10
justincormack wrote: post #9

Earlier quoted context omitted.

NPU also has no more memory bandwidth than CPU, but then the GPU on these machines doesnt either.

Agree on NPU vs CPU memory bandwidth, but not sure about characterizing the GPU that way. GDDR is usually faster than DDR of the same generation, and on higher end graphics cards has a width bus width. A few GPUs have HBM and pretty much all datacenter ML accelerators (NVidia B200 / H100 / A100, Google TPU, etc). The PCIe bus between the host memory and GPU memory is a bottleneck for intensive workloads. To perform a…

Yeah standalone GPUs do indeed have more bandwidth, but most of these Copilot PCs that have NPUs just have shared memory for everything I think.

fetching 16 8 bit values vs 32 4 bit values is the same, this is the form they are stored in memory. Doing some unpacking into more registers and back is more or less free anyway, if you are memory bandwidth bound. Largely on these lower end machines everything is memory bound not compute bound, although the CPUs cant often use the full memory bandwidth in some systems (eg the Macs) but the GPU can.

Re: Running DeepSeek R1 Models Locally on NPU

#12

Post by rahimnathwani »

RandomBK wrote: post #6
darthrupert wrote: post #4

Earlier quoted context omitted.

Wait, what am I running on my 32GB Macbook then? I thought it was the 32b version of deepseek-r1.

The only 32B distill I'm aware of is `DeepSeek-R1-Distill-Qwen-32B`, which would be a base model of `Qwen-32B` distilled (further trained) on outputs from the full R1 model.

That model's weights are around 64GB: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-...

GP is likely running the 4-bit quantized version of the finetuned Qwen model.

Re: Running DeepSeek R1 Models Locally on NPU

#13

Post by rahimnathwani »

darthrupert wrote: post #4
RandomBK wrote: post #3

Reminder: DeepSeek distilled models are better thought of as fine-tunes of Qwen/Llama using DeepSeek output, and are not the same as actual DeepSeek v3 or R1. This unfortunate naming has sown plenty of confusion around DeepSeek's quality and resource requirements. Actual DeepSeek v3/R1 continues to require at least ~100GB of VRAM/Mem/SSD, and this does not change that.

Wait, what am I running on my 32GB Macbook then? I thought it was the 32b version of deepseek-r1.

Deepseek R1 has 671 billion parameters. Even if you could quantize each parameter to just 1 bit (from 8 bits), you'd still need 84GB of RAM just for the weights. There is no 32B parameter version of the V3/R1 model architecture.

Re: Running DeepSeek R1 Models Locally on NPU

#14

Post by bestouff »

RandomBK wrote: post #3

Reminder: DeepSeek distilled models are better thought of as fine-tunes of Qwen/Llama using DeepSeek output, and are not the same as actual DeepSeek v3 or R1. This unfortunate naming has sown plenty of confusion around DeepSeek's quality and resource requirements. Actual DeepSeek v3/R1 continues to require at least ~100GB of VRAM/Mem/SSD, and this does not change that.

Out of curiosity, would an A100 80GB work for this ?

Re: Running DeepSeek R1 Models Locally on NPU

#15

Post by bestouff »

bestouff wrote: post #14
RandomBK wrote: post #3

Reminder: DeepSeek distilled models are better thought of as fine-tunes of Qwen/Llama using DeepSeek output, and are not the same as actual DeepSeek v3 or R1. This unfortunate naming has sown plenty of confusion around DeepSeek's quality and resource requirements. Actual DeepSeek v3/R1 continues to require at least ~100GB of VRAM/Mem/SSD, and this does not change that.

Out of curiosity, would an A100 80GB work for this ?

Replying to myself: apparently it's not 100GB VRAM but more around 700GB VRAM that's needed to run DeepSeek R1. The gear needed to run that would cost something in the vincinity of 100K€ !

Re: Running DeepSeek R1 Models Locally on NPU

#16

Post by RandomBK »

bestouff wrote: post #15
bestouff wrote: post #14

Earlier quoted context omitted.

Out of curiosity, would an A100 80GB work for this ?

Replying to myself: apparently it's not 100GB VRAM but more around 700GB VRAM that's needed to run DeepSeek R1. The gear needed to run that would cost something in the vincinity of 100K€ !

Yup. I was referring to the 1.58B quant which seemed to be performing alright and would be the smallest real-DeepSeek model. That requires ~140GB, which is just barely doable on a 128GB RAM + 24GB VRAM setup + a lot of patience. Others have made it work at 64GB RAM + a fast SSD.

The true minimally-quantized DeepSeek experience will need one or possibly two 8xH100 nodes, so well upwards of $100K in CapEx.

Re: Running DeepSeek R1 Models Locally on NPU

#17

Post by tamlin »

justincormack wrote: post #11
tamlin wrote: post #10

Earlier quoted context omitted.

Agree on NPU vs CPU memory bandwidth, but not sure about characterizing the GPU that way. GDDR is usually faster than DDR of the same generation, and on higher end graphics cards has a width bus width. A few GPUs have HBM and pretty much all datacenter ML accelerators (NVidia B200 / H100 / A100, Google TPU, etc). The PCIe bus between the host memory and GPU memory is a bottleneck for intensive workloads. To perform a…

Yeah standalone GPUs do indeed have more bandwidth, but most of these Copilot PCs that have NPUs just have shared memory for everything I think. fetching 16 8 bit values vs 32 4 bit values is the same, this is the form they are stored in memory. Doing some unpacking into more registers and back is more or less free anyway, if you are memory bandwidth bound. Largely on these lower end machines everything is memory bou…

Yes, agree. Probably the main thing is the NPU is just a dedicated unit without the generality / complexity of a CPU and so able to crunch matmuls more efficiently.
Post reply on HN