Running DeepSeek R1 Models Locally on NPU
blogs.windows.com
Running DeepSeek R1 Models Locally on NPU
1–10 of 17 posts
Re: Running DeepSeek R1 Models Locally on NPU
#2How much are NPUs more efficient than GPUs ? What are the limitations , it seems it will have support for deepseek R1 soon
Re: Running DeepSeek R1 Models Locally on NPU
#3Reminder: DeepSeek distilled models are better thought of as fine-tunes of Qwen/Llama using DeepSeek output, and are not the same as actual DeepSeek v3 or R1.
This unfortunate naming has sown plenty of confusion around DeepSeek's quality and resource requirements. Actual DeepSeek v3/R1 continues to require at least ~100GB of VRAM/Mem/SSD, and this does not change that.
Re: Running DeepSeek R1 Models Locally on NPU
#4Reminder: DeepSeek distilled models are better thought of as fine-tunes of Qwen/Llama using DeepSeek output, and are not the same as actual DeepSeek v3 or R1. This unfortunate naming has sown plenty of confusion around DeepSeek's quality and resource requirements. Actual DeepSeek v3/R1 continues to require at least ~100GB of VRAM/Mem/SSD, and this does not change that.
Wait, what am I running on my 32GB Macbook then? I thought it was the 32b version of deepseek-r1.
Re: Running DeepSeek R1 Models Locally on NPU
#5Reminder: DeepSeek distilled models are better thought of as fine-tunes of Qwen/Llama using DeepSeek output, and are not the same as actual DeepSeek v3 or R1. This unfortunate naming has sown plenty of confusion around DeepSeek's quality and resource requirements. Actual DeepSeek v3/R1 continues to require at least ~100GB of VRAM/Mem/SSD, and this does not change that.
Wait, what am I running on my 32GB Macbook then? I thought it was the 32b version of deepseek-r1.
You are running Qwen2.5 32b that has been fine tuned on data that was generated by R1
Re: Running DeepSeek R1 Models Locally on NPU
#6Reminder: DeepSeek distilled models are better thought of as fine-tunes of Qwen/Llama using DeepSeek output, and are not the same as actual DeepSeek v3 or R1. This unfortunate naming has sown plenty of confusion around DeepSeek's quality and resource requirements. Actual DeepSeek v3/R1 continues to require at least ~100GB of VRAM/Mem/SSD, and this does not change that.
Wait, what am I running on my 32GB Macbook then? I thought it was the 32b version of deepseek-r1.
The only 32B distill I'm aware of is `DeepSeek-R1-Distill-Qwen-32B`, which would be a base model of `Qwen-32B` distilled (further trained) on outputs from the full R1 model.
Re: Running DeepSeek R1 Models Locally on NPU
#7How much are NPUs more efficient than GPUs ? What are the limitations , it seems it will have support for deepseek R1 soon
A decent chunk of AI computation is the ability to do matrix multiplication fast. Part of that is reducing the amount of data transferred to and from the matrix multiplication hardware on the NPU and GPU; memory bandwidth is a significant bottleneck. The article is highlighting 4-bit format use.
GPUs are an evolving target. New GPUs have tensor cores and support all kinds of interesting numeric formats, older GPUs don't support any of the formats that AI workloads are using today (e.g. BF16, int4, all the various smaller FP types).
NPU will be more efficient because it is much less general an GPU and doesn't have any gates for graphics. However, it is also fairly restricted. Cloud hardware is orders of magnitude faster (due to much higher compute resources I/O bandwidth), e.g. https://cloud.google.com/tpu/docs/v6e.
Re: Running DeepSeek R1 Models Locally on NPU
#8Reminder: DeepSeek distilled models are better thought of as fine-tunes of Qwen/Llama using DeepSeek output, and are not the same as actual DeepSeek v3 or R1. This unfortunate naming has sown plenty of confusion around DeepSeek's quality and resource requirements. Actual DeepSeek v3/R1 continues to require at least ~100GB of VRAM/Mem/SSD, and this does not change that.
Wait, what am I running on my 32GB Macbook then? I thought it was the 32b version of deepseek-r1.
[dead]
Re: Running DeepSeek R1 Models Locally on NPU
#9How much are NPUs more efficient than GPUs ? What are the limitations , it seems it will have support for deepseek R1 soon
A decent chunk of AI computation is the ability to do matrix multiplication fast. Part of that is reducing the amount of data transferred to and from the matrix multiplication hardware on the NPU and GPU; memory bandwidth is a significant bottleneck. The article is highlighting 4-bit format use. GPUs are an evolving target. New GPUs have tensor cores and support all kinds of interesting numeric formats, older GPUs do…
NPU also has no more memory bandwidth than CPU, but then the GPU on these machines doesnt either.
Re: Running DeepSeek R1 Models Locally on NPU
#10Earlier quoted context omitted.
A decent chunk of AI computation is the ability to do matrix multiplication fast. Part of that is reducing the amount of data transferred to and from the matrix multiplication hardware on the NPU and GPU; memory bandwidth is a significant bottleneck. The article is highlighting 4-bit format use. GPUs are an evolving target. New GPUs have tensor cores and support all kinds of interesting numeric formats, older GPUs do…
NPU also has no more memory bandwidth than CPU, but then the GPU on these machines doesnt either.
Agree on NPU vs CPU memory bandwidth, but not sure about characterizing the GPU that way. GDDR is usually faster than DDR of the same generation, and on higher end graphics cards has a width bus width. A few GPUs have HBM and pretty much all datacenter ML accelerators (NVidia B200 / H100 / A100, Google TPU, etc). The PCIe bus between the host memory and GPU memory is a bottleneck for intensive workloads.
To perform a multiplication on CPU, even SIMD, that values have to fetched and converted to a form the CPU has multipliers for. This means smaller numeric types penalised. For a 128-bit memory bus, an NPU can fetch 32 4-bit values per transfer; the best case for a CPU is 16 8-bit values.
Details are scant on Microsoft's NPU, but it probably has many parallel multipliers; either in the form of tensor cores or a systolic array. The effective number of matmul's per second (or per memory operation) is higher.