Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

github.com

Post reply on HN
Print view View original

61–70 of 162 posts

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#61

Post by PhilipRoman »

znpy wrote: post #47
semi-extrinsic wrote: post #35

Earlier quoted context omitted.

I honestly don't understand the meme with RPi clusters. For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance. If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre…

You can buy it but you can't run it, unless you're fairly wealthy. In my country (italy) a basic colocation service is like 80 euros/month + vat, and that only includes 100Wh of power and a 100mbps connection. +100wh/month upgrades are like +100 euros. I looked up the kind of servers and cpus you're talking about and the cpu alone can pull something like 180W/h, without accounting for fans, disks and other stuff (stu…

Interesting, I run a small cluster of 4 mini pcs (22 cores total). I think it should be comparable to the aforementioned EPYC. Power load is a rounding error compared to appliances like electric iron at 1700W, etc. The impact on electrical bill is minimal as well. Idle power draw is about 5W per server, which translates to ~80 cents a month. Frankly my monitor uses more power on average than the rest of the homelab.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#62

Post by derefr »

semi-extrinsic wrote: post #35
memhole wrote: post #2

This is the modern Beowulf cluster.

I honestly don't understand the meme with RPi clusters. For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance. If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre…

> For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance.

You could also get one or two Ryzen mini PCs with similar specs for that price. Which might be a good idea, if you want to leave O(N) of them running on your desk, house without spending much on electricity or cooling. (Also, IMHO, the advantages of having an Epyc really only become apparent when you're tossing around multiple 10Gbit NICs, 16+ NVMe disks, etc. and so saturating all the PCIe lanes.)

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#63

Post by baq »

JKCalhoun wrote: post #44
zdw wrote: post #12

Does adding memory help? There's a Rpi 5 with 16GB RAM recently available.

The 16 GB Pi 5 comes and goes. I was able to snag one recently when Adafruit got a delivery in — then they sold right out again. But, yeah, performance aside, there are models that Ollama won't run at all as they need more than 8GB to run.

Rpi 5 is difficult to justify. I’d like to see a 4x N150 minipc benchmark.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#64

Post by baq »

behnamoh wrote: post #20
ripped_britches wrote: post #8

Earlier quoted context omitted.

Lots of use cases don’t require low latency. Background work for agents. CI jobs. Other stuff I haven’t thought of.

If my "automated" CI job takes more than 5 minutes, I'll do it myself..

Where I’ve been doing CI 5 minutes was barely enough to warm caches on a cold runner

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#65

Post by rafaelmn »

yk wrote: post #57
sgt101 wrote: post #37

Earlier quoted context omitted.

Well, Deepseek trained them?

Yes, but it would've been nice to call them D1-something, instead of constantly having to switch back and forth between Deepseek R1 (here I mean the 604B model) as distinguished from Deepseek R1 (the reasoning model and it's distillates.)

You can say R1-604b to disambiguate, just like we have llama 3 8b/70b etc.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#67

Post by jckahn »

varispeed wrote: post #58
replete wrote: post #52

That's not a bad result, although for £320 for 4x Pi5s you could probably find a used 12GB 3080 and probably more than 10x token speed

> Deepseek R1 Distill 8B Q40 on 1x 3080, 60.43 tok/s (eval 110.68 tok/s) That wouldn't get on Hacker News ;-)

HNDD: Hacker News Driven Development

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#68

Post by jonatron »

JKCalhoun wrote: post #54

I did not see (understand) how multiple Raspberry Pis are being used in parallel. Maybe someone can point me in the right direction to understand this.

Blog post from same author explaining https://b4rtaz.medium.com/how-to-run-llama-3-405b-on-home-de...

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#69

Post by hangonhn »

tofof wrote: post #4

This continues the pattern of all other announcements of running 'Deepseek R1' on raspberry pi - that they are running llama (or qwen), modified by deepseek's distillation technique.

Can you explain to a non ML software engineer what these distillation methods mean? What does it mean to have R1 train a Llama model? What is special about DeepSeek’s distillation methods? Thanks!

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#70

Post by HPsquared »

tofof wrote: post #4

This continues the pattern of all other announcements of running 'Deepseek R1' on raspberry pi - that they are running llama (or qwen), modified by deepseek's distillation technique.

And DeepSeek itself is (allegedly) a distillation of OpenAI models.
Post reply on HN