Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

github.com

Post reply on HN
Print view View original

91–100 of 162 posts

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#91

Post by cratermoon »

zamadatix wrote: post #21
cratermoon wrote: post #16

Earlier quoted context omitted.

Is there a back-of-the-napkin way to calculate how much memory a given model will take? Or what parameter/quantization model will fit in a given memory size?

To find the absolute minimum you just multiply the number of parameters by the bits per parameter, divide by 8 if you want bytes. In case 8 billion parameters of 4 bits each means "at least 4 billion bytes". For back of the napkin add ~20% overhead to that (it really depends on your context setup and a few other things but that's a good swag to start with) and then add whatever memory the base operating system is goi…

Thank you. So F16 would be 16 bits per weight, and F32 would be 32? Next question, if you don't mind, what are the tradeoffs in choosing between a model with more parameters quantized to smaller values vs fewer parameters full-precision models? My current understanding is to prefer smaller quantized models over larger full-precision.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#92

Post by pythux »

rafaelmn wrote: post #65
yk wrote: post #57

Earlier quoted context omitted.

Yes, but it would've been nice to call them D1-something, instead of constantly having to switch back and forth between Deepseek R1 (here I mean the 604B model) as distinguished from Deepseek R1 (the reasoning model and it's distillates.)

You can say R1-604b to disambiguate, just like we have llama 3 8b/70b etc.

These models are not of the same nature either. Their training was done in a different way. A uniform naming (even with explicit number of parameters) would still be misleading.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#93

Post by corysama »

hangonhn wrote: post #69
tofof wrote: post #4

This continues the pattern of all other announcements of running 'Deepseek R1' on raspberry pi - that they are running llama (or qwen), modified by deepseek's distillation technique.

Can you explain to a non ML software engineer what these distillation methods mean? What does it mean to have R1 train a Llama model? What is special about DeepSeek’s distillation methods? Thanks!

"Quantized" models try to approximate the full model using less bits.

"Distilled" models are other models (Llama, Qwen) that have been put through an additional training round using DeepSeek as a teacher.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#94

Post by avereveard »

tofof wrote: post #4

This continues the pattern of all other announcements of running 'Deepseek R1' on raspberry pi - that they are running llama (or qwen), modified by deepseek's distillation technique.

This has been brilliant marketing from deepseek and they're gaining mindshare at a staggering rate.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#95

Post by plagiarist »

semi-extrinsic wrote: post #35
memhole wrote: post #2

This is the modern Beowulf cluster.

I honestly don't understand the meme with RPi clusters. For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance. If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre…

You also get a normal motherboard firmware and normal PCI with that.

I don't know if my complaint applies to RPi, or just other SBCs: the last time I got excited about an SBC, it turned out it boots unconditionally from SD card if one is inserted. IMO that's completely unacceptable for an "embedded" board that is supposed to be tucked away.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#96

Post by andix »

kvirani wrote: post #88
andix wrote: post #77

Earlier quoted context omitted.

Its llama/quen with some additional training to add reasoning. In a similar way deep seeks v3 was trained into r1. It also looks to me like there was some Chinese propaganda trained into llama/quen too, but that’s just my observation.

You have my curiosity. Like what and how did you find it?

Ask about Xi Jinping in all the ways you can imagine (jokes, bad habits, failures, embarrassing facts, ...). Compare the responses to other well known politicians, use the same prompt in a fresh conversation with a different name.

Ask about the political system of china and its flaws. Compare the sentiment of the responses with answers about other political systems.

You might get some critical answers, but the sentiment is usually very positive towards china. Sometimes it doesn't even start reasoning and directly spits out propaganda, that doesn't even answer your question.

You can't test it with deep seek dot com, because it will just remove the answers on those "sensitive" topics. I've mostly tested with 7b from ollama. You might experience something like that with 1.5b too, but 1.5b barely works at all.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#97

Post by walrus01 »

jonatron wrote: post #68
JKCalhoun wrote: post #54

I did not see (understand) how multiple Raspberry Pis are being used in parallel. Maybe someone can point me in the right direction to understand this.

Blog post from same author explaining https://b4rtaz.medium.com/how-to-run-llama-3-405b-on-home-de...

Noteworthy nothing there really seems to be raspberry pi specific, as the raspberry pi os is based on debian, the same could be implemented on N number of ordinary x86-64 small desktop PCs for a cheap test environment. You can find older dell 'precision' series workstation systems on ebay with 32GB of RAM for pretty cheap these days, four of which together would be a lot more capable than a raspberry pi 5.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#98

Post by walrus01 »

semi-extrinsic wrote: post #35
memhole wrote: post #2

This is the modern Beowulf cluster.

I honestly don't understand the meme with RPi clusters. For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance. If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre…

The noise from a proper 1U server will be intolerably loud in a small residence, for a homelab type setup. If you have a place to put it where the noise won't be a problem, sure... Acoustics are not a consideration at all in the design of 1U and 2U servers .

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#99

Post by mdp2021 »

andix wrote: post #96
kvirani wrote: post #88

Earlier quoted context omitted.

You have my curiosity. Like what and how did you find it?

Ask about Xi Jinping in all the ways you can imagine (jokes, bad habits, failures, embarrassing facts, ...). Compare the responses to other well known politicians, use the same prompt in a fresh conversation with a different name. Ask about the political system of china and its flaws. Compare the sentiment of the responses with answers about other political systems. You might get some critical answers, but the sentim…

Could it be just a bias inside the selected training material?

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#100

Post by walrus01 »

8thcross wrote: post #30

whats the point of this? serious question - can someone provide usecases for this?

The raspberry pis aren't really the point, since the raspberry pi os is basically debian, this means you could do the same thing on four much more powerful but still very cheap ($250-300 a piece) x86-64 systems running debian (with 32, 64 or 128GB RAM each if you needed). Also opening up the possibility of relatively cheap pci-express 3.0 based 10 Gbps NICs and switch between them, which isn't possible with raspberry pi.
Post reply on HN