Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

github.com

Post reply on HN
Print view View original

111–120 of 162 posts

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#111

Post by vel0city »

znpy wrote: post #47
semi-extrinsic wrote: post #35

Earlier quoted context omitted.

I honestly don't understand the meme with RPi clusters. For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance. If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre…

You can buy it but you can't run it, unless you're fairly wealthy. In my country (italy) a basic colocation service is like 80 euros/month + vat, and that only includes 100Wh of power and a 100mbps connection. +100wh/month upgrades are like +100 euros. I looked up the kind of servers and cpus you're talking about and the cpu alone can pull something like 180W/h, without accounting for fans, disks and other stuff (stu…

Just a note, you're mixing kW and kWh.

A connection to a home wouldn't be rated in kilowatt-hours, it would likely be rated in amps, but could also be expressed in kilowatts.

> 100wh/month upgrades are like +100 euros.

I can't imagine anybody paying €1/Wh. Even if this was €1/kWh (1000x cheaper) it's still a few times more expensive than what most places would consider expensive.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#112

Post by czk »

is it just me or does calling these distilled models 'DeepSeek R1' seem to be a gross misrepresentation of what they actually are

people think they can run these tiny models distilled from deepseek r1 and are actually running deepseek r1 itself

its kinda like if you drove a civic with a tesla bodykit and said it was a tesla

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#113

Post by scubbo »

IAmGraydon wrote: post #83
alexhjones wrote: post #78

Earlier quoted context omitted.

Never heard that claim before, only that a certain subset of re-enforced learning may have used ChatGPT to grade responses. Is there more detail about it being allegedly a distilled OpenAI model?

He didn't say it's a distilled OpenAI model. He said it's a distillation of an OpenAI model. They are not at all the same thing.

How so? (Genuine question, not a challenge - I hadn't heard the terms "distilled/distillation" in an AI context until this thread)

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#114

Post by PhilipRoman »

semi-extrinsic wrote: post #85
PhilipRoman wrote: post #61

Earlier quoted context omitted.

Interesting, I run a small cluster of 4 mini pcs (22 cores total). I think it should be comparable to the aforementioned EPYC. Power load is a rounding error compared to appliances like electric iron at 1700W, etc. The impact on electrical bill is minimal as well. Idle power draw is about 5W per server, which translates to ~80 cents a month. Frankly my monitor uses more power on average than the rest of the homelab.

I'm pretty sure if you run a compute benchmark like Streams or hpgmg, the Epyc server will eat your mini pcs for breakfast.

You're probably right, I meant that the power consumption should be roughly comparable between them (due to inefficiency added by each mini).

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#115

Post by emaciatedslug »

mdp2021 wrote: post #108
andix wrote: post #104

Earlier quoted context omitted.

Feel free to call propaganda a bias if you like. But if it walks like a duck, quacks like a duck, ...

This is HN: my focus is technical (here specifically), maybe "technical" in world assessment and future prediction (in other pages). I.e.: I am just trying to understand the facts.

Yes, in some ways the output is based on training material. The deep learning model will find the "ground truth" of the corpus in theory. But China's political enforcement since the "great firewall of china" was instituted, 2 and a half decades ago, have directly or indirectly made content scraped from any Chinese site bias by default. The whole Tienanmen Square meme isn't a meme because it is funny, it is a meme because it consequentially qualifies the discrepancy between the CCP and it's own history. Sure there is bias in all models, but a quantized version will only loose accuracy.. but if a distillation process used a teacher LLM without the censorship bias discussed (i.e., a teacher trained on a more open and less politically manipulated dataset), the resulting distilled student LLM would, in most important respects, be more accurate and significantly more useful in a broader sense in theory but is seems not to matter based on my limited query. I have deepseek-r1-distill-llama-8b installed on LM Studio....if I ask "where is Tienanmen square and what is it's significance?" i get this:

I am sorry, I cannot answer that question. I am an AI assistant designed to provide helpful and harmless responses.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#116

Post by blackeyeblitzar »

alexhjones wrote: post #78
HPsquared wrote: post #70

Earlier quoted context omitted.

And DeepSeek itself is (allegedly) a distillation of OpenAI models.

Never heard that claim before, only that a certain subset of re-enforced learning may have used ChatGPT to grade responses. Is there more detail about it being allegedly a distilled OpenAI model?

https://www.newsweek.com/openai-warns-deepseek-distilled-ai-...

There are many sources and discussions on this. Also DeepSeek recently changed their responses to hide references to various OpenAI things after all this came out, which is weird.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#117

Post by replete »

geerlingguy wrote: post #71
replete wrote: post #52

That's not a bad result, although for £320 for 4x Pi5s you could probably find a used 12GB 3080 and probably more than 10x token speed

Or attach a 12 or 16 GB GPU to a single Pi 5 directly, and get 20+ tokens/s on an even larger model :D https://github.com/geerlingguy/ollama-benchmark?tab=readme-o...

Thanks for sharing. Pi5 + cheap AMD GPU = convenient modest LLM api server? ...if you find the right magic rocm incantations I guess

Double thanks for the 3rd party mac mini SSD tip - eagerly awaiting delivery!

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#118

Post by lr1970 »

dcre wrote: post #75
hangonhn wrote: post #69

Earlier quoted context omitted.

Can you explain to a non ML software engineer what these distillation methods mean? What does it mean to have R1 train a Llama model? What is special about DeepSeek’s distillation methods? Thanks!

Distilling means fine-tuning an existing model using outputs from the bigger model. The special technique is in the details of what you choose to generate from the bigger model, how long to train for, and a bunch of other nitty gritty stuff I don’t know about because I’m also not an ML engineer. Google it!

> Distilling means fine-tuning an existing model using outputs from the bigger model.

Crucially, the output of the teacher model includes token probabilities so that the fine-tuning is trying to learn the entire output distribution.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#119

Post by ziml77 »

littlestymaar wrote: post #79
geerlingguy wrote: post #71

Earlier quoted context omitted.

Or attach a 12 or 16 GB GPU to a single Pi 5 directly, and get 20+ tokens/s on an even larger model :D https://github.com/geerlingguy/ollama-benchmark?tab=readme-o...

Reading the beginning of your comment I was like “ah yes I saw Jeff Geerling do that on a video”. Then I saw you github link and your HN handle and I was like “Wait, it is Jeff Geerling!”. :D

Haha I had nearly the same thing happen. First I was like "that sounds like something Jeff Geerling would do". Then I saw the github link and was like "ah yeah Jeff Geerling did do it" and then I saw the username and was like "oh it's Jeff Geerling!"

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#120

Post by 01100011 »

mdp2021 wrote: post #99
andix wrote: post #96

Earlier quoted context omitted.

Ask about Xi Jinping in all the ways you can imagine (jokes, bad habits, failures, embarrassing facts, ...). Compare the responses to other well known politicians, use the same prompt in a fresh conversation with a different name. Ask about the political system of china and its flaws. Compare the sentiment of the responses with answers about other political systems. You might get some critical answers, but the sentim…

Could it be just a bias inside the selected training material?

Companies probably do several things(at least I would if it were me):

- The pre-training dataset is sanitized culturally/politically and pro-regime material is added.

- Supervised fine tuning dataset provides further enforcement of these biases.

- The output is filtered to prevent hallucinations from resulting in anything offensive to the regime. This could(?) also prevent the reasoning loop from straying into ideologically dangerous territory.

So you have multiple opportunities to bend to the will of the authorities.

Post reply on HN