Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

github.com

Post reply on HN
Print view View original

121–130 of 162 posts

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#121

Post by pockmarked19 »

btown wrote: post #87
tofof wrote: post #4

This continues the pattern of all other announcements of running 'Deepseek R1' on raspberry pi - that they are running llama (or qwen), modified by deepseek's distillation technique.

Specifically, I've seen that a common failure mode of the distilled Deepseek models is that they don't know when they're going in circles. Deepseek incentivizes the distilled LLM to interrupt itself with "Wait." which incentivizes a certain degree of reasoning, but it's far less powerful than the reasoning of the full model, and can get into cycles of saying "Wait." ad infinitum, effectively second-guessing itself on…

The full model also gets into these infinite cycles. I just tried asking the old river crossing boat problem but with two goats and a cabbage and it goes on and on forever.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#122

Post by numba888 »

lr1970 wrote: post #118
dcre wrote: post #75

Earlier quoted context omitted.

Distilling means fine-tuning an existing model using outputs from the bigger model. The special technique is in the details of what you choose to generate from the bigger model, how long to train for, and a bunch of other nitty gritty stuff I don’t know about because I’m also not an ML engineer. Google it!

> Distilling means fine-tuning an existing model using outputs from the bigger model. Crucially, the output of the teacher model includes token probabilities so that the fine-tuning is trying to learn the entire output distribution.

That's possible only if they use the same tokens. Which likely requires they share the same tokenizer. Not sure that's the case here, R1 was built on OpenAI closed model's output.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#123

Post by geerlingguy »

replete wrote: post #117
geerlingguy wrote: post #71

Earlier quoted context omitted.

Or attach a 12 or 16 GB GPU to a single Pi 5 directly, and get 20+ tokens/s on an even larger model :D https://github.com/geerlingguy/ollama-benchmark?tab=readme-o...

Thanks for sharing. Pi5 + cheap AMD GPU = convenient modest LLM api server? ...if you find the right magic rocm incantations I guess Double thanks for the 3rd party mac mini SSD tip - eagerly awaiting delivery!

llama.cpp runs great with Vulkan, so no ROCm magic required!

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#124

Post by numba888 »

littlestymaar wrote: post #81
tofof wrote: post #4

This continues the pattern of all other announcements of running 'Deepseek R1' on raspberry pi - that they are running llama (or qwen), modified by deepseek's distillation technique.

Meanwhile on /r/localllama, people are running the full R1 on CPU with NVMe drives in lieu of VRAM.

Did they get the first token out? ;) Just curious, NVidia ported it, and they claim almost 4 tokens/sec on 8xH100 server. At this performance there are much cheaper option.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#125

Post by numba888 »

NitpickLawyer wrote: post #3

As always, take those t/s stats with a huge boulder of salt. The demo shows a question "solved" in < 500 tokens. Still amazing that it's possible, but you'll get nowhere near those speeds when dealing with real-world problems at real-world useful context lengths for "thinking" models (8-16k tokens). Even epyc's with lots of channels go down to 2-4 t/s after ~4096 context length.

Smaller robots tend to have smaller problems. Even little help from the model will make them a lot more capable than they are today.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#126

Post by anon373839 »

numba888 wrote: post #122
lr1970 wrote: post #118

Earlier quoted context omitted.

> Distilling means fine-tuning an existing model using outputs from the bigger model. Crucially, the output of the teacher model includes token probabilities so that the fine-tuning is trying to learn the entire output distribution.

That's possible only if they use the same tokens. Which likely requires they share the same tokenizer. Not sure that's the case here, R1 was built on OpenAI closed model's output.

That was an (as far as I can tell) unsubstantiated claim made by OpenAI. It doesn’t even make sense, as o1’s reasoning traces are not provided to the user.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#127

Post by Ray20 »

semi-extrinsic wrote: post #85
PhilipRoman wrote: post #61

Earlier quoted context omitted.

Interesting, I run a small cluster of 4 mini pcs (22 cores total). I think it should be comparable to the aforementioned EPYC. Power load is a rounding error compared to appliances like electric iron at 1700W, etc. The impact on electrical bill is minimal as well. Idle power draw is about 5W per server, which translates to ~80 cents a month. Frankly my monitor uses more power on average than the rest of the homelab.

I'm pretty sure if you run a compute benchmark like Streams or hpgmg, the Epyc server will eat your mini pcs for breakfast.

I think results would be rather comparable.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#128

Post by int_19h »

anon373839 wrote: post #126
numba888 wrote: post #122

Earlier quoted context omitted.

That's possible only if they use the same tokens. Which likely requires they share the same tokenizer. Not sure that's the case here, R1 was built on OpenAI closed model's output.

That was an (as far as I can tell) unsubstantiated claim made by OpenAI. It doesn’t even make sense, as o1’s reasoning traces are not provided to the user.

One reason to believe OpenAI here is that R1 will occasionally claim to be made by OpenAI, which in e.g. LLaMA finetunes is indicative of using synthetic data generated by ChatGPT.

Note that this isn't necessarily o1. While o1 is specifically trained to do CoT, you can also make 4o etc produce it with the appropriate prompts, and then train on that output.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#130

Post by semi-extrinsic »

Ray20 wrote: post #127
semi-extrinsic wrote: post #85

Earlier quoted context omitted.

I'm pretty sure if you run a compute benchmark like Streams or hpgmg, the Epyc server will eat your mini pcs for breakfast.

I think results would be rather comparable.

That would at the very least require a hefty interconnect, I'd guess at least 25Gbit with RoCE.
Post reply on HN