Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

github.com

Post reply on HN
Print view View original

151–160 of 162 posts

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#151

Post by dcre »

sumedh wrote: post #136
dcre wrote: post #75

Earlier quoted context omitted.

Distilling means fine-tuning an existing model using outputs from the bigger model. The special technique is in the details of what you choose to generate from the bigger model, how long to train for, and a bunch of other nitty gritty stuff I don’t know about because I’m also not an ML engineer. Google it!

> Google it! Or you could provide some example links

This only makes sense if I have some great canonical explanation of distillation on hand. But it’s a simple concept. There are hundreds of identical explanations online.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#152

Post by amelius »

diggan wrote: post #148
dheera wrote: post #89

Earlier quoted context omitted.

Why tf isn't ollama in apt-get yet? F these curl|sh installs.

> Why tf isn't ollama in apt-get yet? There are three broad groups of people in packaging: A) People who package stuff you and others need. B) People who don’t package stuff but use what’s available. C) People who don’t package stuff and complain about what’s available without taking further action. If you find yourself in group C while having no interest in contributing to group A or working within the limits of gro…

This is not entirely fair. We can't all be involved in package management, and GP might contribute in other ways.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#153

Post by ninetyninenine »

johntash wrote: post #141
ninetyninenine wrote: post #82

Really there needs to be a product based off of LLMs similar to Alexa or Google home where instead of connecting to the cloud it’s a locally run LLM. I don’t know why one doesn’t exist yet or why no one is working on this

You can kind of get there with home assistant. I'm not sure if it can use tools yet, but you can expose stuff you'd ask about like the weather/etc.

I have both and the one offered is really weak. If you pay you can use Gemini but it doesn’t have agentic access to smart home controls meaning you can no longer ask it to turn on or off the lights

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#154

Post by aljarry »

talldayo wrote: post #110
agilob wrote: post #102

Earlier quoted context omitted.

Some of us also worry about energy consumption

They idle at pretty low wattages, and since the bulk of the TDP is rated for raster workloads you usually won't see them running at full-power on compute workloads. My 300w 3070ti doesn't really exceed 100w during inference workloads. Boot up a 1440p video game and it's a different story altogether, but for inference and transcoding those 3060s are some of the most power efficient options on the consumer market.

Interesting, my 3060 uses 150-170W with 14B model on Ollama, according to nvidia-smi.

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#155

Post by beefnugs »

corysama wrote: post #93
hangonhn wrote: post #69

Earlier quoted context omitted.

Can you explain to a non ML software engineer what these distillation methods mean? What does it mean to have R1 train a Llama model? What is special about DeepSeek’s distillation methods? Thanks!

"Quantized" models try to approximate the full model using less bits. "Distilled" models are other models (Llama, Qwen) that have been put through an additional training round using DeepSeek as a teacher.

What is the effect of this less bits? Is it like truncating hashes where you start going off into the wrong thing entirely, or more like less accuracy so that if you are talking about soft penguins it will start thinking you mean wet penguins?

And is there a domain specific term I can look into if I wanted to read about someone trying to keep all the bits, but the runtime (trying to save ram) focusing in on parts of the data instead of this quantization?

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#156

Post by andix »

menaerus wrote: post #146
andix wrote: post #135

Earlier quoted context omitted.

Sorry, it felt to me like you're trying to troll. Those behaviours are extremely likely intentionally added. I can't prove it, but the responses read like they are from a propaganda text book. Not the nuanced new fashioned kind of propaganda from social media, but classic blunt and authoritarian style. You really notice it from the answers. The output token come really fast, at least 3 times faster than in any other…

So we get a new model that is one of the ~two best performing models on the market, and yet we are not discussing its technical capabilities but rather its inclination towards the history events. Sorry, I don't get this obsession.

Responsible AI is a really important aspect of it. Maybe even the most important. Look at what social media did to us.

And you were the one starting the discussion ;)

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#157

Post by jmorgan »

dheera wrote: post #89
dzikimarian wrote: post #42

Earlier quoted context omitted.

"ollama pull" is pretty close

Why tf isn't ollama in apt-get yet? F these curl|sh installs.

This is a great point. apt-get would definitely be a better install experience and upgrade experience (that's what I would want too). Tailscale does this amazing well: https://tailscale.com/download/linux

The main issue for the maintainer team would be the work in hosting and maintaining all the package repos for apt, yum, etc, and making sure the we handle the case where nvidia/amd drivers aren't installed (quite common on cloud VMs). Mostly a matter of time and putting in the work.

For now every release of Ollama includes a minimal archive with the ollama binary and required dynamic libraries: https://github.com/ollama/ollama/blob/main/docs/linux.md#man.... But we could definitely do better

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#158

Post by corysama »

beefnugs wrote: post #155
corysama wrote: post #93

Earlier quoted context omitted.

"Quantized" models try to approximate the full model using less bits. "Distilled" models are other models (Llama, Qwen) that have been put through an additional training round using DeepSeek as a teacher.

What is the effect of this less bits? Is it like truncating hashes where you start going off into the wrong thing entirely, or more like less accuracy so that if you are talking about soft penguins it will start thinking you mean wet penguins? And is there a domain specific term I can look into if I wanted to read about someone trying to keep all the bits, but the runtime (trying to save ram) focusing in on parts of…

I'm pretty far from an expert. But, at it's core ML is a bunch of matrix multiplications glued together with non-linear functions. So, quantization leads to less accuracy in the matrices of weights. Not, changes in hashes where 1 wrong bit is meaningless.

The folks who quantized DeepSeek say they used a piece of tech called "BitsAndBytes". https://unsloth.ai/blog/dynamic-4bit

Googling around for "bitsandbytes ai quantization" turns up this article which looks nice

https://generativeai.pub/practical-guide-of-llm-quantization...

Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5

#160

Post by menaerus »

andix wrote: post #156
menaerus wrote: post #146

Earlier quoted context omitted.

So we get a new model that is one of the ~two best performing models on the market, and yet we are not discussing its technical capabilities but rather its inclination towards the history events. Sorry, I don't get this obsession.

Responsible AI is a really important aspect of it. Maybe even the most important. Look at what social media did to us. And you were the one starting the discussion ;)

I wasn't, you're confusing me with someone else from this thread. It's literally my first comment here.
Post reply on HN