Okay but does any one actually _want_ a reasoning model at such low tok/sec speeds?!
It's local?
Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
51–60 of 162 posts
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#52That's not a bad result, although for £320 for 4x Pi5s you could probably find a used 12GB 3080 and probably more than 10x token speed
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#53All these have quickly become this generation’s “can it run Doom?”
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#54I did not see (understand) how multiple Raspberry Pis are being used in parallel. Maybe someone can point me in the right direction to understand this.
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#55Earlier quoted context omitted.
It is shown running on 2 or 4 raspberry pis; the point is that you can add more (ordinary, non GPU) hardware for faster inference. It's a distributed system. The sky is the limit.
ah, Thanks! but what can a distributed system like this do? is this a fun to do, for the sake of doing it project or does it have practical applications? just curious about applicability thats all.
I'm going to get downvoted for saying the B-word, but I imagine this growing up into some kind of blockchain thing where the AI has some goal and once there's consensus that some bit of data would further that goal it goes in a block on the chain (which is then referenced by humans who also have that goal and also is used to fine tune the AI for the next round of inference). Events in the real world are slow enough that gradually converging on the next move over the course of a few days would probably be fine.
The advantage over centralizing the compute is that you can just connect your node and start contributing to the cause (both by providing compute and by being its eyes and hands out there in the real world), there's no confusion over things like who is paying the cloud compute bill and nobody has invested overmuch in hardware.
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#56When can I "apt-get install" all this fancy new AI stuff?
Not apt-get per se, but most of the infra components underlying the "AI stuff" can be `conda install`-ed.
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#57Earlier quoted context omitted.
I really don't like that these models can be branded as Deepseek R1.
Well, Deepseek trained them?
Yes, but it would've been nice to call them D1-something, instead of constantly having to switch back and forth between Deepseek R1 (here I mean the 604B model) as distinguished from Deepseek R1 (the reasoning model and it's distillates.)
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#58That's not a bad result, although for £320 for 4x Pi5s you could probably find a used 12GB 3080 and probably more than 10x token speed
> Deepseek R1 Distill 8B Q40 on 1x 3080, 60.43 tok/s (eval 110.68 tok/s)
That wouldn't get on Hacker News ;-)
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#59On cpu or the NPU “AI” hat?
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#60Does adding memory help? There's a Rpi 5 with 16GB RAM recently available.
Inference speed is heavily dependent on memory read/write speed versus size. As long as you can fit the model in memory, what’ll determine functionality is the mem bandwidth.