The interesting thing here is being able to run llama inference in a distributed fashion across multiple computers.
Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
11–20 of 162 posts
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#12Does adding memory help? There's a Rpi 5 with 16GB RAM recently available.
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#13Okay but does any one actually _want_ a reasoning model at such low tok/sec speeds?!
You can have questions that are not urgent. It's like Cursor, I'm fine with the slow version until a certain point, I launch the request then I alt-tab to something else.
Yes it's slower, but well, for free (or cheap) it is acceptable.
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#14Can’t you run larger models easily on MacBook Pro laptops with the bigger memory options? I think I read that people are getting 100 tokens a second on 70B models.
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#15Does adding memory help? There's a Rpi 5 with 16GB RAM recently available.
Memory capacity in itself doesn't help so long as the model+context fits in memory (and and 8B parameter Q4 model should fit in a single 8 GB Pi).
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#16Does adding memory help? There's a Rpi 5 with 16GB RAM recently available.
Memory capacity in itself doesn't help so long as the model+context fits in memory (and and 8B parameter Q4 model should fit in a single 8 GB Pi).
Is there a back-of-the-napkin way to calculate how much memory a given model will take? Or what parameter/quantization model will fit in a given memory size?
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#17Can’t you run larger models easily on MacBook Pro laptops with the bigger memory options? I think I read that people are getting 100 tokens a second on 70B models.
haven’t measured but the 70b runs fine on an m1 macbook pro with 64gb of ram, although you can’t do much until it has finished
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#18Okay but does any one actually _want_ a reasoning model at such low tok/sec speeds?!
Only interactive uses cases need high tps, if you just want a process running somewhere ingesting and synthesizing data it’s fine. It’s done when it’s done.
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#19Earlier quoted context omitted.
Memory capacity in itself doesn't help so long as the model+context fits in memory (and and 8B parameter Q4 model should fit in a single 8 GB Pi).
Is there a back-of-the-napkin way to calculate how much memory a given model will take? Or what parameter/quantization model will fit in a given memory size?
q4=4bits per weight
So Q4 8B would be ~4GB.