When can I "apt-get install" all this fancy new AI stuff?
On my mac:
brew install ollama
Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
41–50 of 162 posts
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#42When can I "apt-get install" all this fancy new AI stuff?
"ollama pull" is pretty close
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#43When can I "apt-get install" all this fancy new AI stuff?
[deleted]
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#44Does adding memory help? There's a Rpi 5 with 16GB RAM recently available.
The 16 GB Pi 5 comes and goes. I was able to snag one recently when Adafruit got a delivery in — then they sold right out again.
But, yeah, performance aside, there are models that Ollama won't run at all as they need more than 8GB to run.
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#45When can I "apt-get install" all this fancy new AI stuff?
Once either your current distro starts packaging any LLM tool or when you chose a different distro.
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#46Can’t you run larger models easily on MacBook Pro laptops with the bigger memory options? I think I read that people are getting 100 tokens a second on 70B models.
I find 100tps unlikely, I see 13tps on an 80GB A100 for a 70B 4bit quantized model. Can you link - I am answering an email on monday where this info would be very useful!
People on Apple Silicon are running special models for MLX. Maybe start with the MLX Community page: https://huggingface.co/mlx-community
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#47This is the modern Beowulf cluster.
I honestly don't understand the meme with RPi clusters. For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance. If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre…
You can buy it but you can't run it, unless you're fairly wealthy.
In my country (italy) a basic colocation service is like 80 euros/month + vat, and that only includes 100Wh of power and a 100mbps connection. +100wh/month upgrades are like +100 euros.
I looked up the kind of servers and cpus you're talking about and the cpu alone can pull something like 180W/h, without accounting for fans, disks and other stuff (stuff like GPUs, which are power hungry).
Yeah you could run it at home in theory, but you'll end up paying power at consumer price rather than datacenter pricing (and if you live in a flat, that's going to be a problem).
Unless you're really wealthy, you have your own home with sufficient power[1] delivery and cooling.
[1] not sure where you live, but here most residential power connections are below 3 KWh.
If otherwise you can point me at some datacenter that will let me run a normal server like the ones you're pointing at for like 100-150 euros/month, please DO let me know and i'll rush there first thing next business day and I will be throwing money at them.
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#48This is the modern Beowulf cluster.
I honestly don't understand the meme with RPi clusters. For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance. If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre…
Same. Also I don't get using RPis for hosting all kinds of services at home - there are hundreds of mini-pcs on ebay cheaper than Rpi, with more power, more ports, where you can put in a normal SSD drive and RAM, with sturdy factory made enclosure... To me Rpi seems a weird choice unless you are tinkering with the gpio ports.
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#49Earlier quoted context omitted.
I honestly don't understand the meme with RPi clusters. For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance. If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre…
The TDP of 4 PIs combined is still smaller than a larger server, which is probably the whole point of such an experiment?
The combined TDP of 4 raspberry PIs is likely less than what the fans of that kind of server pull from the power outlet.
Re: Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
#50Earlier quoted context omitted.
I honestly don't understand the meme with RPi clusters. For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance. If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre…
You can buy it but you can't run it, unless you're fairly wealthy. In my country (italy) a basic colocation service is like 80 euros/month + vat, and that only includes 100Wh of power and a 100mbps connection. +100wh/month upgrades are like +100 euros. I looked up the kind of servers and cpus you're talking about and the cpu alone can pull something like 180W/h, without accounting for fans, disks and other stuff (stu…
> You can buy it but you can't run it, unless you're fairly wealthy.
Why do I need a colocation service to put a used 1U server from eBay in my house? I'd just plug it in, much like any other PC tower you might run at home.
> Unless you're really wealthy, you have your own home with sufficient power[1] delivery and cooling.
> not sure where you live, but here most residential power connections are below 3 KWh.
It's a single used 1U server, not a datacenter... It will plug into your domestic powersupply just fine. The total draw will likely be similar or even less than many gaming PC builds out there, and even then only when under peak loads etc.