71–80 of 162 posts
That's not a bad result, although for £320 for 4x Pi5s you could probably find a used 12GB 3080 and probably more than 10x token speed
https://github.com/geerlingguy/ollama-benchmark?tab=readme-o...
That's not a bad result, although for £320 for 4x Pi5s you could probably find a used 12GB 3080 and probably more than 10x token speed
Earlier quoted context omitted.
I honestly don't understand the meme with RPi clusters. For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance. If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre…
You can buy it but you can't run it, unless you're fairly wealthy. In my country (italy) a basic colocation service is like 80 euros/month + vat, and that only includes 100Wh of power and a 100mbps connection. +100wh/month upgrades are like +100 euros. I looked up the kind of servers and cpus you're talking about and the cpu alone can pull something like 180W/h, without accounting for fans, disks and other stuff (stu…
This continues the pattern of all other announcements of running 'Deepseek R1' on raspberry pi - that they are running llama (or qwen), modified by deepseek's distillation technique.
Can you explain to a non ML software engineer what these distillation methods mean? What does it mean to have R1 train a Llama model? What is special about DeepSeek’s distillation methods? Thanks!
This continues the pattern of all other announcements of running 'Deepseek R1' on raspberry pi - that they are running llama (or qwen), modified by deepseek's distillation technique.
Can you explain to a non ML software engineer what these distillation methods mean? What does it mean to have R1 train a Llama model? What is special about DeepSeek’s distillation methods? Thanks!
Google it!
whats the point of this? serious question - can someone provide usecases for this?
It is shown running on 2 or 4 raspberry pis; the point is that you can add more (ordinary, non GPU) hardware for faster inference. It's a distributed system. The sky is the limit.
This continues the pattern of all other announcements of running 'Deepseek R1' on raspberry pi - that they are running llama (or qwen), modified by deepseek's distillation technique.
Can you explain to a non ML software engineer what these distillation methods mean? What does it mean to have R1 train a Llama model? What is special about DeepSeek’s distillation methods? Thanks!
It also looks to me like there was some Chinese propaganda trained into llama/quen too, but that’s just my observation.
This continues the pattern of all other announcements of running 'Deepseek R1' on raspberry pi - that they are running llama (or qwen), modified by deepseek's distillation technique.
And DeepSeek itself is (allegedly) a distillation of OpenAI models.
That's not a bad result, although for £320 for 4x Pi5s you could probably find a used 12GB 3080 and probably more than 10x token speed
Or attach a 12 or 16 GB GPU to a single Pi 5 directly, and get 20+ tokens/s on an even larger model :D https://github.com/geerlingguy/ollama-benchmark?tab=readme-o...
Then I saw you github link and your HN handle and I was like “Wait, it is Jeff Geerling!”. :D
When can I "apt-get install" all this fancy new AI stuff?