31–40 of 162 posts
whats the point of this? serious question - can someone provide usecases for this?
This continues the pattern of all other announcements of running 'Deepseek R1' on raspberry pi - that they are running llama (or qwen), modified by deepseek's distillation technique.
whats the point of this? serious question - can someone provide usecases for this?
It is shown running on 2 or 4 raspberry pis; the point is that you can add more (ordinary, non GPU) hardware for faster inference. It's a distributed system. The sky is the limit.
This is the modern Beowulf cluster.
If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre or k8s+OpenStack+Ceph or some other goodness. Spare me this doesnt-even-scale-linearly-to-four-slowass-nodes BS.
whats the point of this? serious question - can someone provide usecases for this?
It is shown running on 2 or 4 raspberry pis; the point is that you can add more (ordinary, non GPU) hardware for faster inference. It's a distributed system. The sky is the limit.
This is the modern Beowulf cluster.
I honestly don't understand the meme with RPi clusters. For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance. If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre…
Can’t you run larger models easily on MacBook Pro laptops with the bigger memory options? I think I read that people are getting 100 tokens a second on 70B models.
Can you link - I am answering an email on monday where this info would be very useful!