[dead]
Petals: Run LLMs at home, BitTorrent-style
31–40 of 42 posts
Re: Petals: Run LLMs at home, BitTorrent-style
#32Hm, looks like this is an old project (2022) associated with HuggingFace ( https://huggingface.co/bigscience ). Doesn't look like it's very active nowadays: https://github.com/bigscience-workshop/petals Article with more details: https://techcrunch.com/2022/12/20/petals-is-creating-a-free-... > [...] volunteers can donate their hardware power to tackle a portion of a text-generating workload and team up others to com…
Maybe it's time to refresh this idea then. There's a clear demand for running LLMs at home, but the good ones are too big, and you're not using it most of the time. Sharing with others sounds like a perfect solution to me.
Re: Petals: Run LLMs at home, BitTorrent-style
#33It's an interesting concept but the timing is probably too early. If more people had reliable low latency gigabit or ideally 10gbit throughput, then it might start to approach feasibility. There's other blockers too, but that springs to mind immediately. It's cool to imagine a planet wide neural network interconnected with fiber - the nervous system of a planetary intelligence. But perhaps mushrooms do that already (…
Relevant: Why Switzerland has 25 Gbit internet and America doesn't https://news.ycombinator.com/item?id=47652400
The discussion points out, though, that many places in the US do have those speeds, while Switzerland also has places with much slower speeds.
Re: Petals: Run LLMs at home, BitTorrent-style
#34Hm, looks like this is an old project (2022) associated with HuggingFace ( https://huggingface.co/bigscience ). Doesn't look like it's very active nowadays: https://github.com/bigscience-workshop/petals Article with more details: https://techcrunch.com/2022/12/20/petals-is-creating-a-free-... > [...] volunteers can donate their hardware power to tackle a portion of a text-generating workload and team up others to com…
That explains the very old models mentioned on the page. There are models you can run locally on anything that would be more useful than llama 2 70B. e.g. the new Prism Bonsai 27B ternary model is 6GB, runs on any modern Mac, any PC with a gaming GPU bigger than 8GB (12GB or 16GB would be better), and probably runs on CPU faster than this runs any of those models. Gemma 4 E2B or E4B or 12B in the QAT quantization are fantastic for their size, and run on a lot of stuff. Models at every level have gotten a lot better since 2022.
Re: Petals: Run LLMs at home, BitTorrent-style
#35[flagged]
Re: Petals: Run LLMs at home, BitTorrent-style
#36Petals is from 2022. Nowadays intelligence of smaller models, quantization techs, and optimizations to run models faster on consumer GPUs have improved a lot. For distributed inference of smaller LLMs and diffusion models that fits in one consumer GPU rather than splits on multiple machines, there are already pretty good solutions such as AI Horde (formerly Stable Horde) [0]. Notably, it's the default provider that p…
> For distributed inference of smaller LLMs and diffusion models that fits in one consumer GPU rather than splits on multiple machines
But isn't what everyone want almost the complete opposite of this? For smaller LLMs you can already run them with consumer hardware, sometimes consumer hardware 2x, but it's the rest of the stuff where distributed inference could actually "democratize" access, as almost no one has 500GB VRAM available at home, or whatever the latest SOTA local models take nowadays.
Re: Petals: Run LLMs at home, BitTorrent-style
#37The recently discussed https://meshllm.cloud/ is the one I've been playing with but don't have the hardware to try with a model split between nodes, which apparently is supported and just not part of the public demo.
Can't even scroll the website on mobile, too much going on
https://public.meshllm.cloud/ - This should give the status of their public mesh. On an average I notice around 10 nodes on their public mesh serving various SLMs. You can use meshllm to share your GPUs like SETI@Home. Wondering how security is handled at a random node.
Re: Petals: Run LLMs at home, BitTorrent-style
#38It's an interesting concept but the timing is probably too early. If more people had reliable low latency gigabit or ideally 10gbit throughput, then it might start to approach feasibility. There's other blockers too, but that springs to mind immediately. It's cool to imagine a planet wide neural network interconnected with fiber - the nervous system of a planetary intelligence. But perhaps mushrooms do that already (…
A planetwide hallucination – sounds awful.
Re: Petals: Run LLMs at home, BitTorrent-style
#39The reason i want to run models at home is to be able to experiment on them, change things, see how they get better or worse.
Re: Petals: Run LLMs at home, BitTorrent-style
#40Earlier quoted context omitted.
Relevant: Why Switzerland has 25 Gbit internet and America doesn't https://news.ycombinator.com/item?id=47652400
The discussion points out, though, that many places in the US do have those speeds, while Switzerland also has places with much slower speeds.
I would say affordable 25gbit symmetric for residential is vanishingly rare in the US.