Petals: Run LLMs at home, BitTorrent-style
1–10 of 42 posts
Re: Petals: Run LLMs at home, BitTorrent-style
#2Not gonna work, will be starved by inter node bandwidth.
Re: Petals: Run LLMs at home, BitTorrent-style
#3Hm, looks like this is an old project (2022) associated with HuggingFace (https://huggingface.co/bigscience).
Doesn't look like it's very active nowadays: https://github.com/bigscience-workshop/petals
Article with more details: https://techcrunch.com/2022/12/20/petals-is-creating-a-free-...
> [...] volunteers can donate their hardware power to tackle a portion of a text-generating workload and team up others to complete larger tasks, similar to Folding@home and other distributed compute setups.
Re: Petals: Run LLMs at home, BitTorrent-style
#4It's an interesting concept but the timing is probably too early. If more people had reliable low latency gigabit or ideally 10gbit throughput, then it might start to approach feasibility. There's other blockers too, but that springs to mind immediately.
It's cool to imagine a planet wide neural network interconnected with fiber - the nervous system of a planetary intelligence. But perhaps mushrooms do that already (Alpha Centauri ever relevant).
Re: Petals: Run LLMs at home, BitTorrent-style
#5I have been thinking of something on these lines but with much smaller models. The entire model has to fit on a single computer. Host owner would choose the model they prefer, perhaps because they already use it. Then it is more about utilizing the GPU for LLM requests.
Peer to peer, consumers get to route their request to a host with compatible model. Consumers have to also contribute GPU but it does not have to be equal - I have not thought through the fairness part. Perhaps initially it starts with "create your friends group and have access to all the host nodes".
Re: Petals: Run LLMs at home, BitTorrent-style
#6I have been thinking of something on these lines but with much smaller models. The entire model has to fit on a single computer. Host owner would choose the model they prefer, perhaps because they already use it. Then it is more about utilizing the GPU for LLM requests. Peer to peer, consumers get to route their request to a host with compatible model. Consumers have to also contribute GPU but it does not have to be…
Can't wait to run a Sybil farm of subsidized proxy nodes and log all the juicy prompts.
Re: Petals: Run LLMs at home, BitTorrent-style
#7Petals is from 2022. Nowadays intelligence of smaller models, quantization techs, and optimizations to run models faster on consumer GPUs have improved a lot.
For distributed inference of smaller LLMs and diffusion models that fits in one consumer GPU rather than splits on multiple machines, there are already pretty good solutions such as AI Horde (formerly Stable Horde) [0]. Notably, it's the default provider that powers SillyTavern. It also has an interesting economy model of kudos.
Re: Petals: Run LLMs at home, BitTorrent-style
#8I have been thinking of something on these lines but with much smaller models. The entire model has to fit on a single computer. Host owner would choose the model they prefer, perhaps because they already use it. Then it is more about utilizing the GPU for LLM requests. Peer to peer, consumers get to route their request to a host with compatible model. Consumers have to also contribute GPU but it does not have to be…
Can't wait to run a Sybil farm of subsidized proxy nodes and log all the juicy prompts.
I do not know what/who Sybil is but yes, there is a lot of plumbing in order to make sure that host nodes are ZDR* compliant and much more. Zero visibility of source prompt.
Also, this is the reason I want to start with trust based groups only - invite people you already know. I have friends who have Macs with 32GB or 48GB unified memory but sending prompts to them is not a out of the box thing.
* typo
Re: Petals: Run LLMs at home, BitTorrent-style
#9I have been thinking of something on these lines but with much smaller models. The entire model has to fit on a single computer. Host owner would choose the model they prefer, perhaps because they already use it. Then it is more about utilizing the GPU for LLM requests. Peer to peer, consumers get to route their request to a host with compatible model. Consumers have to also contribute GPU but it does not have to be…
Can't wait to run a Sybil farm of subsidized proxy nodes and log all the juicy prompts.
AI Horde has some measures to prevent Sybil attack that returns wrong results, but not enforce zero data retention. Prompts belong to the whole open source community. For example https://huggingface.co/datasets/la-ji/sd-prompt-in-the-wild
Re: Petals: Run LLMs at home, BitTorrent-style
#10It's an interesting concept but the timing is probably too early. If more people had reliable low latency gigabit or ideally 10gbit throughput, then it might start to approach feasibility. There's other blockers too, but that springs to mind immediately. It's cool to imagine a planet wide neural network interconnected with fiber - the nervous system of a planetary intelligence. But perhaps mushrooms do that already (…
Relevant: Why Switzerland has 25 Gbit internet and America doesn't https://news.ycombinator.com/item?id=47652400