Run LLMs at home, BitTorrent‑style • Hacker News Board

Re: Run LLMs at home, BitTorrent‑style

#11

Post by sn0wf1re » Sun, Sep 17, 2023, 7:51 PM UTC

esafak wrote: post #2

The first question I had was "what are the economics?" From the FAQ: Will Petals incentives be based on crypto, blockchain, etc.? No, we are working on a centralized incentive system similar to the AI Horde kudos, even though Petals is a fully decentralized system in all other aspects. We do not plan to provide a service to exchange these points for money, so you should see these incentives as "game" points designed…

Similarly there have been distributed render farms for graphic design for a long time. No incentives other than higher points means your jobs are prioritized.

https://www.sheepit-renderfarm.com/home

Re: Run LLMs at home, BitTorrent‑style

#12

Post by version_five » Sun, Sep 17, 2023, 7:54 PM UTC

swyx wrote: post #9

so given that GGML can serve like 100 tok/s on an M2 Max, and this thing advertises 6 tok/s distributed, is this basically for people with lower end devices?

It's talking about 70B and 160B models. Even heavily quantized can ggml run those that fast? (I'm guessing possibly). So maybe this is for people that dont have a high end computer? I have a decent linux laptop a couple years old and there's no way I could run those models that fast. I get a few tokens per second on a quantized 7B model.

Re: Run LLMs at home, BitTorrent‑style

#13

Post by beardog » Sun, Sep 17, 2023, 7:56 PM UTC

esafak wrote: post #2

The first question I had was "what are the economics?" From the FAQ: Will Petals incentives be based on crypto, blockchain, etc.? No, we are working on a centralized incentive system similar to the AI Horde kudos, even though Petals is a fully decentralized system in all other aspects. We do not plan to provide a service to exchange these points for money, so you should see these incentives as "game" points designed…

>What's the motivation for people to host model layers in the public swarm?

>People who run inference and fine-tuning themselves get a certain speedup if they host a part of the model locally. Some may be also motivated to "give back" to the community helping them to run the model (similarly to how BitTorrent users help others by sharing data they have already downloaded).

>Since it may be not enough for everyone, we are also working on introducing explicit incentives ("bloom points") for people donating their GPU time to the public swarm. Once this system is ready, we will display the top contributors on our website. People who earned these points will be able to spend them on inference/fine-tuning with higher priority or increased security guarantees, or (maybe) exchange them for other rewards.

It does seem like they want a sort of centralized token however.

Re: Run LLMs at home, BitTorrent‑style

#14

Post by brucethemoose2 » Sun, Sep 17, 2023, 8:24 PM UTC

swyx wrote: post #8

brucethemoose2 wrote: post #3

Earlier quoted context omitted.

> similar to the AI Horde kudos What they are referencing, which is super cool and (IMO) criminally underused: https://lite.koboldai.net/ https://tinybots.net/artbot https://aihorde.net/ In fact, I can host a 13B-70B finetune in the afternoon if anyone on HN wants to test a particular one out: https://huggingface.co/models?sort=modified&search=70B+gguf

> GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp. GGUF offers numerous advantages over GGML, such as better tokenisation, and support for special tokens. It is also supports metadata, and is designed to be extensible is there a more canonical blogpost or link to learn more about the technical decisions here?

https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md#...

It is (IMO) a necessary and good change.

I just specified gguf because my 3090 cannot host a 70B model without offloading outside of exLlama's very new ~2 bit quantization. And pre quantized gguf is a much smaller download than raw fp16 for conversion.

Re: Run LLMs at home, BitTorrent‑style

#15

Post by brucethemoose2 » Sun, Sep 17, 2023, 8:29 PM UTC

version_five wrote: post #12

swyx wrote: post #9

so given that GGML can serve like 100 tok/s on an M2 Max, and this thing advertises 6 tok/s distributed, is this basically for people with lower end devices?

It's talking about 70B and 160B models. Even heavily quantized can ggml run those that fast? (I'm guessing possibly). So maybe this is for people that dont have a high end computer? I have a decent linux laptop a couple years old and there's no way I could run those models that fast. I get a few tokens per second on a quantized 7B model.

Yeah. My 3090 gets like ~5 tokens/s on 70B Q3KL.

This is a good idea, as splitting up llms is actually pretty efficient with pipelined requests.

Re: Run LLMs at home, BitTorrent‑style

#16

Post by russellbeattie » Sun, Sep 17, 2023, 8:30 PM UTC

swyx wrote: post #9

so given that GGML can serve like 100 tok/s on an M2 Max, and this thing advertises 6 tok/s distributed, is this basically for people with lower end devices?

> ...lower end devices

So, pretty much every other consumer PC available? Those losers.

Re: Run LLMs at home, BitTorrent‑style

#17

Post by jmorgan » Sun, Sep 17, 2023, 8:31 PM UTC

This is neat. Model weights are split into their layers and distributed across several machines who then report themselves in a big hash table when they are ready to perform inference or fine tuning "as a team" over their subset of the layers.

It's early but I've been working on hosting model weights in a Docker registry for https://github.com/jmorganca/ollama. Mainly for the content addressability (Ollama will verify the correct weights are downloaded every time) and ultimately weights can be fetched by their content instead of by their name or url (which may change!). Perhaps a good next step might be to split the models by layers and store each layer independently for use cases like this (or even just for downloading + running larger models over several "local" machines).

Re: Run LLMs at home, BitTorrent‑style

#18

Post by brucethemoose2 » Sun, Sep 17, 2023, 8:33 PM UTC

> and fine‑tune them for your tasks

This is the part that raised my eyebrows.

Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"

Re: Run LLMs at home, BitTorrent‑style

#19

Post by Zetobal » Sun, Sep 17, 2023, 8:35 PM UTC

brucethemoose2 wrote: post #18

> and fine‑tune them for your tasks This is the part that raised my eyebrows. Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"

An H100 is maybe a car but not nearly close to a house...

Re: Run LLMs at home, BitTorrent‑style

#20

Post by AaronFriel » Sun, Sep 17, 2023, 8:39 PM UTC

brucethemoose2 wrote: post #18

> and fine‑tune them for your tasks This is the part that raised my eyebrows. Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"

That's true for conventional fine-tuning, but is it the case for parameter efficient fine tuning and qLORA? My understanding is that for a N billion parameter model, fine tuning can occur with a slightly-less-than-N gigabyte of VRAM GPU.

For that 70B parameter model: an A100?