I’ve always thought crowdsourcing is the future. Crowdsourcing information or compute. The fact is we have the “resources” already. It’s a matter of deployment.
Run LLMs at home, BitTorrent‑style
101–110 of 135 posts
Re: Run LLMs at home, BitTorrent‑style
#102Are trained LLM's composable in any way? Like if you and I trust 99% of the same data, but each have 1% where we disagree, must we have two entirely separate models, or can we pool compute in the 99% case (along with the others who agree) and then create a derivative model for ourselves which covers for the differences in our trust models?
I have only a rudimentary understanding of neural nets but it doesn't seem crazy that the weights could be manipulated in such a way while preserving the utility of the model.
I ask because I think it would be useful to know which statements two LLMs of equal power agree on and which they disagree on. You could then map that backwards to differences in their training data (only feasible if the differences are small).
If instead two LLMs of equal power represent a missed opportunity to have one of greater power, and the disagreement analysis is prohibitively expensive to do, then that's a bit of a different world.
Re: Run LLMs at home, BitTorrent‑style
#103> and fine‑tune them for your tasks This is the part that raised my eyebrows. Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"
That's true for conventional fine-tuning, but is it the case for parameter efficient fine tuning and qLORA? My understanding is that for a N billion parameter model, fine tuning can occur with a slightly-less-than-N gigabyte of VRAM GPU. For that 70B parameter model: an A100?
If one is training the full 70B parameters, then the total memory usage far exceeds the memory for simply storing the 70B parameters (think derivatives and optimizer parameters such as momentum.) This is the main reason why models are split or why techniques like the fully distributed data sharing are used during training. During training of a distributed model, at every step of the optimizer these multiple-of-70B parameters need to go through a network wire (though not to all nodes, thankfully). As you suggested, LoRA could work well in a distributed setting because the trainable parameters are very small in number (tens of thousand of times less trainable parameters) and the info required to go through the network for non trainable parameters is also small. However, training this model on a single A100 is impractical as it would require mimicking a distributed training buffering things on a TB-sized CPU RAM (or slower) to swap pieces in and out of the model during every step in an otherwise distributed operation (and is not natively supported in existing frameworks to the best of my knowledge, even though one could technically write this code without too much difficulty.)
Re: Run LLMs at home, BitTorrent‑style
#104This is neat. Model weights are split into their layers and distributed across several machines who then report themselves in a big hash table when they are ready to perform inference or fine tuning "as a team" over their subset of the layers. It's early but I've been working on hosting model weights in a Docker registry for https://github.com/jmorganca/ollama . Mainly for the content addressability (Ollama will veri…
Ah, is it possible to tone down the self-promotion? I've been seeing your comments for ollama on many LLM-related posts here. > Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity. Surely in this case it would've been possible to comment about OP's work while leaving out the free backlink to your project. Just my 0.02
There is nothing wrong with self-promotion if, as in this case, it is relevant to the discussion.
Re: Run LLMs at home, BitTorrent‑style
#105Earlier quoted context omitted.
Thats not the best counterargument, because Bitcoin has privacy qualities by default. You can hop on to any block explorer and accept every address as another user, but you cant verify that (without expensive analysis, on a case-by-case basis) those are not owned by the same guy. Same with Tor, while some data like bridge usage is being collected somehow (i havent looked into it) you cant reliably prove that thousand…
It's pretty obvious that the majority of transaction volume and value is rubbish. Bots buying, selling, and trading to each other with millions of addresses. The actual real user count for crypto would be a very tiny % of the active addresses. And the real value not even close to the claimed market caps.
How can you verify that? Other than, you know, "something that a anti-crypto bro on the Internet told you?"
I'm being slightly salty here but i dont get the backlash on crypto. It has a huge potential for safeguarding privacy (Monero) and avoiding corporate walled gardens and banks.
Re: Run LLMs at home, BitTorrent‑style
#106Are trained LLM's composable in any way? Like if you and I trust 99% of the same data, but each have 1% where we disagree, must we have two entirely separate models, or can we pool compute in the 99% case (along with the others who agree) and then create a derivative model for ourselves which covers for the differences in our trust models? I have only a rudimentary understanding of neural nets but it doesn't seem cra…
Somewhat yes. See "LoRA": https://arxiv.org/abs/2106.09685
They're not composable in the sense that you can take these adaptation layers and arbitrarily combine them, but training different models while sharing a common base of weights is a solved problem.
Re: Run LLMs at home, BitTorrent‑style
#107Are trained LLM's composable in any way? Like if you and I trust 99% of the same data, but each have 1% where we disagree, must we have two entirely separate models, or can we pool compute in the 99% case (along with the others who agree) and then create a derivative model for ourselves which covers for the differences in our trust models? I have only a rudimentary understanding of neural nets but it doesn't seem cra…
This is called ensembling. https://blog.allenai.org/llm-blender-a-simple-ensemble-learn...
Re: Run LLMs at home, BitTorrent‑style
#108Earlier quoted context omitted.
This way too nobody can copyright-cancel the LLM like OpenAI or whatever
Exactly, litigation has never been applied to content delivered over BitTorrent-style networks
Litigation is one thing, totally erasing it from the public internet if it were hosted centrally is something else.
Re: Run LLMs at home, BitTorrent‑style
#109Earlier quoted context omitted.
> It's what their proponent claim that they are useful for, yet there's no single instance of a successful blockchain project actually achieving this kind of resource-sharing goal Sia and Filecoin already work in this way to for people to share storage. > In fact, it's likely that trying to add money into the mix will just break the system instead of improving it This depends on the amount of money people are willing…
> Sia and Filecoin already work in this way to for people to share storage. You'll notice that I said “successful” in my original sentence. > This depends on the amount of money people are willing to pay for processing power. Volunteer contributions would be reduced, but the paid contributions could make up for it if the people who want to train their model pay enough to attract more people into the system and if tho…
> the distributed nature of things is always going to make it more expensive than a traditional solution
> especially if you need byzantine fault tolerance
For storage this can be done much more efficiently with erasure coding and hashing.
For compute, reputation. A node with no reputation has all of its output verified (and so gets paid less). A node with a good reputation history only gets random spot checks, but fail a spot check and you're back to getting paid less, maybe even retroactively.
> For instance filecoin only has a low thousands nodes, the average filecoin node has something like 10PB of available storage, the top three having 90PB each and making barely $1600 a day, which is $6.4 a year per TB.
So it costs too much and it's too cheap?
The nature of something like this is low barrier to entry, so the high competitiveness is going to result in low prices. That's kind of the idea.
The result is going to be two main categories of supplier. One, huge nodes with economies of scale. These might take lower prices than some retail cloud offering, but they also don't have customer acquisition or support costs. Two, nodes with "free" storage, e.g. you built a media center which is already on 24/7 but still has a few TB of free space, so until you get around to using it yourself you'll take the however much in free money. In both cases because they have lower costs than competing providers.
It sounds like the network is providing several exabytes of storage for an extremely competitive price. How is that not a success?
Re: Run LLMs at home, BitTorrent‑style
#110Cool service. It's worth noting that, with quantization/QLORA, models as big as llama2-70b can be run on consumer hardware (2xRTX 3090) at acceptable speeds (~20t/s) using frameworks like llama.cpp. Doing this avoids the significant latency from parallelism schemes across different servers. p.s. from experience instruct-finetuning falcon180b, it's not worth using over llama2-70b as it's significantly undertrained.
AFAIK you cannot train 70B on 2x 3090, even with GPTQ/qlora. And the inference is pretty inefficient. Pooling the hardware would achieve much better GPU utilization and (theoretically) faster responses for the host's requests
For training you would need more memory. As for the pooling, Theoretically yes but wouldn't latency play as much, if not a greater part in the response time here? Imagine a tensor-parallel gather where the other nodes are in different parts of the country.
Here I'm assuming that Petal uses a large number of small, heterogenous nodes like consumer gpus. It might as well be something much simpler.