Earlier quoted context omitted.
This way too nobody can copyright-cancel the LLM like OpenAI or whatever
Exactly, litigation has never been applied to content delivered over BitTorrent-style networks
Aha, touché salesman :)
Run LLMs at home, BitTorrent‑style
111–120 of 135 posts
Re: Run LLMs at home, BitTorrent‑style
#112Earlier quoted context omitted.
AFAIK you cannot train 70B on 2x 3090, even with GPTQ/qlora. And the inference is pretty inefficient. Pooling the hardware would achieve much better GPU utilization and (theoretically) faster responses for the host's requests
For training you would need more memory. As for the pooling, Theoretically yes but wouldn't latency play as much, if not a greater part in the response time here? Imagine a tensor-parallel gather where the other nodes are in different parts of the country. Here I'm assuming that Petal uses a large number of small, heterogenous nodes like consumer gpus. It might as well be something much simpler.
> Theoretically yes but wouldn't latency play as much, if not a greater part in the response time here?
For inference? Yeah, but its still better than nothing if your hardware can't run the full model, or run it extremely slowly.
I think frameworks like MLC-LLM and llama.cpp kinda throw a wrench in this though, as you can get very acceptable throughput on an IGP or split across a CPU/dGPU, without that huge networking penalty. And pooling complete hosts (like AI Horde) is much cheaper.
I'm not sure what the training requirements are, but ultimately throughput is all that matters for training, especially if you can "buy" training time with otherwise idle GPU time.
Re: Run LLMs at home, BitTorrent‑style
#113Earlier quoted context omitted.
> GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp. GGUF offers numerous advantages over GGML, such as better tokenisation, and support for special tokens. It is also supports metadata, and is designed to be extensible is there a more canonical blogpost or link to learn more about the technical decisions here?
https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md#... It is (IMO) a necessary and good change. I just specified gguf because my 3090 cannot host a 70B model without offloading outside of exLlama's very new ~2 bit quantization. And pre quantized gguf is a much smaller download than raw fp16 for conversion.
thanks very much!
Re: Run LLMs at home, BitTorrent‑style
#114Earlier quoted context omitted.
2x 40/48GB GPUs would be the cheapest. But that's still a very expensive system, especially if you don't have a beefy workstation with 2x PCIe slots just lying around.
Even mATX boards tend to come with two (full-length) PCIe slots, and that's easy sub-$1k territory. Not exactly a beefy workstation. Source: have a $200 board in my computer right now with two full-length PCIe slots.
Not with full x16/x16, though I suppose you don't necessarily need that.
Re: Run LLMs at home, BitTorrent‑style
#115Earlier quoted context omitted.
Even mATX boards tend to come with two (full-length) PCIe slots, and that's easy sub-$1k territory. Not exactly a beefy workstation. Source: have a $200 board in my computer right now with two full-length PCIe slots.
Whats more difficult is trying to cool gpus with 24-48gb of RAM… they all seem to be passively cooled
The Quadros/Firepros have blower coolers.
Re: Run LLMs at home, BitTorrent‑style
#116Earlier quoted context omitted.
> Decommissioned Nvidia Tesla P100 or V100 can be purchased cheaply for several hundreds dollars. Meh. If you want 16GB of VRAM for several hundred dollars, can't you just pull a brand new 30-series off the shelf and have ten times more computing power than those old pascal cards? You'll even have more VRAM if you go for the 3080 or 3090. Admittedly, the 3090 is closer to $700 or so, but it should still make a P100 v…
Yeah, these GPUs became less appealing after the prices of 30-series GPUs have dropped. The price of SXM cards are still somewhat unbeatable though if you have a compatible server motherboard [1]. Nvidia P100s are being sold for as low as $100 each, there are similar savings for the Nvidia V100s. But yeah, a saving around $100 to $200 is not really worthwhile... Another curious contender is the decommissioned Nvidia…
I didn't know the CMP had full bandwidth. that would be a an excellent card for smallish networks (like stable diffusion, GANs, audio networks)
...But it doesn't seem to be cheap. Not really worth it over a 4090 for the same price.
Re: Run LLMs at home, BitTorrent‑style
#117Earlier quoted context omitted.
Yeah, these GPUs became less appealing after the prices of 30-series GPUs have dropped. The price of SXM cards are still somewhat unbeatable though if you have a compatible server motherboard [1]. Nvidia P100s are being sold for as low as $100 each, there are similar savings for the Nvidia V100s. But yeah, a saving around $100 to $200 is not really worthwhile... Another curious contender is the decommissioned Nvidia…
I didn't know the CMP had full bandwidth. that would be a an excellent card for smallish networks (like stable diffusion, GANs, audio networks) ...But it doesn't seem to be cheap. Not really worth it over a 4090 for the same price.
It seems that the CMP 170HX is being sold for $500 +/- $100 on the flea markets in China as closed mining farms are dumping any remaining inventory. Not sure if the prices are real, I'm currently trying to purchase some.
Re: Run LLMs at home, BitTorrent‑style
#118Earlier quoted context omitted.
Even mATX boards tend to come with two (full-length) PCIe slots, and that's easy sub-$1k territory. Not exactly a beefy workstation. Source: have a $200 board in my computer right now with two full-length PCIe slots.
Not with full x16/x16, though I suppose you don't necessarily need that.
Of course, usually the other PCIe slots are something stupid, but there's still a second full-length one, so this could potentially fit two GPUs with the right power supply.
Re: Run LLMs at home, BitTorrent‑style
#119Earlier quoted context omitted.
> Sia and Filecoin already work in this way to for people to share storage. You'll notice that I said “successful” in my original sentence. > This depends on the amount of money people are willing to pay for processing power. Volunteer contributions would be reduced, but the paid contributions could make up for it if the people who want to train their model pay enough to attract more people into the system and if tho…
> the distributed nature of things is always going to make it more expensive than a traditional solution > especially if you need byzantine fault tolerance For storage this can be done much more efficiently with erasure coding and hashing. For compute, reputation. A node with no reputation has all of its output verified (and so gets paid less). A node with a good reputation history only gets random spot checks, but f…
> For storage this can be done much more efficiently with erasure coding and hashing.
More efficient that what exactly? It's still far less efficient than not having to hash and run erasure coding…
> For compute, reputation. A node with no reputation has all of its output verified (and so gets paid less). A node with a good reputation history only gets random spot checks, but fail a spot check and you're back to getting paid less, maybe even retroactively.
That only works if the attacker cannot make big gains from a single cheat after a period of building reputation. There's a reason why this isn't being used in the wild by blockchains…
> So it costs too much and it's too cheap?
Yes, it costs too much to operate, and it's too cheap as a product so operators are losing money. The only reason why there's an offering at all is that some people invested lots of money on hardware in 2021 when the token price was 50 times higher (but then the storage cost was prohibitive).
> It sounds like the network is providing several exabytes of storage for an extremely competitive price. How is that not a success?
Barely anyone using it despite a price so low that it doesn't even allow operators to break even, how is that supposed to be a success?
Re: Run LLMs at home, BitTorrent‑style
#120Earlier quoted context omitted.
[flagged]
Didn't etherum cut power consumption by 99.95% by switching to Proof of Stake? So what are you securing exactly with all those hashes? Kinda crazy how people stick to Bitcoin but preach decentralisation. You can't be half way noble.
Yeah, and by doing so they got rid of 99.99% of their security and censorship resistance. PoS is Fiat 2.0. It's not worth mentioning in the same breath as Bitcoin, not that it ever was.