Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run LLMs at home, BitTorrent‑style

petals.dev

Post reply on HN
Print view View original

41–50 of 135 posts

Re: Run LLMs at home, BitTorrent‑style

#41

Post by brucethemoose2 »

AaronFriel wrote: post #20
brucethemoose2 wrote: post #18

> and fine‑tune them for your tasks This is the part that raised my eyebrows. Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"

That's true for conventional fine-tuning, but is it the case for parameter efficient fine tuning and qLORA? My understanding is that for a N billion parameter model, fine tuning can occur with a slightly-less-than-N gigabyte of VRAM GPU. For that 70B parameter model: an A100?

2x 40/48GB GPUs would be the cheapest. But that's still a very expensive system, especially if you don't have a beefy workstation with 2x PCIe slots just lying around.

Re: Run LLMs at home, BitTorrent‑style

#42

Post by [deleted] »

tossl568 wrote: post #38
behnamoh wrote: post #34

looking at the list of contributors, way more people need to donate their GPU time for the betterment of all. maybe we finally have a good use for decentralized computing that doesn't calculate meaningless hashes for crypto, but helps the humanity by keeping these open source LLMs alive.

[flagged]

[deleted]

Re: Run LLMs at home, BitTorrent‑style

#43

Post by brucethemoose2 »

malwrar wrote: post #30
brucethemoose2 wrote: post #18

> and fine‑tune them for your tasks This is the part that raised my eyebrows. Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"

Impossible? It’s just a bunch of math, you don’t need to keep the entire network in memory the whole time.

Well, any scheme where weights are dynamically loaded/unloaded from memory enough to fit on a 48GB GPU are so slow that training is basically impractical. Your 70B model would be obsolete by the time the finetuning is done.

Some inference frameworks came up with schemes for just this, and it was horrifically slow.

Re: Run LLMs at home, BitTorrent‑style

#44

Post by __rito__ »

I have used Petals at a past project. I share my GPU as well as wrote code for the project.

The Petals part was abstracted away from me. I had a normal experience writing code.

I don't have the project listed anywhere. Don't really know what happened to it. But, it was mainly some five or so guys spearheading the thing.

Re: Run LLMs at home, BitTorrent‑style

#45

Post by Gigachad »

december456 wrote: post #40
cheema33 wrote: post #39

Earlier quoted context omitted.

> Those "meaningless hashes" help secure hundreds of billions in savings of Bitcoin for hundreds of millions of people. Can you back that up with actual data? Other than something that a crypto bro on the Internet told you?

Thats not the best counterargument, because Bitcoin has privacy qualities by default. You can hop on to any block explorer and accept every address as another user, but you cant verify that (without expensive analysis, on a case-by-case basis) those are not owned by the same guy. Same with Tor, while some data like bridge usage is being collected somehow (i havent looked into it) you cant reliably prove that thousand…

It's pretty obvious that the majority of transaction volume and value is rubbish. Bots buying, selling, and trading to each other with millions of addresses. The actual real user count for crypto would be a very tiny % of the active addresses. And the real value not even close to the claimed market caps.

Re: Run LLMs at home, BitTorrent‑style

#46

Post by matheusmoreira »

tossl568 wrote: post #38
behnamoh wrote: post #34

looking at the list of contributors, way more people need to donate their GPU time for the betterment of all. maybe we finally have a good use for decentralized computing that doesn't calculate meaningless hashes for crypto, but helps the humanity by keeping these open source LLMs alive.

[flagged]

I wouldn't use bitcoin as an example. Monero is far more important.

Re: Run LLMs at home, BitTorrent‑style

#47

Post by pavelstoev »

judge2020 wrote: post #37
behnamoh wrote: post #34

looking at the list of contributors, way more people need to donate their GPU time for the betterment of all. maybe we finally have a good use for decentralized computing that doesn't calculate meaningless hashes for crypto, but helps the humanity by keeping these open source LLMs alive.

It can cost a lot to run a GPU, especially at full load. The 4090 stock pulls 500 watts of power under full load[0], which is 12 kWh/day or just under 4380 kWh a year, or over $450 in a year assuming $0.10-$0.11/kWh for average residential rates. The only variable is whether or not training requires the same power draw as hitting it with furmark. 0: https://youtu.be/j9vC9NBL8zo?t=983

Imagine someone paid you 25c/hour for 4090 compute sharing.

Re: Run LLMs at home, BitTorrent‑style

#48

Post by pavelstoev »

brucethemoose2 wrote: post #18

> and fine‑tune them for your tasks This is the part that raised my eyebrows. Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"

You can finetune 40B falcon on 4 x A10 with compiler optimization technology from CentML. No changes to the model.

Re: Run LLMs at home, BitTorrent‑style

#49

Post by whatyesaid »

tossl568 wrote: post #38
behnamoh wrote: post #34

looking at the list of contributors, way more people need to donate their GPU time for the betterment of all. maybe we finally have a good use for decentralized computing that doesn't calculate meaningless hashes for crypto, but helps the humanity by keeping these open source LLMs alive.

[flagged]

Didn't etherum cut power consumption by 99.95% by switching to Proof of Stake? So what are you securing exactly with all those hashes?

Kinda crazy how people stick to Bitcoin but preach decentralisation. You can't be half way noble.

Re: Run LLMs at home, BitTorrent‑style

#50

Post by senectus1 »

so how long until "tokens" are used to pay for GPU cycles.. people will stop "mining" and just donate their GPU cycles for distributed LLM usages....

in fact, if they did this so that it followed the sun so that the vast majority of it was powered by daylight Solar PV energy I wouldn't even be upset by that.

Post reply on HN