Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run LLMs at home, BitTorrent‑style

petals.dev

Post reply on HN
Print view View original

81–90 of 135 posts

Re: Run LLMs at home, BitTorrent‑style

#81

Post by LoganDark »

brucethemoose2 wrote: post #41
AaronFriel wrote: post #20

Earlier quoted context omitted.

That's true for conventional fine-tuning, but is it the case for parameter efficient fine tuning and qLORA? My understanding is that for a N billion parameter model, fine tuning can occur with a slightly-less-than-N gigabyte of VRAM GPU. For that 70B parameter model: an A100?

2x 40/48GB GPUs would be the cheapest. But that's still a very expensive system, especially if you don't have a beefy workstation with 2x PCIe slots just lying around.

Even mATX boards tend to come with two (full-length) PCIe slots, and that's easy sub-$1k territory. Not exactly a beefy workstation.

Source: have a $200 board in my computer right now with two full-length PCIe slots.

Re: Run LLMs at home, BitTorrent‑style

#82

Post by dTal »

akomtu wrote: post #24
brucethemoose2 wrote: post #18

> and fine‑tune them for your tasks This is the part that raised my eyebrows. Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"

What prevents parallel LLM training? If you read book 1 first and then book 2, the resulting update in your knowledge will be the same if you read the books in the reverse order. It seems reasonable to assume that LLM is trained on each book independently, the two deltas in the LLM weights can be just added up.

The "deltas" are calculated by the error in how well the current state of the network predicts the output, backpropagated. Sequential runs are not commutative because the state changes.

Consider the trivial example of training a network to distinguish between sample A and sample B. Give it a hundred As in a row and it just learns "everything is A". Give it a hundred Bs in a row and it relearns "no, everything is B". To train it to distinguish, you must alternate As and Bs (and not too regularly, either!)

Re: Run LLMs at home, BitTorrent‑style

#83

Post by 7speter »

LoganDark wrote: post #81
brucethemoose2 wrote: post #41

Earlier quoted context omitted.

2x 40/48GB GPUs would be the cheapest. But that's still a very expensive system, especially if you don't have a beefy workstation with 2x PCIe slots just lying around.

Even mATX boards tend to come with two (full-length) PCIe slots, and that's easy sub-$1k territory. Not exactly a beefy workstation. Source: have a $200 board in my computer right now with two full-length PCIe slots.

Whats more difficult is trying to cool gpus with 24-48gb of RAM… they all seem to be passively cooled

Re: Run LLMs at home, BitTorrent‑style

#84

Post by LoganDark »

7speter wrote: post #83
LoganDark wrote: post #81

Earlier quoted context omitted.

Even mATX boards tend to come with two (full-length) PCIe slots, and that's easy sub-$1k territory. Not exactly a beefy workstation. Source: have a $200 board in my computer right now with two full-length PCIe slots.

Whats more difficult is trying to cool gpus with 24-48gb of RAM… they all seem to be passively cooled

Good point, I think most of them are designed for a high-airflow server chassis, with airflow in a direction that a desktop case wouldn't necessarily facilitate (parallel to the card).

Re: Run LLMs at home, BitTorrent‑style

#85

Post by AnthonyMouse »

seydor wrote: post #6
esafak wrote: post #2

The first question I had was "what are the economics?" From the FAQ: Will Petals incentives be based on crypto, blockchain, etc.? No, we are working on a centralized incentive system similar to the AI Horde kudos, even though Petals is a fully decentralized system in all other aspects. We do not plan to provide a service to exchange these points for money, so you should see these incentives as "game" points designed…

It's a shame that every decentralized projects needs to be compared to cryptocoins now

It's not the comparison, it's that it's one of the things cryptocoins are actually useful for: You have people all over the world with GPUs, some of them want to pay the others for use of them, but their countries use different payment networks or the developers want to be able to automate it without forcing the users to all sign up with the same mercurial payment processor who could screw over any of the users at random.

Re: Run LLMs at home, BitTorrent‑style

#86

Post by AnthonyMouse »

vorpalhex wrote: post #10
kordlessagain wrote: post #7

Earlier quoted context omitted.

The logical conclusion is that they (the models) will eventually be linked to crypto payments though. This is where Lightning becomes important... Edit: To clarify, I'm not suggesting linking these Petal "tokens" to any payment system. I'm talking about, in general, calls to clusters of machine learning models, decentralized or not, will likely use crypto payments because it gives you auth and a means of payment. I d…

I mean, I can sell you Eve or Runescape currency but we don't need any crypto to execute on it. "Gold sellers" existed well before crypto.

Is there an API for that which doesn't require each of the users to create a separate account on something else?

Re: Run LLMs at home, BitTorrent‑style

#87

Post by segfaultbuserr »

LoganDark wrote: post #84
7speter wrote: post #83

Earlier quoted context omitted.

Whats more difficult is trying to cool gpus with 24-48gb of RAM… they all seem to be passively cooled

Good point, I think most of them are designed for a high-airflow server chassis, with airflow in a direction that a desktop case wouldn't necessarily facilitate (parallel to the card).

Waterblocks exist for some compute-only GPUs, including the Nvidia A100. Also, there are a few small vendors in China that offer mounting kits that allow you to mod these compute-only GPUs to use off-the-shelf AIO watercoolers. Certainly, not many people are going to take the risk to modify the expensive Nvidia A100, but these solutions are moderately popular among the DIY home lab developers to convert older server cards for home workstation use. Decommissioned Nvidia Tesla P100 or V100 can be purchased cheaply for several hundreds dollars.

Re: Run LLMs at home, BitTorrent‑style

#88

Post by LoganDark »

segfaultbuserr wrote: post #87
LoganDark wrote: post #84

Earlier quoted context omitted.

Good point, I think most of them are designed for a high-airflow server chassis, with airflow in a direction that a desktop case wouldn't necessarily facilitate (parallel to the card).

Waterblocks exist for some compute-only GPUs, including the Nvidia A100. Also, there are a few small vendors in China that offer mounting kits that allow you to mod these compute-only GPUs to use off-the-shelf AIO watercoolers. Certainly, not many people are going to take the risk to modify the expensive Nvidia A100, but these solutions are moderately popular among the DIY home lab developers to convert older server…

> Decommissioned Nvidia Tesla P100 or V100 can be purchased cheaply for several hundreds dollars.

Meh. If you want 16GB of VRAM for several hundred dollars, can't you just pull a brand new 30-series off the shelf and have ten times more computing power than those old pascal cards? You'll even have more VRAM if you go for the 3080 or 3090. Admittedly, the 3090 is closer to $700 or so, but it should still make a P100 very sad in comparison.

Re: Run LLMs at home, BitTorrent‑style

#89

Post by littlestymaar »

AnthonyMouse wrote: post #85
seydor wrote: post #6

Earlier quoted context omitted.

It's a shame that every decentralized projects needs to be compared to cryptocoins now

It's not the comparison, it's that it's one of the things cryptocoins are actually useful for: You have people all over the world with GPUs, some of them want to pay the others for use of them, but their countries use different payment networks or the developers want to be able to automate it without forcing the users to all sign up with the same mercurial payment processor who could screw over any of the users at ra…

> it's that it's one of the things cryptocoins are actually useful for

It's what their proponent claim that they are useful for, yet there's no single instance of a successful blockchain project actually achieving this kind of resource-sharing goal.

> You have people all over the world with GPUs, some of them want to pay the others for use of them

The gigantic success of bitTorrent shows that humans as a group don't need to have monetary incentives to share their spare hardware. In fact, it's likely that trying to add money into the mix will just break the system instead of improving it: https://en.wikipedia.org/wiki/Overjustification_effect

Post reply on HN