Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run LLMs at home, BitTorrent‑style

petals.dev

Post reply on HN
Print view View original

71–80 of 135 posts

Re: Run LLMs at home, BitTorrent‑style

#71

Post by dleeftink »

borzunov wrote: post #67
malwrar wrote: post #63

How does this defend against a malicious participant altering the output of their share of the larger computation? Even without some kind of method for e.g. producing attacker-determined network output, this system seems vulnerable to lots of nodes joining and simply returning junk results, effectively DoSing the system.

Hi, a Petals dev here. We're developing validators that periodically go over all servers and ban the ones that return incorrect results. Additionally, clients can run data through multiple disjoint routes in the network and check that the results match. This catches frequent attackers but doesn't provide 100% protection - so we expect people to set up a _private_ swarm if they want full correctness guarantees. For ex…

How about tried and tested reputation systems for GPUs/providers to join certain swarms?

Yes, this can also be gamed (and I do not wish to bring yet another scoring system into this world), but it might just work for users wanting to choose between various levels of LLM security.

You might be able to even tie this into 'energy per compute unit' spent, enticing users to opt for more energy efficient offerings. Potentially, an all-round metric (or multiple metrics) for the viability of a GPU provider.

Re: Run LLMs at home, BitTorrent‑style

#73

Post by quickthrower2 »

borzunov wrote: post #65
quickthrower2 wrote: post #60

I got a lurid NSFW comment, just asking for the time (using the Colab), so I assume some people are trolling the network? Human: what is the time? The time is 12:30 PM. Human: are you sure? Yes, I am sure. The time is 12:30 PM.^ ^ I'm a young {...}

Hi, a Petals dev here. means "end of sequence" for LLMs. If a model generates it, it forgets everything and continues with an unrelated random text (I'm sorry to hear that the model generated a disturbing text in this case). Still, I doubt that malicious actors are involved here. Apparently, the Colab code snippet is just too simplified and does not handle correctly. This is not the case with the full chatbot app at…

Thanks for the reply. One way to guard against that would be if the LLM architecture refused to serve against just as a token?

Re: Run LLMs at home, BitTorrent‑style

#74

Post by judge2020 »

pavelstoev wrote: post #47
judge2020 wrote: post #37

Earlier quoted context omitted.

It can cost a lot to run a GPU, especially at full load. The 4090 stock pulls 500 watts of power under full load[0], which is 12 kWh/day or just under 4380 kWh a year, or over $450 in a year assuming $0.10-$0.11/kWh for average residential rates. The only variable is whether or not training requires the same power draw as hitting it with furmark. 0: https://youtu.be/j9vC9NBL8zo?t=983

Imagine someone paid you 25c/hour for 4090 compute sharing.

That's pretty much what Nicehash does, but after you pay for that electricity it isn't super profitable - especially if you use it for 1/3 or more of the day for your own purposes (gaming/etc).

Re: Run LLMs at home, BitTorrent‑style

#75

Post by brucethemoose2 »

nextaccountic wrote: post #26
Zetobal wrote: post #19

Earlier quoted context omitted.

An H100 is maybe a car but not nearly close to a house...

Is a single H100 enough?

80GB is enough, yeah.

I'm not sure what exact LORA/quantization settings would be ideal, but check out https://github.com/OpenAccess-AI-Collective/axolotl#config

Re: Run LLMs at home, BitTorrent‑style

#76

Post by brucethemoose2 »

quickthrower2 wrote: post #60

I got a lurid NSFW comment, just asking for the time (using the Colab), so I assume some people are trolling the network? Human: what is the time? The time is 12:30 PM. Human: are you sure? Yes, I am sure. The time is 12:30 PM.^ ^ I'm a young {...}

Base llama has lots of lurid in it already.

Re: Run LLMs at home, BitTorrent‑style

#78

Post by brucethemoose2 »

sumo43 wrote: post #68

Cool service. It's worth noting that, with quantization/QLORA, models as big as llama2-70b can be run on consumer hardware (2xRTX 3090) at acceptable speeds (~20t/s) using frameworks like llama.cpp. Doing this avoids the significant latency from parallelism schemes across different servers. p.s. from experience instruct-finetuning falcon180b, it's not worth using over llama2-70b as it's significantly undertrained.

AFAIK you cannot train 70B on 2x 3090, even with GPTQ/qlora.

And the inference is pretty inefficient. Pooling the hardware would achieve much better GPU utilization and (theoretically) faster responses for the host's requests

Re: Run LLMs at home, BitTorrent‑style

#80

Post by latchkey »

behnamoh wrote: post #34

looking at the list of contributors, way more people need to donate their GPU time for the betterment of all. maybe we finally have a good use for decentralized computing that doesn't calculate meaningless hashes for crypto, but helps the humanity by keeping these open source LLMs alive.

For the most part, gpus are no longer used for hashing. Once ETH switched to PoS, it decimated the entire GPU mining market.
Post reply on HN