Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run LLMs at home, BitTorrent‑style

petals.dev

Post reply on HN
Print view View original

91–100 of 135 posts

Re: Run LLMs at home, BitTorrent‑style

#91

Post by segfaultbuserr »

LoganDark wrote: post #88
segfaultbuserr wrote: post #87

Earlier quoted context omitted.

Waterblocks exist for some compute-only GPUs, including the Nvidia A100. Also, there are a few small vendors in China that offer mounting kits that allow you to mod these compute-only GPUs to use off-the-shelf AIO watercoolers. Certainly, not many people are going to take the risk to modify the expensive Nvidia A100, but these solutions are moderately popular among the DIY home lab developers to convert older server…

> Decommissioned Nvidia Tesla P100 or V100 can be purchased cheaply for several hundreds dollars. Meh. If you want 16GB of VRAM for several hundred dollars, can't you just pull a brand new 30-series off the shelf and have ten times more computing power than those old pascal cards? You'll even have more VRAM if you go for the 3080 or 3090. Admittedly, the 3090 is closer to $700 or so, but it should still make a P100 v…

Yeah, these GPUs became less appealing after the prices of 30-series GPUs have dropped. The price of SXM cards are still somewhat unbeatable though if you have a compatible server motherboard [1]. Nvidia P100s are being sold for as low as $100 each, there are similar savings for the Nvidia V100s. But yeah, a saving around $100 to $200 is not really worthwhile...

Another curious contender is the decommissioned Nvidia CMP series GPUs from miners. For example, the Nvidia CMP 170HX basically uses the same Nvidia A100 PCB with its features downsized or disabled (8 GB VRAM, halved shaders, etc). But interestingly, it seems to preserve the full 1500 GB/s memory bandwidth, making it potentially an interesting card for running memory-bound simulations.

[1] Prices are so low exactly because most people don't. SXM-to-PCIe adapters also exist which cost $100-$200 - nearly as much as you have saved. It should be trivial to reverse-engineer the pinout to make a free and open source version.

Re: Run LLMs at home, BitTorrent‑style

#92

Post by LoganDark »

segfaultbuserr wrote: post #91
LoganDark wrote: post #88

Earlier quoted context omitted.

> Decommissioned Nvidia Tesla P100 or V100 can be purchased cheaply for several hundreds dollars. Meh. If you want 16GB of VRAM for several hundred dollars, can't you just pull a brand new 30-series off the shelf and have ten times more computing power than those old pascal cards? You'll even have more VRAM if you go for the 3080 or 3090. Admittedly, the 3090 is closer to $700 or so, but it should still make a P100 v…

Yeah, these GPUs became less appealing after the prices of 30-series GPUs have dropped. The price of SXM cards are still somewhat unbeatable though if you have a compatible server motherboard [1]. Nvidia P100s are being sold for as low as $100 each, there are similar savings for the Nvidia V100s. But yeah, a saving around $100 to $200 is not really worthwhile... Another curious contender is the decommissioned Nvidia…

Is it possible to take something like a CMP 170HX and do board-level work to add more memory chips? Or are they not connected to silicon?

Re: Run LLMs at home, BitTorrent‑style

#93

Post by segfaultbuserr »

LoganDark wrote: post #92
segfaultbuserr wrote: post #91

Earlier quoted context omitted.

Yeah, these GPUs became less appealing after the prices of 30-series GPUs have dropped. The price of SXM cards are still somewhat unbeatable though if you have a compatible server motherboard [1]. Nvidia P100s are being sold for as low as $100 each, there are similar savings for the Nvidia V100s. But yeah, a saving around $100 to $200 is not really worthwhile... Another curious contender is the decommissioned Nvidia…

Is it possible to take something like a CMP 170HX and do board-level work to add more memory chips? Or are they not connected to silicon?

I don't believe it's possible. The HBM2e chips are integrated onto the package of the GPU die, making them impossible to remove or modify in a non-destructive manner.

Re: Run LLMs at home, BitTorrent‑style

#94

Post by andirk »

Gigachad wrote: post #45
december456 wrote: post #40

Earlier quoted context omitted.

Thats not the best counterargument, because Bitcoin has privacy qualities by default. You can hop on to any block explorer and accept every address as another user, but you cant verify that (without expensive analysis, on a case-by-case basis) those are not owned by the same guy. Same with Tor, while some data like bridge usage is being collected somehow (i havent looked into it) you cant reliably prove that thousand…

It's pretty obvious that the majority of transaction volume and value is rubbish. Bots buying, selling, and trading to each other with millions of addresses. The actual real user count for crypto would be a very tiny % of the active addresses. And the real value not even close to the claimed market caps.

It costs real world dollars to transact so it's not nothing. This argument can be made for stonks as well, right?

Re: Run LLMs at home, BitTorrent‑style

#95

Post by mkii »

jmorgan wrote: post #17

This is neat. Model weights are split into their layers and distributed across several machines who then report themselves in a big hash table when they are ready to perform inference or fine tuning "as a team" over their subset of the layers. It's early but I've been working on hosting model weights in a Docker registry for https://github.com/jmorganca/ollama . Mainly for the content addressability (Ollama will veri…

Ah, is it possible to tone down the self-promotion? I've been seeing your comments for ollama on many LLM-related posts here.

> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.

Surely in this case it would've been possible to comment about OP's work while leaving out the free backlink to your project. Just my 0.02

Re: Run LLMs at home, BitTorrent‑style

#96

Post by joshuaissac »

littlestymaar wrote: post #89
AnthonyMouse wrote: post #85

Earlier quoted context omitted.

It's not the comparison, it's that it's one of the things cryptocoins are actually useful for: You have people all over the world with GPUs, some of them want to pay the others for use of them, but their countries use different payment networks or the developers want to be able to automate it without forcing the users to all sign up with the same mercurial payment processor who could screw over any of the users at ra…

> it's that it's one of the things cryptocoins are actually useful for It's what their proponent claim that they are useful for, yet there's no single instance of a successful blockchain project actually achieving this kind of resource-sharing goal. > You have people all over the world with GPUs, some of them want to pay the others for use of them The gigantic success of bitTorrent shows that humans as a group don't…

> It's what their proponent claim that they are useful for, yet there's no single instance of a successful blockchain project actually achieving this kind of resource-sharing goal

Sia and Filecoin already work in this way to for people to share storage.

> In fact, it's likely that trying to add money into the mix will just break the system instead of improving it

This depends on the amount of money people are willing to pay for processing power. Volunteer contributions would be reduced, but the paid contributions could make up for it if the people who want to train their model pay enough to attract more people into the system and if those people can compete with conventional commercial offerings.

Re: Run LLMs at home, BitTorrent‑style

#97

Post by YetAnotherNick »

brucethemoose2 wrote: post #18

> and fine‑tune them for your tasks This is the part that raised my eyebrows. Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"

Finetuning in a distributed way with questionable network would be lot more energy/cost inefficient than doing it with a single node or a well connected cluster. Also, you can finetune 70b model for million tokens for $2 in lambda cloud or <$10 in replicate.

Re: Run LLMs at home, BitTorrent‑style

#98

Post by Double_a_92 »

Am I the only one that really really hates pages like google Colab? I never know what is going on there. Is it free? Is it running on my machine, or is it running on googles Cloud? If the latter, again is it really free?!

Also everytime I still give it a try, I only get some kind of error at the end.

Edit: Here we go. Literally the first line that it wanted to execute: "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. tensorflow-metadata 1.14.0 requires protobuf=3.20.3, but you have protobuf 4.24.3 which is incompatible."

Re: Run LLMs at home, BitTorrent‑style

#99

Post by 0x008 »

timost wrote: post #90

You can host your own swarm of servers apparently [0]. I would be curious to have a ballpark estimate of the finetunning performance of a "private" petals cluster. [0] https://github.com/bigscience-workshop/petals/wiki/Launch-yo...

I think if you run a cluster in a trusted environment it should be more efficient to use ray or something similar

Re: Run LLMs at home, BitTorrent‑style

#100

Post by littlestymaar »

joshuaissac wrote: post #96
littlestymaar wrote: post #89

Earlier quoted context omitted.

> it's that it's one of the things cryptocoins are actually useful for It's what their proponent claim that they are useful for, yet there's no single instance of a successful blockchain project actually achieving this kind of resource-sharing goal. > You have people all over the world with GPUs, some of them want to pay the others for use of them The gigantic success of bitTorrent shows that humans as a group don't…

> It's what their proponent claim that they are useful for, yet there's no single instance of a successful blockchain project actually achieving this kind of resource-sharing goal Sia and Filecoin already work in this way to for people to share storage. > In fact, it's likely that trying to add money into the mix will just break the system instead of improving it This depends on the amount of money people are willing…

> Sia and Filecoin already work in this way to for people to share storage.

You'll notice that I said “successful” in my original sentence.

> This depends on the amount of money people are willing to pay for processing power. Volunteer contributions would be reduced, but the paid contributions could make up for it if the people who want to train their model pay enough to attract more people into the system and if those people can compete with conventional commercial offerings.

That's a very big “if”: the distributed nature of things is always going to make it more expensive than a traditional solution, especially if you need byzantine fault tolerance (which you need as soon as their monetary value to earn by cheating), the same way that a blockchain is orders of magnitude more expensive than a cloud KV store database, and by pushing the volunteers away you'll end up with a small pool of for-profit actors and these actors themselves likely would be better off if they provided their own cloud offering.

For instance filecoin only has a low thousands nodes, the average filecoin node has something like 10PB of available storage, the top three having 90PB each and making barely $1600 a day, which is $6.4 a year per TB.

Post reply on HN