Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook LLAMA is being openly distributed via torrents

github.com

Post reply on HN
Print view View original

111–120 of 719 posts

Re: Facebook LLAMA is being openly distributed via torrents

#111

Post by asah »

rowanG077 wrote: post #76
asah wrote: post #67

Earlier quoted context omitted.

-1: as an ex-googler, I can say it was hard enough for Google itself to get its code to run, given gonzo infrastructure assumptions, proprietary libraries/languages, etc.

That speaks volumes of the code quality @ Google.

sorry, but that's not how code works. It's true that code quality could be terrible but in fact Google is famous/notorious for extreme code review at the line-by-line granularity, plus comments, design docs and more.

The real issues are (again) in dependencies and complex tooling. You can have beautiful code and then in the middle of it, an ML inference call that assumes a crazy ML model and set of hardware to run it on.

Re: Facebook LLAMA is being openly distributed via torrents

#112

Post by version_five »

greggsy wrote: post #110
version_five wrote: post #106

Earlier quoted context omitted.

Finding a sha256 hash with N leading zeros is basically arbitrarily computationally expensive but could be written on a piece of paper. I don't see training an ML model as an egregious example of concentrating compute power

It’s not the power they’re referring to, it’s the density of information and effort that went into its creation.

  But these ML models are like neutron stars of computation density
I guess I interpreted that differently

Re: Facebook LLAMA is being openly distributed via torrents

#113

Post by moffkalast »

londons_explore wrote: post #65
px43 wrote: post #57

Earlier quoted context omitted.

Hell, I'd love to be able to buy a $30k server to run these models. I think to run BLOOM required something more along the lines of a $200k server.

With code modifications, it should be possible to run this with a very modest machine as long as you're happy for performance to suck. Transformer models typically need to read all the weights per 'word' output, so if your model is 20GB and you have not enough ram or vram, but have an SSD that reads 1GB/sec, expect 3 words per minute output speed. However, code changes are necessary to achieve that, although they won…

The most time/cost optimal solution is probably to buy 32 or 64 gigs of ram. That'll still be slow but most people are already half way there.

Re: Facebook LLAMA is being openly distributed via torrents

#114

Post by VadimPR »

px43 wrote: post #57
kif wrote: post #37

I wonder what the memory requirements would be to run such a large model. I'd love to be able to run this model, alas my MacBook can barely run toy models.

Hell, I'd love to be able to buy a $30k server to run these models. I think to run BLOOM required something more along the lines of a $200k server.

True, and that's why there is a project that is using volunteered, distributed GPUs to run BLOOM/BLOOMZ: https://github.com/bigscience-workshop/petals, http://chat.petals.ml.

Re: Facebook LLAMA is being openly distributed via torrents

#115

Post by fancyfredbot »

nodja wrote: post #93
krisoft wrote: post #73

Earlier quoted context omitted.

> It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent. I was trying to come to grasp with how much resource there is concentrated in one of these models. Somehow I come to the conclusion that it cost more than buying a jet airliner to train one of these models. And it is about the same order of money as commissi…

For anyone curious, it took 2048 A100 GPUs to train LLaMa, each GPU costs roughly $15k, facebook probably gets some sort of discount. That's a $30Mil if you want to train at that scale. Also IIRC it took 23 days to train the biggest model. Someone else can do the power consumption cost calculations.

$30m training cost is too high. Amazon's p4d.24xlarge is $32.77 an hour for 8 A100 GPUs. 2048 A100 GPUs for 23 days costs $4.6m at that rate. You might even get a discount.

Re: Facebook LLAMA is being openly distributed via torrents

#116

Post by koheripbal »

seydor wrote: post #45
pavlov wrote: post #41

It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent. This has not been the case for most commercial software for the past 20 years, during the cloud era. If you could steal a dump of random Facebook source code, it would be 99% useless because it’s so closely tied to the infrastructure. There’s almost nothing yo…

> massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent. So, like movies or software

Or a copy of Windows or Office source code.

Re: Facebook LLAMA is being openly distributed via torrents

#118

Post by esperent »

moffkalast wrote: post #113
londons_explore wrote: post #65

Earlier quoted context omitted.

With code modifications, it should be possible to run this with a very modest machine as long as you're happy for performance to suck. Transformer models typically need to read all the weights per 'word' output, so if your model is 20GB and you have not enough ram or vram, but have an SSD that reads 1GB/sec, expect 3 words per minute output speed. However, code changes are necessary to achieve that, although they won…

The most time/cost optimal solution is probably to buy 32 or 64 gigs of ram. That'll still be slow but most people are already half way there.

Doesn't it need to be GPU ram?

Re: Facebook LLAMA is being openly distributed via torrents

#119

Post by optimalsolver »

Tiberium wrote: post #100
pimterry wrote: post #62

I give it a week before we see tools for subtly watermarking your secret LLM's weights, so you can trace leaks like this later.

The original 4chan thread seems to indicate that the leaker verified that his hashes matched with another person who had access to the weights, to make sure that the weights aren't watermarked [0] 0: https://boards.4channel.org/g/thread/91848262#p91849855

The leaker accidentally doxxed themselves by adding the original download script to the torrent:

https://boards.4channel.org/g/thread/91848262#p91850503

Re: Facebook LLAMA is being openly distributed via torrents

#120

Post by rowanG077 »

asah wrote: post #111
rowanG077 wrote: post #76

Earlier quoted context omitted.

That speaks volumes of the code quality @ Google.

sorry, but that's not how code works. It's true that code quality could be terrible but in fact Google is famous/notorious for extreme code review at the line-by-line granularity, plus comments, design docs and more. The real issues are (again) in dependencies and complex tooling. You can have beautiful code and then in the middle of it, an ML inference call that assumes a crazy ML model and set of hardware to run it…

[flagged]
Post reply on HN