Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook LLAMA is being openly distributed via torrents

github.com

Post reply on HN
Print view View original

61–70 of 719 posts

Re: Facebook LLAMA is being openly distributed via torrents

#61

Post by londons_explore »

Yoric wrote: post #56
px43 wrote: post #53

Earlier quoted context omitted.

My understanding is that weights are normally stored as pickled python blobs, which means arbitrary code execution as they are unpickled.

If it's PyTorch, it can definitely contain and execute arbitrary code. One of the reasons I'm not a huge fan of PyTorch.

They could contain arbitrary code... But typically do not. That means that with the right viewer application it will be trivial to know for sure.

It isn't like a multi gigabyte game for example, where knowing if there is any malicious code could easily be a multi-month reverse engineering project to get to the answer of 'probably not, but we don't have time to check every byte with a fine tooth comb'

Re: Facebook LLAMA is being openly distributed via torrents

#63

Post by DeathArrow »

px43 wrote: post #57
kif wrote: post #37

I wonder what the memory requirements would be to run such a large model. I'd love to be able to run this model, alas my MacBook can barely run toy models.

Hell, I'd love to be able to buy a $30k server to run these models. I think to run BLOOM required something more along the lines of a $200k server.

No need to spend $30k, use Azure or AWS.

Re: Facebook LLAMA is being openly distributed via torrents

#64

Post by px43 »

fancyfredbot wrote: post #27

opt-175B weights are already openly available as I understand. Hugging-face also has openly available weights for a 176B parameter LLM called Bloom. Is LLAMA offering something over and above these?

Yeah, their recent papers show the smaller LLAMA models outperforming the major LLMs today, and they also have bigger models. This isn't just an alternative, it's a multi order of magnitude optimization.

https://aibusiness.com/meta/meta-s-llama-language-model-outp...

Re: Facebook LLAMA is being openly distributed via torrents

#65

Post by londons_explore »

px43 wrote: post #57
kif wrote: post #37

I wonder what the memory requirements would be to run such a large model. I'd love to be able to run this model, alas my MacBook can barely run toy models.

Hell, I'd love to be able to buy a $30k server to run these models. I think to run BLOOM required something more along the lines of a $200k server.

With code modifications, it should be possible to run this with a very modest machine as long as you're happy for performance to suck. Transformer models typically need to read all the weights per 'word' output, so if your model is 20GB and you have not enough ram or vram, but have an SSD that reads 1GB/sec, expect 3 words per minute output speed.

However, code changes are necessary to achieve that, although they won't be crazy complex.

Re: Facebook LLAMA is being openly distributed via torrents

#66

Post by rnosov »

q1w2 wrote: post #59
kaszanka wrote: post #25

Here is the magnet link for posterity: magnet:?xt=urn:btih:ZXXDAUWYLRUXXBHUYEMS6Q5CE5WA3LVA&dn=LLaMA

Great, now how do I run it? Do I need a GPU with over 65GB RAM?

Generally, you'll need multiply model size by two to get required amount of video RAM. There are 4 sizes, so you might get away with even smaller GPU for say 13B model.

Re: Facebook LLAMA is being openly distributed via torrents

#67

Post by asah »

londons_explore wrote: post #55
pavlov wrote: post #41

It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent. This has not been the case for most commercial software for the past 20 years, during the cloud era. If you could steal a dump of random Facebook source code, it would be 99% useless because it’s so closely tied to the infrastructure. There’s almost nothing yo…

If you were to steal a chunk of source code or a binary from meta/Google, you could probably get it running inside a few weeks effort. Sure, the binary probably depends on a lot of internal proprietary infrastructure, but also most of that infrastructure is easy to write a mock implementation of, as long as you are happy for it to be in-ram, not multi-homed and don't need it to scale to billions of users. Most of the…

-1: as an ex-googler, I can say it was hard enough for Google itself to get its code to run, given gonzo infrastructure assumptions, proprietary libraries/languages, etc.

Re: Facebook LLAMA is being openly distributed via torrents

#68

Post by sebzim4500 »

IanCal wrote: post #30
ot wrote: post #16

In case it's not clear what's happening here (and from the comments it doesn't seem like it is), someone (not Meta) leaked the models and had the brilliant idea of advertising the magnet link through a GitHub pull request. The part about saving bandwidth is a joke. Meta employees may have not noticed or are still figuring out how to react, so the PR is still up. (Disclaimer: I work at Meta, but have no relationship w…

It's not even clear someone has leaked the models. A random person has put a download link on a PR, it could be anything.

The folder structure definitely looks like model weights, I didn't download or run it though so for all I know it only generates the words to "Never Gonna Give You Up".

Re: Facebook LLAMA is being openly distributed via torrents

#69

Post by londons_explore »

pimterry wrote: post #62

I give it a week before we see tools for subtly watermarking your secret LLM's weights, so you can trace leaks like this later.

Watermarking the weights is trivial.

Watermarking the output is also possible, but more complex and with a statistical success rate Vs performance tradeoff.

Re: Facebook LLAMA is being openly distributed via torrents

#70

Post by hiddencost »

londons_explore wrote: post #55
pavlov wrote: post #41

It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent. This has not been the case for most commercial software for the past 20 years, during the cloud era. If you could steal a dump of random Facebook source code, it would be 99% useless because it’s so closely tied to the infrastructure. There’s almost nothing yo…

If you were to steal a chunk of source code or a binary from meta/Google, you could probably get it running inside a few weeks effort. Sure, the binary probably depends on a lot of internal proprietary infrastructure, but also most of that infrastructure is easy to write a mock implementation of, as long as you are happy for it to be in-ram, not multi-homed and don't need it to scale to billions of users. Most of the…

This is just wrong.
Post reply on HN