Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook LLAMA is being openly distributed via torrents

github.com

Post reply on HN
Print view View original

661–670 of 719 posts

Re: Facebook LLAMA is being openly distributed via torrents

#661

Post by Wingman4l7 »

astrange wrote: post #634
Wingman4l7 wrote: post #587

Earlier quoted context omitted.

I wouldn't give them too much credit. With the addition of the T2 "security" chip, your devices are bricks if they can't authenticate against your Apple ID. Combine that with them soldering together formerly modular components, and it's a very expensive lesson in you not owning your own hardware.

Is this supposed to be about activation lock? Phones don't contain a T2, that's a brand name for a component of Intel Macs. And of course it's good for phones to make them worthless to steal, or else they'd be hard to use in public.

I'm specifically talking about the T2 chips in the laptops. The newest Macbooks can be activation-locked, just like the phones. This has already bitten legitimate owners who are trying to restore backup images, etc.

Re: Facebook LLAMA is being openly distributed via torrents

#662

Post by Taek »

verall wrote: post #567
yunwal wrote: post #510

Earlier quoted context omitted.

True, but previous chips have been manufactured in China, and they’re also developing and manufacturing their successor to A100s (H100s) in China. https://www.cnbc.com/2022/09/01/nvidia-says-us-government-al...

That says "partially developed" in China. H100s will probably be manufactured in Taiwan just like other x100 chips. All consumer SKUs that I know of are manufactured in China. By volume this is certainly the majority of manufacturing.

I believe in terms of climate impact the chips (made in Taiwan) overwhelm everything else.

Re: Facebook LLAMA is being openly distributed via torrents

#663

Post by Taek »

yunwal wrote: post #392
CGamesPlay wrote: post #326

Earlier quoted context omitted.

No, I'm willing to bet the CO2 cost of the cards is also way higher than the electricity. Those things are built on the global supply chain, with materials potentially making multiple thousands of kms journeys between each step.

Long term I also imagine it's much cheaper to run these large model trainings on renewables. It's a very centralized process that doesn't necessarily need 100% availability. The manufacturing process, however, is totally decentralized, and NVIDIA mostly manufactures in China where coal is cheap.

Because the cards are so expensive, you really do want them running 24/7. The electricity is not a big deal for these really expensive chips

Re: Facebook LLAMA is being openly distributed via torrents

#664

Post by hnfong »

riedel wrote: post #514
ddtaylor wrote: post #40

FWIW this information was already freely available via DHT scrapers like btdig [1] I think everyone at Facebook knows that torrents aren't secret and the Google form is basically a legal tool to shield them from liability while making litigation against anyone misusing the model easier. [1]: https://btdig.com/b8287ebfa04f879b048d4d4404108cf3e8014352/l...

The fun question is anyway if a ML model is copyright protectable. Probably not as it is produced by an algorithm (which even is GPL'ed). So the only tool would have been watermarking and pulling NDA type clauses, however a Google form seems not the best way in the first place also it is close to impossible to identify the leak (if they are not as stupid as it seems). Or am I missing anything? One backdoor would be i…

photos are copyrightable by the person taking the photo only because they decided where and when to press a button. the rest are algorithms and hardware.

I believe the AI models would also be copyrightable as such, subject to arguments that the underlying data was protected and thus it was subject to prior copyrights instead

Re: Facebook LLAMA is being openly distributed via torrents

#665

Post by MacsHeadroom »

joker99 wrote: post #318
underlines wrote: post #280

- how much vRAM needed to run each model parameter size? - any inference optimization we can use similar to StableDiffusion, to bring down the vRAM requirements? I only know about these: - use 8bit precision - https://github.com/bigscience-workshop/petals - https://github.com/FMInference/FlexGen - https://github.com/microsoft/DeepSpeed Anything that could bring this to a 10GB 3080 or 24GB 3090 without 60s/it per toke…

If I may tack on a question as someone with zero clue of ML: when, if ever, will someone like me be able to run this on a Mac Studio with a M1 Ultra and 128GB of ram?

You can run 7B (equal to GPT-3 175B), 13B (better than GPT-3 175B), or 30B (better than anything else publicly available) but probably not 65B with that much RAM on an M1.

That would be using the CPU, as the M1 GPU is not yet supported.

Re: Facebook LLAMA is being openly distributed via torrents

#666

Post by MacsHeadroom »

terafo wrote: post #472
eightysixfour wrote: post #416

Earlier quoted context omitted.

I don't believe they could, need CUDA and more VRAM...

128 gigs is more than enough to load 13B model into. Pytorch has M1 support for some time now so CUDA isn't required.

13B fits on a single 3090 (24GB) in int8.

128 gigs might even be enough for 65B, if slowly.

Re: Facebook LLAMA is being openly distributed via torrents

#667

Post by MacsHeadroom »

mlboss wrote: post #256
aent wrote: post #32

For anyone wondering, it includes 4 models: 7/13/30/65 billion parameters, the smallest one is 14Gb, the largest one is 131GB, all four are 235Gb.

Is it possible to run the smallest one on a consumer gpu with 24gb ram ?

You can do even better!. You can run the second smallest one (better than GPT-3 175B) on 24GB of vram, ie LLaMA-13B. https://github.com/oobabooga/text-generation-webui/issues/14...

Re: Facebook LLAMA is being openly distributed via torrents

#668

Post by MacsHeadroom »

eightysixfour wrote: post #418
rihegher wrote: post #269

Earlier quoted context omitted.

I would be surprised if you can't. The smallest weight file is 14gb apparently

https://github.com/facebookresearch/llama/blob/main/FAQ.md#3 Looks like it needs 14gb for weights and it isn't clear what the minimum size for the decoding cache is, but it defaults to settings for 30gb GPUs.

In int8 7B needs only 9GB of VRAM and 13B needs only 20GB on a single GPU. https://github.com/oobabooga/text-generation-webui/issues/14...

Re: Facebook LLAMA is being openly distributed via torrents

#669

Post by turmeric_root »

q1w2 wrote: post #557
turmeric_root wrote: post #509

Earlier quoted context omitted.

> so does this mean you got it working on one GPU with an NVLink to a 2nd, or is it really running on all 4 A40s? it's sharded across all 4 GPUs (as per the readme here: https://github.com/facebookresearch/llama ). I'd wait a few weeks to a month for people to settle on a solution for running the model, people are just going to be throwing pytorch code at the wall and seeing what sticks right now.

> people are just going to be throwing pytorch code at the wall The pytorch 2.0 nightly has a number of performance enhancements as well as ways to reduce the memory footprint needed. But also, looking at the README, it appears that model alone needs 2x the model size, eg 65B needs 130GB NVRAM, PLUS the decoding cache which stores 2 * 2 * n_layers * max_batch_size * max_seq_len * n_heads * head_dim bytes = 17GB for t…

So since making that comment I managed to get 65B running on 1 x A100 80GB using 8-bit quantization. Though I did need ~130GB of regular RAM on top of it.

Re: Facebook LLAMA is being openly distributed via torrents

#670

Post by UltimateEdge »

turmeric_root wrote: post #669
q1w2 wrote: post #557

Earlier quoted context omitted.

> people are just going to be throwing pytorch code at the wall The pytorch 2.0 nightly has a number of performance enhancements as well as ways to reduce the memory footprint needed. But also, looking at the README, it appears that model alone needs 2x the model size, eg 65B needs 130GB NVRAM, PLUS the decoding cache which stores 2 * 2 * n_layers * max_batch_size * max_seq_len * n_heads * head_dim bytes = 17GB for t…

So since making that comment I managed to get 65B running on 1 x A100 80GB using 8-bit quantization. Though I did need ~130GB of regular RAM on top of it.

So is the model any good?
Post reply on HN