Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook LLAMA is being openly distributed via torrents

github.com

Post reply on HN
Print view View original

551–560 of 719 posts

Re: Facebook LLAMA is being openly distributed via torrents

#551

Post by Name_Chawps »

lukeplato wrote: post #548
Tiberium wrote: post #99

It seems that the leak originated from 4chan [1]. Two people in the same thread had access to the weights and verified that their hashes match [2][3] to make sure that the model isn't watermarked. However, the leaker made a mistake of adding the original download script which had his unique download URL to the torrent [4], so Meta can easily find them if they want to. [1]: https://boards.4channel.org/g/thread/9184826…

It would be interesting if there was a WikiLeaks-type of organization that facilitates safely leaking large models from big corporations. Not sure how that would play out for accelerationism and existential risk, but I certainly don't trust the current powers that be.

Open sourcing is widely recognized to be a bad thing when it comes to AI existential risk. (For the same reason you don't want simple instructions for how to build bio weapons posted to the internet.)

Modern AI is pretty harmless though, so it doesn't matter yet.

Re: Facebook LLAMA is being openly distributed via torrents

#552

Post by byteknight »

TaylorAlexander wrote: post #529
TaylorAlexander wrote: post #465

Earlier quoted context omitted.

The checkpoint for the 7B parameter model is 13.5GB, so maybe? Larger models are multiple chunks at 13.6GB each or 16.3GB each. I am hoping I will be able to run on my 16GB Vram but I don't know how much overhead is needed. Maybe people on reddit will do their tricks and squeeze the models in to smaller cards. EDIT: There seems to be a lot of overhead. Here someone struggles to fit the 7B parameter model (13.5GB chec…

Following up. After rebooting in to GUI that was enough to get it to fit, I guess xorg just accumulated some cruft in my last boot. So I can run it alongside gnome. nvidia-smi reports this model is using 15475MiB after changing the max batch size from 32 to 8 (see link in above post) As others have stated someone may have injected unknown code in to the pickled checkpoint, so I recommend running this in docker. I use…

Looks like you need multiple GPUs for anything >7B.

https://github.com/facebookresearch/llama/issues/55#issuecom...

Re: Facebook LLAMA is being openly distributed via torrents

#553

Post by gclawes »

binarymax wrote: post #539
abandonliberty wrote: post #395

Earlier quoted context omitted.

I'm curious if the blocking of adult content has to do with moralism, commercial interests, or something deeper. An eager to please conversational partner who can generate endless content seems quite dangerous and addictive, especially when it crosses over into romantic areas. There's already posts of people spending entire days interacting with LLMs, using as their therapist, romantic partner, etc. Combined with fin…

“An eager to please conversational partner who can generate endless content seems quite dangerous and addictive” Don’t date robots! https://youtu.be/wJ6knaienVE

He never saw the propaganda film!

Re: Facebook LLAMA is being openly distributed via torrents

#554

Post by yieldcrv »

riedel wrote: post #514
ddtaylor wrote: post #40

FWIW this information was already freely available via DHT scrapers like btdig [1] I think everyone at Facebook knows that torrents aren't secret and the Google form is basically a legal tool to shield them from liability while making litigation against anyone misusing the model easier. [1]: https://btdig.com/b8287ebfa04f879b048d4d4404108cf3e8014352/l...

The fun question is anyway if a ML model is copyright protectable. Probably not as it is produced by an algorithm (which even is GPL'ed). So the only tool would have been watermarking and pulling NDA type clauses, however a Google form seems not the best way in the first place also it is close to impossible to identify the leak (if they are not as stupid as it seems). Or am I missing anything? One backdoor would be i…

commercial derivative works have always been legal when you did not agree to other terms.

one person broke their agreement with Meta, they're the only person that has a problem and the only person who gets to find out if the agreement was applicable at all.

if you released a chat bot that could be prompted to regurgitate some copyrighted information, so what? it just proves that you didn't need the $30 million in funding yet to train your own because you are using an existing model. So either use the funding for that or don't sell shares or a product based on that pretext. Nobody else has a problem.

Anything I missed? Now I wouldn't reshare the model, but aside from use and commercial use of its output? Not everyone gets their way, that's not controversial.

Re: Facebook LLAMA is being openly distributed via torrents

#555

Post by [deleted] »

pavlov wrote: post #41

It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent. This has not been the case for most commercial software for the past 20 years, during the cloud era. If you could steal a dump of random Facebook source code, it would be 99% useless because it’s so closely tied to the infrastructure. There’s almost nothing yo…

[deleted]

Re: Facebook LLAMA is being openly distributed via torrents

#556

Post by thaumasiotes »

javajosh wrote: post #542
dawnbreez wrote: post #490

Earlier quoted context omitted.

This argument has been made since at least the start of written records: > And so it is that you by reason of your tender regard for the writing that is your offspring have declared the very opposite of its true effect. If men learn this, it will implant forgetfulness in their souls. They will cease to exercise memory because they rely on that which is written, calling things to remembrance no longer from within them…

> This argument has been made since at least the start of written records: That argument has been made since only slightly later. The key difference is that this truly is a unique time in history by population numbers. It's also unique in that humans could destroy the biosphere if we wanted to - that was never possible before the mid-20th. Just because people jumped the gun in the past doesn't mean they are wrong now…

> It's also unique in that humans could destroy the biosphere if we wanted to - that was never possible before the mid-20th.

It's not possible now either. If all of humanity's efforts were devoted to this task, they would not even make a noticeable difference.

Re: Facebook LLAMA is being openly distributed via torrents

#557

Post by q1w2 »

turmeric_root wrote: post #509
q1w2 wrote: post #314

Earlier quoted context omitted.

The 30B is 64.8GB and the A40s have 48GB NVRAM ea - so does this mean you got it working on one GPU with an NVLink to a 2nd, or is it really running on all 4 A40s? Is there a sub/forum/discord where folks talk about the nitty-gritty?

> so does this mean you got it working on one GPU with an NVLink to a 2nd, or is it really running on all 4 A40s? it's sharded across all 4 GPUs (as per the readme here: https://github.com/facebookresearch/llama ). I'd wait a few weeks to a month for people to settle on a solution for running the model, people are just going to be throwing pytorch code at the wall and seeing what sticks right now.

> people are just going to be throwing pytorch code at the wall

The pytorch 2.0 nightly has a number of performance enhancements as well as ways to reduce the memory footprint needed.

But also, looking at the README, it appears that model alone needs 2x the model size, eg 65B needs 130GB NVRAM, PLUS the decoding cache which stores 2 * 2 * n_layers * max_batch_size * max_seq_len * n_heads * head_dim bytes = 17GB for the 7B model (not sure if it needs to increase for the 65B model), but maybe a total of 147GB total NVRAM for the 65B model.

That should fit on 4 Nvidia A40s. Did you get memory errors, or you haven't tried yet?

Re: Facebook LLAMA is being openly distributed via torrents

#558

Post by atleastoptimal »

narrator wrote: post #493
abandonliberty wrote: post #395

Earlier quoted context omitted.

I'm curious if the blocking of adult content has to do with moralism, commercial interests, or something deeper. An eager to please conversational partner who can generate endless content seems quite dangerous and addictive, especially when it crosses over into romantic areas. There's already posts of people spending entire days interacting with LLMs, using as their therapist, romantic partner, etc. Combined with fin…

I think that as long as people can run their AI girlfriends on their own computers without having a corporation acting as an intermediary and thus a virtual "pimp" [for lack of a better word] in the relationship, I think it's fine. The problems come when people have to pay monthly to talk to their AI girlfriends and get charged extra if they want them to act a certain way or do certain things.

Corporations will do anything they can to keep that from happening. That's why every software product has gradually veered towards subscription models. They want you hooked to Microsoft/Apple/Facebook's AI girlfriend who will subtly insult your virtue as a partner if you don't buy extra credits. If you want to try out a politically incorrect fetish that's an extra 500 dollars per month for "extra premium"

Re: Facebook LLAMA is being openly distributed via torrents

#560

Post by sva_ »

basch wrote: post #172
sva_ wrote: post #85

Earlier quoted context omitted.

The crazy thing is that all these models are just one local minimum, out of a staggering (unknown?!) number of such points on the plane.

“Brute forcing a really inefficient approximation/estimator” is a good way to summarize it. It’s like having an overfit equation to a sample of data points, instead of the simpler actual line they fall near. They end up being black boxes, we have almost no idea how they work inside, and we have no idea how overtrained they are when something simpler could do the same thing.

I don't think the term "brute forcing" is an adequate term to describe gradient descent. Brute forcing would be to try all random weights with no system imo.
Post reply on HN