Facebook LLAMA is being openly distributed via torrents • Hacker News Board

Re: Facebook LLAMA is being openly distributed via torrents

#171

Post by iszomer » Fri, Mar 03, 2023, 1:42 PM UTC

ithkuil wrote: post #152

nodja wrote: post #143

Earlier quoted context omitted.

15k is the price to buy a GPU, not to run it, you'd have to account for electricity costs which isn't so straight forward.

The point was that if you only need to train once, then it's cheaper to rent the GPUs than to buy them. If you need to train it multiple times, then the cost of buying the GPUs is amortized among runs. In any case the cost per run is going to be lower than 30m

I'm sure that's the case. The latest sku I'm responsible for QC testing now contains 4x A100's in a 2U chassis. And oh man the number of QSFP ports it utilizes..

Re: Facebook LLAMA is being openly distributed via torrents

#172

Post by basch » Fri, Mar 03, 2023, 1:42 PM UTC

sva_ wrote: post #85

krisoft wrote: post #73

Earlier quoted context omitted.

> It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent. I was trying to come to grasp with how much resource there is concentrated in one of these models. Somehow I come to the conclusion that it cost more than buying a jet airliner to train one of these models. And it is about the same order of money as commissi…

The crazy thing is that all these models are just one local minimum, out of a staggering (unknown?!) number of such points on the plane.

“Brute forcing a really inefficient approximation/estimator” is a good way to summarize it.

It’s like having an overfit equation to a sample of data points, instead of the simpler actual line they fall near.

They end up being black boxes, we have almost no idea how they work inside, and we have no idea how overtrained they are when something simpler could do the same thing.

Re: Facebook LLAMA is being openly distributed via torrents

#173

Post by ithkuil » Fri, Mar 03, 2023, 1:44 PM UTC

ot wrote: post #16

In case it's not clear what's happening here (and from the comments it doesn't seem like it is), someone (not Meta) leaked the models and had the brilliant idea of advertising the magnet link through a GitHub pull request. The part about saving bandwidth is a joke. Meta employees may have not noticed or are still figuring out how to react, so the PR is still up. (Disclaimer: I work at Meta, but have no relationship w…

> and have no internal information on this

But you just said it was not Meta. Is that based on internal information? ;-)

Re: Facebook LLAMA is being openly distributed via torrents

#174

Post by renonce » Fri, Mar 03, 2023, 1:44 PM UTC

spencerchubb wrote: post #96

pavlov wrote: post #41

It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent. This has not been the case for most commercial software for the past 20 years, during the cloud era. If you could steal a dump of random Facebook source code, it would be 99% useless because it’s so closely tied to the infrastructure. There’s almost nothing yo…

I fail to see how this is different from other software in that regard. If you have parameters but not the network architecture, then it's not very useful.

You do need to guess things like activation functions, number of attention heads, order of attention layers, etc. Often the parameter names reveal something about these.

Re: Facebook LLAMA is being openly distributed via torrents

#175

Post by vintermann » Fri, Mar 03, 2023, 1:44 PM UTC

eigenvalue wrote: post #60

I'm not surprised-- I recently suggested that someone might try to pull an Aaron Swartz with the LLAMA weights (i.e., release them in an uncontrolled way similar to how Aaron attempted to release the JSTOR database). It's quite misleading for FB to claim that they are being so open, but then hoard the weights and only release it to a few academics. If the paper is to be believed, this is a major development, allowing…

It was already the most open language model in its class, given that the code for training and inference was available and it only used public data for training.

For Google and OpenAIs offerings, have fun reimplementing it from descriptions in the paper (including small crucial details that they may have left out), training it for a month, and then wondering if the implementation or the training data is the reason your model isn't as good as theirs.

Re: Facebook LLAMA is being openly distributed via torrents

#176

Post by fluoridation » Fri, Mar 03, 2023, 1:46 PM UTC

rowanG077 wrote: post #120

asah wrote: post #111

Earlier quoted context omitted.

sorry, but that's not how code works. It's true that code quality could be terrible but in fact Google is famous/notorious for extreme code review at the line-by-line granularity, plus comments, design docs and more. The real issues are (again) in dependencies and complex tooling. You can have beautiful code and then in the middle of it, an ML inference call that assumes a crazy ML model and set of hardware to run it…

[flagged]

Arguably it's not low quality code, but low quality system. Code can be correct, clear, and documented, and still be fragile and sensitive to platform configuration changes. E.g. "how many switches do I have to change in the build system before the code no longer builds?", "how many network jacks can I move this server over before I lobotomize the system?"

Re: Facebook LLAMA is being openly distributed via torrents

#177

Post by vintermann » Fri, Mar 03, 2023, 1:46 PM UTC

DeathArrow wrote: post #63

px43 wrote: post #57

Earlier quoted context omitted.

Hell, I'd love to be able to buy a $30k server to run these models. I think to run BLOOM required something more along the lines of a $200k server.

No need to spend $30k, use Azure or AWS.

I can see some issues with uploading a leaked model to a cloud provider.

Re: Facebook LLAMA is being openly distributed via torrents

#178

Post by elcomet » Fri, Mar 03, 2023, 1:46 PM UTC

linearalgebra45 wrote: post #123

Hypothetically, what would the consequences be if I ran this on my university's computing cluster?

None

Re: Facebook LLAMA is being openly distributed via torrents

#179

Post by htrp » Fri, Mar 03, 2023, 1:46 PM UTC

linearalgebra45 wrote: post #123

Hypothetically, what would the consequences be if I ran this on my university's computing cluster?

Either you get a nice invitation to collaborate on research with one of your uni's professors..... or you get sent to academic/disciplinary review and probably suspended for the semester.

Re: Facebook LLAMA is being openly distributed via torrents

#180

Post by renonce » Fri, Mar 03, 2023, 1:47 PM UTC

ot wrote: post #16

In case it's not clear what's happening here (and from the comments it doesn't seem like it is), someone (not Meta) leaked the models and had the brilliant idea of advertising the magnet link through a GitHub pull request. The part about saving bandwidth is a joke. Meta employees may have not noticed or are still figuring out how to react, so the PR is still up. (Disclaimer: I work at Meta, but have no relationship w…

It's fairly easy to obtain the weights. I've had two of my friends downloading these weights and sharing them with me, so it's probably not surprising that the weights got leaked.