Live data from Hacker News

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

github.com

21–30 of 143 posts

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#21
post #16

If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…

Eagerly awaiting the int8 vs 4 benchmarks. Also, it can run on CPU https://github.com/markasoftware/llama-cpu So, an int8 patch could allow the 65B to run on a standard 128 GB setup assuming the 65B model's cache bursts fit, which if I were to speculate is why the released models stop @ 65B & meta likely already has larger unreleased internal ones.

early int4 experiments seem to indicate it's possible but you do lose performance, see this thread https://www.reddit.com/r/MachineLearning/comments/11i4olx/d_...

edit: to clarify, it may be possible to get this loss back and there is reason to be optimistic

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#22
post #18

It's been enough time since this leaked, so my question is why aren't there blog posts already of people blowing their $300 of starter credit with ${cloud_provider} on a few hours' experimentation running inference on this 65B model? Edit: I read the linked README. > I was impatient and curious to try to run 65B on an 8xA100 cluster Well?

The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWea…

Are you sure about that? I can't remember where I saw the table of memory requirements, but I'm sure some of the larger instances here [1] will surely be able to cope (assuming they're available!)

Oracle gives you a $300 free trial, which equates to running BM.GPU4.8 for over 10 hours - enough for a focused day of prompting

[1] https://www.oracle.com/cloud/compute/gpu/

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#23
post #18

Earlier quoted context omitted.

The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWea…

Are you sure about that? I can't remember where I saw the table of memory requirements, but I'm sure some of the larger instances here [1] will surely be able to cope (assuming they're available!) Oracle gives you a $300 free trial, which equates to running BM.GPU4.8 for over 10 hours - enough for a focused day of prompting [1] https://www.oracle.com/cloud/compute/gpu/

> Are you sure about that?

I'm not. The only way to know it is to try :) thank you for the link!

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#24
post #23

Earlier quoted context omitted.

Are you sure about that? I can't remember where I saw the table of memory requirements, but I'm sure some of the larger instances here [1] will surely be able to cope (assuming they're available!) Oracle gives you a $300 free trial, which equates to running BM.GPU4.8 for over 10 hours - enough for a focused day of prompting [1] https://www.oracle.com/cloud/compute/gpu/

> Are you sure about that? I'm not. The only way to know it is to try :) thank you for the link!

You only get a single month-long window to spend the credit! And I'm sure not going to spend any of my own money on prompting experiments.

I might be suffering from FOMO to some degree, I've just got to tell myself that this won't have been the only time model weights get leaked!

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#25

I womder, could Facebook take legal action here? While some (most of) the data used to train the model is copyrighted, I don't think the model is. It's the result of a mathematical process applied to a series of facts and works with no more creativity put onto them.

That definition would apply to almost anything software produces ^^; We can already have different licenses for compiled binaries vs the source. Also the output of ML seems to belong to whoever pressed the generate button atm.

> That definition would apply to almost anything software produces

Not really. The reason software can be copyrighted at all is because the actual code (and resulting object code) is creative. Courts have named this threshold the "Structure, sequence and organization" of the work. ML models don't follow any creative SSO the way actual code does.

> Also the output of ML seems to belong to whoever pressed the generate button atm.

The output, it seems to me, is uncopyrightable. Copyright only cares about who provides the creativity for the work at issue, not who put in the effort to make it happen. You may own the copyright to your prompt, but the result is generated entirely by the AI and thus lacks human autorship.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#26
post #16

If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…

why is it that these models tend to be released as float16 and converting to int8 is left to the reader? is there something special about training that defaults you to float16?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#27

I womder, could Facebook take legal action here? While some (most of) the data used to train the model is copyrighted, I don't think the model is. It's the result of a mathematical process applied to a series of facts and works with no more creativity put onto them.

As far as my understanding of American copyright goes, a computer produced work cannot be copyrighted as computers are not human, in the same way a photograph taken by a chimp cannot be copyrighted no matter who owned the camera that took the photo. This is one of the major challenges with the legal status of AI as well that will soon be fought over in court.

It's possible that the automated processing of the dataset is considered to be non-creative enough that the generated AI model cannot be copyrighted. The code to train the model and the input dataset (and the works therein) definitely can be, but not the model itself.

In that case, Facebook would be out of luck, as long as the code to train the model isn't shared. If the courts find AI models to be a different type of work that does produce copyrightable models, Facebook may follow in the footsteps of other copyright giants and start filing lawsuits against anyone who they can catch. I very much doubt they'd go so far, especially since by the time they can even start a lawsuit confidently, the leaked model is probably already outdated and irrelevant.

Personally, I expect the model to end up being uncopyrightable, as would be the output of the model.

This may or may not have very interesting results. The dataset itself is probably copyrightable (a human or set of humans composed it, unless that was also done completely automatically) but if that copyright is claimed, the individual right holders of the included works may demand a licensing fee similar to how sound bytes work in music; "you want to use my work, pay me a fee".

Or maybe the dataset is considered to be diverse enough that individual works cannot be expected to be compensated for their inclusion and you can get around copyright law by amassing enough content at once, who knows.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#28
post #18

It's been enough time since this leaked, so my question is why aren't there blog posts already of people blowing their $300 of starter credit with ${cloud_provider} on a few hours' experimentation running inference on this 65B model? Edit: I read the linked README. > I was impatient and curious to try to run 65B on an 8xA100 cluster Well?

The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWea…

I'm running LLaMA-65B on a single A100 80GB with 8bit quantization. $1.5/hr on vast.ai

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#29

I womder, could Facebook take legal action here? While some (most of) the data used to train the model is copyrighted, I don't think the model is. It's the result of a mathematical process applied to a series of facts and works with no more creativity put onto them.

As far as my understanding of American copyright goes, a computer produced work cannot be copyrighted as computers are not human, in the same way a photograph taken by a chimp cannot be copyrighted no matter who owned the camera that took the photo. This is one of the major challenges with the legal status of AI as well that will soon be fought over in court. It's possible that the automated processing of the dataset…

[deleted]

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#30
post #21

Earlier quoted context omitted.

Eagerly awaiting the int8 vs 4 benchmarks. Also, it can run on CPU https://github.com/markasoftware/llama-cpu So, an int8 patch could allow the 65B to run on a standard 128 GB setup assuming the 65B model's cache bursts fit, which if I were to speculate is why the released models stop @ 65B & meta likely already has larger unreleased internal ones.

early int4 experiments seem to indicate it's possible but you do lose performance, see this thread https://www.reddit.com/r/MachineLearning/comments/11i4olx/d_... edit: to clarify, it may be possible to get this loss back and there is reason to be optimistic

Probably the best method is to just train it on int4 in the first place. Fine tuning after quantization would definitely help though.
Post reply on HN