Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

github.com

Post reply on HN
Print view View original

41–50 of 143 posts

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#41

Post by nextaccountic »

v64 wrote: post #16

If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…

If the model weights are stored as int8, does this mean that the floating point capacity of the GPU is wasted? Or the int8 is converted to float in the GPU?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#42

Post by brookst »

sp332 wrote: post #38
CuriouslyC wrote: post #30

Earlier quoted context omitted.

Probably the best method is to just train it on int4 in the first place. Fine tuning after quantization would definitely help though.

Isn't that backwards? You need fairly good resolution during training or your gradients will be pointing all over the place. Once you've found a good minimum point, moving a little away from it with reduced precision is probably OK.

I have no idea what the right answer is, but I think the argument for int4 training is that the loss measurements would take the lower resolution of the model as a whole into account.

Is it better to have billions of high resolution parameters and quantize them at the end, or to train low resolution parameters where the training algorithms see the lower resolution? It’s beyond me, but I’d love to know.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#43

Post by nextaccountic »

kuroguro wrote: post #12
arjvik wrote: post #11

How big is this model? (i.e. disk space to store it)

65B is ~120GB. All of them combined with the smaller versions is ~220GB.

After converting to int8, does it become smaller? Also, can this be further compressed? Like, is there some redundancy a special-purpose compressor could exploit?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#44

Post by smoldesu »

linearalgebra45 wrote: post #22
v64 wrote: post #18

Earlier quoted context omitted.

The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWea…

Are you sure about that? I can't remember where I saw the table of memory requirements, but I'm sure some of the larger instances here [1] will surely be able to cope (assuming they're available!) Oracle gives you a $300 free trial, which equates to running BM.GPU4.8 for over 10 hours - enough for a focused day of prompting [1] https://www.oracle.com/cloud/compute/gpu/

Thanks for sharing it! I'm using their "Always Free" tier to host an Ampere-accelerated GPT-J chatbot right now. Works like a charm, and best of all, it's free!

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#45

Post by brookst »

adossi wrote: post #33
jeroenhd wrote: post #27

Earlier quoted context omitted.

As far as my understanding of American copyright goes, a computer produced work cannot be copyrighted as computers are not human, in the same way a photograph taken by a chimp cannot be copyrighted no matter who owned the camera that took the photo. This is one of the major challenges with the legal status of AI as well that will soon be fought over in court. It's possible that the automated processing of the dataset…

It is intellectual property, regardless of copyright.

“Intellectual property” is a catch-all for copyright, trademark, patent, and trade secrets. There isn’t really law that protects IP as a general concept, just those four.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#46

Post by dspillett »

swyx wrote: post #26
v64 wrote: post #16

If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…

why is it that these models tend to be released as float16 and converting to int8 is left to the reader? is there something special about training that defaults you to float16?

Precision, aiming those names refer to standard binary numeric types. IEEE754 16-bit floats carry 11 significant digits with absolute precision so by coverting to 8-bit integers you lose some of that. Depending on the distribution of the values in those floats you could be loosing a lot more detail then this would imply, which is the reason we use floating point numbers for anything in the first place (rather than using an int16 where you have greater precision at you maximum scale but much less at lower scales).

So if the model is computed using float16s, distribute as-is and let the end user choose to user it like that or compromise for faster processing of there system can deal with many billions of int8s more effectively.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#47

Post by rfoo »

sp332 wrote: post #38
CuriouslyC wrote: post #30

Earlier quoted context omitted.

Probably the best method is to just train it on int4 in the first place. Fine tuning after quantization would definitely help though.

Isn't that backwards? You need fairly good resolution during training or your gradients will be pointing all over the place. Once you've found a good minimum point, moving a little away from it with reduced precision is probably OK.

GP could be mentioning quantization aware training, during which the weight and gradient are still computed in fp16/fp32.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#48

Post by rfoo »

nextaccountic wrote: post #43
kuroguro wrote: post #12

Earlier quoted context omitted.

65B is ~120GB. All of them combined with the smaller versions is ~220GB.

After converting to int8, does it become smaller? Also, can this be further compressed? Like, is there some redundancy a special-purpose compressor could exploit?

Converting to int8 halves the size.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#49

Post by akreal »

MacsHeadroom wrote: post #40
sillysaurusx wrote: post #34

Earlier quoted context omitted.

Careful though — we need to evaluate llama on its own merits. It’s easy to mess up the quantization in subtle ways, then conclude that the outputs aren’t great. So if you’re seeing poor results vs gpt-3, hold off judgement till people have had time to really make sure the quantized models are >97% the effectiveness of the original weights. That said, this is awesome — please share some outputs! What’s it like?

The output is at least as good as davinci. I think some early results are using bad repetition penalty and/or temperature settings. I had to set both fairly high to get the best results. (Some people are also incorrectly comparing it to chatGPT/ChatGPT API which is not a good comparison. But that's a different problem.) I've had it translate, write poems, tell jokes, banter, write executable code. It does it all-- an…

Which prompt did you use for translation? I'd be curious to try it for my task too.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#50

Post by sillysaurusx »

MacsHeadroom wrote: post #40
sillysaurusx wrote: post #34

Earlier quoted context omitted.

Careful though — we need to evaluate llama on its own merits. It’s easy to mess up the quantization in subtle ways, then conclude that the outputs aren’t great. So if you’re seeing poor results vs gpt-3, hold off judgement till people have had time to really make sure the quantized models are >97% the effectiveness of the original weights. That said, this is awesome — please share some outputs! What’s it like?

The output is at least as good as davinci. I think some early results are using bad repetition penalty and/or temperature settings. I had to set both fairly high to get the best results. (Some people are also incorrectly comparing it to chatGPT/ChatGPT API which is not a good comparison. But that's a different problem.) I've had it translate, write poems, tell jokes, banter, write executable code. It does it all-- an…

That's great to hear. Thank you very much, both for reporting this, and especially for the crucial note about temperature.

In fact, sampling settings are so important and so easily underestimated that I should just pester you to post your exact settings. If you get a moment, would you mind sharing your temperature, repetition penalty, top-k, and anything else? I'll be experimenting with those today, but having some known working defaults would be wonderful. (You're also the first person I've seen that got excellent outputs from llama; whatever you did, no one else seems to have noticed yet.)

If you're busy or don't feel like it, no worries though. I'm just grateful you gave us some hope that llama might be really good. There were so many tweet chains showing universally awful outputs that I wasn't sure.

EDIT: I added your comments to the top of the README and credited you. Thanks again.

Post reply on HN