Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

github.com

Post reply on HN
Print view View original

31–40 of 143 posts

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#31

Post by linearalgebra45 »

MacsHeadroom wrote: post #28
v64 wrote: post #18

Earlier quoted context omitted.

The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWea…

I'm running LLaMA-65B on a single A100 80GB with 8bit quantization. $1.5/hr on vast.ai

What instance are you using?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#32

Post by ulnarkressty »

linearalgebra45 wrote: post #13

It's been enough time since this leaked, so my question is why aren't there blog posts already of people blowing their $300 of starter credit with ${cloud_provider} on a few hours' experimentation running inference on this 65B model? Edit: I read the linked README. > I was impatient and curious to try to run 65B on an 8xA100 cluster Well?

https://medium.com/@enryu9000/mini-post-first-look-at-llama-...

*later edit - not the 65G model, but the smaller ones. Performance seems mixed at first glance, not really competitive with ChatGPT fwiw.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#33

Post by adossi »

jeroenhd wrote: post #27
EMIRELADERO wrote: post #6

I womder, could Facebook take legal action here? While some (most of) the data used to train the model is copyrighted, I don't think the model is. It's the result of a mathematical process applied to a series of facts and works with no more creativity put onto them.

As far as my understanding of American copyright goes, a computer produced work cannot be copyrighted as computers are not human, in the same way a photograph taken by a chimp cannot be copyrighted no matter who owned the camera that took the photo. This is one of the major challenges with the legal status of AI as well that will soon be fought over in court. It's possible that the automated processing of the dataset…

It is intellectual property, regardless of copyright.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#34

Post by sillysaurusx »

MacsHeadroom wrote: post #28
v64 wrote: post #18

Earlier quoted context omitted.

The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWea…

I'm running LLaMA-65B on a single A100 80GB with 8bit quantization. $1.5/hr on vast.ai

Careful though — we need to evaluate llama on its own merits. It’s easy to mess up the quantization in subtle ways, then conclude that the outputs aren’t great. So if you’re seeing poor results vs gpt-3, hold off judgement till people have had time to really make sure the quantized models are >97% the effectiveness of the original weights.

That said, this is awesome — please share some outputs! What’s it like?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#35

Post by linearalgebra45 »

ulnarkressty wrote: post #32
linearalgebra45 wrote: post #13

It's been enough time since this leaked, so my question is why aren't there blog posts already of people blowing their $300 of starter credit with ${cloud_provider} on a few hours' experimentation running inference on this 65B model? Edit: I read the linked README. > I was impatient and curious to try to run 65B on an 8xA100 cluster Well?

https://medium.com/@enryu9000/mini-post-first-look-at-llama-... *later edit - not the 65G model, but the smaller ones. Performance seems mixed at first glance, not really competitive with ChatGPT fwiw.

> not the 65G model, but the smaller ones

Haha, that's right! I saw that one too

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#36

Post by sillysaurusx »

swyx wrote: post #26
v64 wrote: post #16

If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…

why is it that these models tend to be released as float16 and converting to int8 is left to the reader? is there something special about training that defaults you to float16?

They were trained in fp16, and researchers tend to release whatever format they trained. It’s hard enough to do a large release that it’s best not to try to have too many goals, for the same reason most software projects try not to do too much lest their schedule slip.

Still, I’m a little sad they didn’t release the optimizer weights. It would’ve given us so much valuable info about the dataset, among other benefits.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#37

Post by ur-whale »

Is this the full model or just the weights?

[EDIT]: are there checksums available?

[EDIT2]: MD5 signatures seem to be included for all models in checklist.chk files next to them

And there's also what the author mentions: the magnet file he provides in his README does seed immediately on the download when loaded in a bt app which is usually a good sign that the files are correct.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#38

Post by sp332 »

CuriouslyC wrote: post #30
v64 wrote: post #21

Earlier quoted context omitted.

early int4 experiments seem to indicate it's possible but you do lose performance, see this thread https://www.reddit.com/r/MachineLearning/comments/11i4olx/d_... edit: to clarify, it may be possible to get this loss back and there is reason to be optimistic

Probably the best method is to just train it on int4 in the first place. Fine tuning after quantization would definitely help though.

Isn't that backwards? You need fairly good resolution during training or your gradients will be pointing all over the place. Once you've found a good minimum point, moving a little away from it with reduced precision is probably OK.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#39

Post by yunyu »

rnosov wrote: post #5
swyx wrote: post #2

thanks for doing this, honestly your writeup seems more valuable than the model weights lol > But for what it's worth, my personal opinion is that LLaMA probably isn't OpenAI-grade -- there's a big difference between training a model in an academic setting vs when your entire company depends on it for wide-scale commercial success. I wasn't impressed that 30B didn't seem to know who Captain Picard was. im new to benc…

You can read the original LLaMA paper which is pretty accessible[1]. For example, they claim to outperform GPT-3 on HellaSwag benchmark ( finishing sentences ). You can find examples of unfinished sentences in the HellaSwag paper [2] on page 13. Unfortunately for LLaMA, most people would be probably just asking questions about Captain Picard and so on, and on this benchmark LLaMA significantly underperforms compared…

Hellaswag is also a deeply flawed benchmark, I wouldn't read too much into it: https://www.surgehq.ai/blog/hellaswag-or-hellabad-36-of-this...

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#40

Post by MacsHeadroom »

sillysaurusx wrote: post #34
MacsHeadroom wrote: post #28

Earlier quoted context omitted.

I'm running LLaMA-65B on a single A100 80GB with 8bit quantization. $1.5/hr on vast.ai

Careful though — we need to evaluate llama on its own merits. It’s easy to mess up the quantization in subtle ways, then conclude that the outputs aren’t great. So if you’re seeing poor results vs gpt-3, hold off judgement till people have had time to really make sure the quantized models are >97% the effectiveness of the original weights. That said, this is awesome — please share some outputs! What’s it like?

The output is at least as good as davinci.

I think some early results are using bad repetition penalty and/or temperature settings. I had to set both fairly high to get the best results. (Some people are also incorrectly comparing it to chatGPT/ChatGPT API which is not a good comparison. But that's a different problem.)

I've had it translate, write poems, tell jokes, banter, write executable code. It does it all-- and all on a single card.

Post reply on HN