How big is this model? (i.e. disk space to store it)
Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
11–20 of 143 posts
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#12How big is this model? (i.e. disk space to store it)
65B is ~120GB. All of them combined with the smaller versions is ~220GB.
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#13It's been enough time since this leaked, so my question is why aren't there blog posts already of people blowing their $300 of starter credit with ${cloud_provider} on a few hours' experimentation running inference on this 65B model?
Edit: I read the linked README.
> I was impatient and curious to try to run 65B on an 8xA100 cluster
Well?
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#14Earlier quoted context omitted.
Because there are many benchmarks that measure different things. You need to look at the benchmark that reflects your specific interest. So in this case ("I wasn't impressed that 30B didn't seem to know who Captain Picard was") the closest relevant benchmark they performed is MMLU (Massive Multitask Language Understanding"[1]. In the LLAMA paper they publish a figure of 63.4% for the 5-shot average setting without fi…
The capability of a language model I care about most is probably its ability to represent or simulate Captain Picard. In the sense of being good at creative tasks but also Captain Picard, specifically. Is OpenAI deliberately doing something different on purpose that makes their models better for this, or is just that OpenAI has a lot more copyrighted data in their dataset, as I noticed just now when skimming the Face…
It's unclear exactly why it doesn't work as well for you.
I have two comments that may be useful:
1) It's very unclear how good the generative capabilities of LLAMA are generally. It benchmarks well for code generation, but for English there aren't really any good benchmarks around. There's good chance the larger model performs much better since generative capabilities seem to be a partially emergent capability.
2) If you just want to "make it work" I'd suggest downloading all the Star Trek scripts you can that include Captain Picard and fine tuning LLAMA using them. It's unclear how well this will work, but that is probably about as good as you can get.
If you care about this probably deeply, it's probably worth trying the same with some of the other open GPT-3 models (GPTJ, GPT-NEOX etc)
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#15I womder, could Facebook take legal action here? While some (most of) the data used to train the model is copyrighted, I don't think the model is. It's the result of a mathematical process applied to a series of facts and works with no more creativity put onto them.
That definition would apply to almost anything software produces ^^;
We can already have different licenses for compiled binaries vs the source. Also the output of ML seems to belong to whoever pressed the generate button atm.
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#16If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software." I'm very thankful we have researchers like this further democratizing access to this data and prying it out of the hands of the gatekeepers who wish to monetize it.
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#17For even better speeds, perhaps use the link from this script (if it ever goes back up) as a webseed for torrent?
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#18It's been enough time since this leaked, so my question is why aren't there blog posts already of people blowing their $300 of starter credit with ${cloud_provider} on a few hours' experimentation running inference on this 65B model? Edit: I read the linked README. > I was impatient and curious to try to run 65B on an 8xA100 cluster Well?
The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWeave.
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#19If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…
Hmmm, the Github repo suggests that you might be able to run the 65B model on a single A100 80gb card. At the moment, the spot price on Google cloud for this card is $1.25/hour which makes it not so crazy expensive...
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#20If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…
Eagerly awaiting the int8 vs 4 benchmarks. Also, it can run on CPU https://github.com/markasoftware/llama-cpu So, an int8 patch could allow the 65B to run on a standard 128 GB setup assuming the 65B model's cache bursts fit, which if I were to speculate is why the released models stop @ 65B & meta likely already has larger unreleased internal ones.