Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

github.com

Post reply on HN
Print view View original

91–100 of 143 posts

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#91

Post by charcircuit »

swyx wrote: post #26
v64 wrote: post #16

If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…

why is it that these models tend to be released as float16 and converting to int8 is left to the reader? is there something special about training that defaults you to float16?

Quantization and other optimizations are more for productionizing models. You start with something accurate and then you start making tradeoffs to get the inference time to fit into your compute, memory, and time budgets.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#92

Post by e12e »

sillysaurusx wrote: post #4

Update: FB disabled the download link, so I mirrored everything to R2 and updated the script to use it. It should be working now (though the speed is "only" around 50MB/s).

I've read the readme - but I'm not sure why this is any faster than just adding seeds to the torrent? More people downloading via torrent than http?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#93

Post by v64 »

MacsHeadroom wrote: post #40
sillysaurusx wrote: post #34

Earlier quoted context omitted.

Careful though — we need to evaluate llama on its own merits. It’s easy to mess up the quantization in subtle ways, then conclude that the outputs aren’t great. So if you’re seeing poor results vs gpt-3, hold off judgement till people have had time to really make sure the quantized models are >97% the effectiveness of the original weights. That said, this is awesome — please share some outputs! What’s it like?

The output is at least as good as davinci. I think some early results are using bad repetition penalty and/or temperature settings. I had to set both fairly high to get the best results. (Some people are also incorrectly comparing it to chatGPT/ChatGPT API which is not a good comparison. But that's a different problem.) I've had it translate, write poems, tell jokes, banter, write executable code. It does it all-- an…

Note that unlike ChatGPT, these models are pure text completers and have not been trained to be prompted. The llama FAQ [1] mentions this and gives tips for how to get out of the ChatGPT mindset and prompt llama better.

[1] https://github.com/facebookresearch/llama/blob/main/FAQ.md#2

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#94

Post by [deleted] »

EMIRELADERO wrote: post #6

I womder, could Facebook take legal action here? While some (most of) the data used to train the model is copyrighted, I don't think the model is. It's the result of a mathematical process applied to a series of facts and works with no more creativity put onto them.

[deleted]

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#96

Post by davrosthedalek »

rnosov wrote: post #64
sillysaurusx wrote: post #54

Earlier quoted context omitted.

Absolutely! I'll make sure to leave a comment here for you whenever something gets written up so you don't miss it. Getting "as good as davinci" on a single A100 is groundbreaking work. Facebook and the community should both be credited here -- maybe llama-int8 would've been created even if the model hadn't leaked, but I don't think it would've happened so quickly. Everyone is doing phenomenal work, and it's so amazi…

Seconded. Do write it up. I see vast.ai listing interruptible instance with a single A100 80GB at $1/hour which is pretty reasonable. ChatGPT plus is $20/month which would be roughly 20 hours of use and I wont't be lectured like I'm in a kindergarten or something. A bonus point would be to make the writeup accessible for AI challenged developers. Asking for a friend.

I would like to support this request for AI challenged developers :)

For things like these, I always wonder: How much slower would it be to run such a model on a CPU? I mean, clearly a lot less interactive, but is it possible at all? Could it be chopped up and "streamed" to a GPU with less memory halfway efficiently? What is the bottleneck currently on GPUs, memory bw or compute?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#97

Post by bloaf »

causality0 wrote: post #57
v64 wrote: post #16

If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…

I feel like we're less than a decade away from being able to hook LLMs into gaming. How incredible would it be to have NPCs driven by LLM?

I'd be satisfied plugging a game log/history into a system that generates the epic tale of your victory/defeat.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#98

Post by zargon »

yumraj wrote: post #89

What’s the minimum single GPU that’ll work for the smallest model?

This reddit post says that the 7B model consumes about 9.7GB of VRAM (using int8). I'm sure very soon people will add support for using system RAM as swap space, which will allow you to use it on an 8GB card, though with a fairly hefty performance penalty.

https://www.reddit.com/r/MachineLearning/comments/11h3p2x/d_...

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#99

Post by charcircuit »

digitallyfree wrote: post #66
EMIRELADERO wrote: post #6

I womder, could Facebook take legal action here? While some (most of) the data used to train the model is copyrighted, I don't think the model is. It's the result of a mathematical process applied to a series of facts and works with no more creativity put onto them.

There is another angle here besides copyright and that is the sharing of prop/trade secret data. This model is only available to specific orgs who request it (i.e. it's non-public) and I imagine that there are confidentiality terms for the orgs that get the access. Not too familiar with the drama but I believe what happened was that someone with access leaked the torrent used to download the weights. In a legal sense…

You can read the license at this link.

https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z...

There isn't any confidentiality terms.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#100

Post by charcircuit »

rany_ wrote: post #65
ur-whale wrote: post #37

Is this the full model or just the weights? [EDIT]: are there checksums available? [EDIT2]: MD5 signatures seem to be included for all models in checklist.chk files next to them And there's also what the author mentions: the magnet file he provides in his README does seed immediately on the download when loaded in a bt app which is usually a good sign that the files are correct.

MD5 signatures don't mean much now that hash collisions could be created instantly on consumer hardware. MD5 is only good for checking for unintentional data corruption.

>hash collisions could be created instantly on consumer hardware

Collisions can be created, but MD5 is still preimage resistant. As long as someone with the actual model made the hash and Meta didn't generate colliding models themselves. You can trust it.

Post reply on HN