Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

github.com

Post reply on HN
Print view View original

71–80 of 143 posts

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#71

Post by damascus »

smoldesu wrote: post #44
linearalgebra45 wrote: post #22

Earlier quoted context omitted.

Are you sure about that? I can't remember where I saw the table of memory requirements, but I'm sure some of the larger instances here [1] will surely be able to cope (assuming they're available!) Oracle gives you a $300 free trial, which equates to running BM.GPU4.8 for over 10 hours - enough for a focused day of prompting [1] https://www.oracle.com/cloud/compute/gpu/

Thanks for sharing it! I'm using their "Always Free" tier to host an Ampere-accelerated GPT-J chatbot right now. Works like a charm, and best of all, it's free!

Do you have any code from your discord bot you're willing to share? I'd be happy to share back any updates I made to it. I've been wanting to play with this idea for a bit.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#72

Post by youssefabdelm »

MacsHeadroom wrote: post #28
v64 wrote: post #18

Earlier quoted context omitted.

The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWea…

I'm running LLaMA-65B on a single A100 80GB with 8bit quantization. $1.5/hr on vast.ai

What's the speed like? How many tokens per second? / Is it as fast as say ChatGPT?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#73

Post by [deleted] »

rany_ wrote: post #65
ur-whale wrote: post #37

Is this the full model or just the weights? [EDIT]: are there checksums available? [EDIT2]: MD5 signatures seem to be included for all models in checklist.chk files next to them And there's also what the author mentions: the magnet file he provides in his README does seed immediately on the download when loaded in a bt app which is usually a good sign that the files are correct.

MD5 signatures don't mean much now that hash collisions could be created instantly on consumer hardware. MD5 is only good for checking for unintentional data corruption.

[deleted]

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#74

Post by minxomat »

ulnarkressty wrote: post #32
linearalgebra45 wrote: post #13

It's been enough time since this leaked, so my question is why aren't there blog posts already of people blowing their $300 of starter credit with ${cloud_provider} on a few hours' experimentation running inference on this 65B model? Edit: I read the linked README. > I was impatient and curious to try to run 65B on an 8xA100 cluster Well?

https://medium.com/@enryu9000/mini-post-first-look-at-llama-... *later edit - not the 65G model, but the smaller ones. Performance seems mixed at first glance, not really competitive with ChatGPT fwiw.

> not really competitive with ChatGPT

That's impossible to judge. LLama is a foundational model. It has received neither instructional fine tuning (davinci-3) nor RLHF (ChatGPT). It cannot be compared to these finetuned models without, well, finetuning.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#75

Post by swalsh »

vanillax wrote: post #70

How or what can someone do with this who isn't a ML expert? Is there some docker app that leverages this? To the average dev, is this useful to me? I know there's lots of "plug and play" style docker apps to get started with Stable Diffusion. I'm curious if I can do something fun with this.

Not an expert, but i've downloaded the model, and used it. What you get is pretty raw, and not super useful by itself. There are some projects to try and do some RLHF on it, and with that we might start to get something you can do some useful stuff with.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#76

Post by [deleted] »

vanillax wrote: post #70

How or what can someone do with this who isn't a ML expert? Is there some docker app that leverages this? To the average dev, is this useful to me? I know there's lots of "plug and play" style docker apps to get started with Stable Diffusion. I'm curious if I can do something fun with this.

[deleted]

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#77

Post by kkielhofner »

vanillax wrote: post #70

How or what can someone do with this who isn't a ML expert? Is there some docker app that leverages this? To the average dev, is this useful to me? I know there's lots of "plug and play" style docker apps to get started with Stable Diffusion. I'm curious if I can do something fun with this.

You can shortcut a lot of the steps in these various guides by using the Pytorch container from Nvidia[0].

It shouldn't be too hard for someone (me?) to create a Dockerfile and Docker hub container FROM this image to get it up and running easily.

[0] - https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorc...

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#78

Post by Scene_Cast2 »

brookst wrote: post #42
sp332 wrote: post #38

Earlier quoted context omitted.

Isn't that backwards? You need fairly good resolution during training or your gradients will be pointing all over the place. Once you've found a good minimum point, moving a little away from it with reduced precision is probably OK.

I have no idea what the right answer is, but I think the argument for int4 training is that the loss measurements would take the lower resolution of the model as a whole into account. Is it better to have billions of high resolution parameters and quantize them at the end, or to train low resolution parameters where the training algorithms see the lower resolution? It’s beyond me, but I’d love to know.

But by default, training algos don't see the lower resolution, your gradient just doesn't work as well. There is a body of research on how to make training aware of / adapt to the lower precision.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#79

Post by causality0 »

SloopJon wrote: post #67
causality0 wrote: post #57

Earlier quoted context omitted.

I feel like we're less than a decade away from being able to hook LLMs into gaming. How incredible would it be to have NPCs driven by LLM?

There was an Ask HN post about that idea a couple of months ago: https://news.ycombinator.com/item?id=34478503 I have long wished for less linear stories in video games, where branching narrative (a la Choose Your Own Adventure) is one possible way to give the player agency. The problem is, true branches are expensive, because you end up writing a bunch of content the player never experiences. I see a lot of potentia…

I much prefer handcrafted stories and quests. Characters that respond dynamically to the story and the player's actions, however, is quite tantalizing.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#80

Post by sillysaurusx »

toomuchtodo wrote: post #52
sillysaurusx wrote: post #4

Update: FB disabled the download link, so I mirrored everything to R2 and updated the script to use it. It should be working now (though the speed is "only" around 50MB/s).

Have you dropped the artifacts in the Internet Archive yet by chance?

I'm surprised Internet Archive is appropriate for a 220GB model weight dump.

Please feel free; it seems like a good idea. I'm not sure I have enough weekend left to figure out yet another upload service today.

Post reply on HN