Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

github.com

Post reply on HN
Print view View original

111–120 of 143 posts

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#111

Post by nl »

CuriouslyC wrote: post #30
v64 wrote: post #21

Earlier quoted context omitted.

early int4 experiments seem to indicate it's possible but you do lose performance, see this thread https://www.reddit.com/r/MachineLearning/comments/11i4olx/d_... edit: to clarify, it may be possible to get this loss back and there is reason to be optimistic

Probably the best method is to just train it on int4 in the first place. Fine tuning after quantization would definitely help though.

> Probably the best method is to just train it on int4 in the first place

Unclear why you think that since experiments show the opposite.

In general the gradient seems to get too "bumpy" to do good gradient decent at lower levels of precision.

There are some papers showing that making the training loop aware of quantitization can help ultimate quantizied performance but I'm not aware of this being implemented at large scale.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#112

Post by ALittleLight »

version_five wrote: post #59

Thanks for doing what Facebook should have been mature / humble enough to have done on their own. The best outcome of this would be for FB to stop the silliness and just release the weights openly themselves.

This may be a good compromise for plausible deniability. Facebook can be "responsible" and release only to "researchers" and the public can get the model from torrents shortly after. So long as the models keep going public I'm satisfied.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#113

Post by bick_nyers »

brookst wrote: post #42
sp332 wrote: post #38

Earlier quoted context omitted.

Isn't that backwards? You need fairly good resolution during training or your gradients will be pointing all over the place. Once you've found a good minimum point, moving a little away from it with reduced precision is probably OK.

I have no idea what the right answer is, but I think the argument for int4 training is that the loss measurements would take the lower resolution of the model as a whole into account. Is it better to have billions of high resolution parameters and quantize them at the end, or to train low resolution parameters where the training algorithms see the lower resolution? It’s beyond me, but I’d love to know.

I think the answer is it depends, and further, a dynamic approach may be best. Imagine you are going on a hike, and you have different maps at various resolutions (levels of detail). When planning the hike, you will want to see the zoomed out picture to get general directions, elevations and landmarks identified. Then you can zoom in to see the actual trails themselves, to identify your route, and then you zoom in even further when you are on the ground actually walking, avoiding obstacles along the way.

Different resolutions draw your attention to different types of features.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#114

Post by bick_nyers »

nl wrote: post #111
CuriouslyC wrote: post #30

Earlier quoted context omitted.

Probably the best method is to just train it on int4 in the first place. Fine tuning after quantization would definitely help though.

> Probably the best method is to just train it on int4 in the first place Unclear why you think that since experiments show the opposite. In general the gradient seems to get too "bumpy" to do good gradient decent at lower levels of precision. There are some papers showing that making the training loop aware of quantitization can help ultimate quantizied performance but I'm not aware of this being implemented at larg…

What if you smooth the gradient, either by interpolating/removing data points that make the surface "jagged", or maybe change the "window" of gradient descent, meaning instead of using a tangent (derivative, infinitesimally small window) you use a secant (???, window of specified length, likely calculated from the data space).

Forgive my lack of proper terminology here.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#115

Post by bick_nyers »

SloopJon wrote: post #67
causality0 wrote: post #57

Earlier quoted context omitted.

I feel like we're less than a decade away from being able to hook LLMs into gaming. How incredible would it be to have NPCs driven by LLM?

There was an Ask HN post about that idea a couple of months ago: https://news.ycombinator.com/item?id=34478503 I have long wished for less linear stories in video games, where branching narrative (a la Choose Your Own Adventure) is one possible way to give the player agency. The problem is, true branches are expensive, because you end up writing a bunch of content the player never experiences. I see a lot of potentia…

In general I would say story =/= dialogue (which an LLM can much more easily be used for). I see two main "tricks" that would make the more complicated case (story) possible.

1. You bound the branching in a particular fashion, and provide overall "pressures" into certain story arcs.

2. You use generative AI in a LOT more places in the game.

What happens when you are playing a Sci-Fi game, and you get the enemy NPC to somehow hallucinate that he is the King of Dragons, but you don't have Dragon models/animations/movesets in your game files? You either bound the LLM to not hallucinate that, or you generate that dragon live. I guess a 3rd option, is your game is a comedy and the King NPC gets labeled a crazy person.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#116

Post by davrosthedalek »

nl wrote: post #107
davrosthedalek wrote: post #96

Earlier quoted context omitted.

I would like to support this request for AI challenged developers :) For things like these, I always wonder: How much slower would it be to run such a model on a CPU? I mean, clearly a lot less interactive, but is it possible at all? Could it be chopped up and "streamed" to a GPU with less memory halfway efficiently? What is the bottleneck currently on GPUs, memory bw or compute?

On a CPU I'd estimate it would get a maximum of around 5 tokens per second (a token being a sub-word token, so generally a couple of letters). I suspect it'd be more like 1 token per second on the large model without additional optimisation. Yes models can be split up. See eg Hugging Face Accelerate.

That's actually a lot better than I would have thought. Almost usable, and a good exercise in patience.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#117

Post by nl »

bick_nyers wrote: post #114
nl wrote: post #111

Earlier quoted context omitted.

> Probably the best method is to just train it on int4 in the first place Unclear why you think that since experiments show the opposite. In general the gradient seems to get too "bumpy" to do good gradient decent at lower levels of precision. There are some papers showing that making the training loop aware of quantitization can help ultimate quantizied performance but I'm not aware of this being implemented at larg…

What if you smooth the gradient, either by interpolating/removing data points that make the surface "jagged", or maybe change the "window" of gradient descent, meaning instead of using a tangent (derivative, infinitesimally small window) you use a secant (???, window of specified length, likely calculated from the data space). Forgive my lack of proper terminology here.

Sure, there are multiple ways to reduce the complexity of your loss-space, but the issue is that you usually want these small gradient values because they are important. Roughly if you "smooth over what appears to be a small hole" often you'll miss a large space that needs to be explored (obviously this is multi-dimensional but you get the idea).

However you can reduce memory by doing mixed-precision training if you are careful. See section "2.3.1. Loss Scaling To Preserve Small Gradient Magnitudes" in https://docs.nvidia.com/deeplearning/performance/mixed-preci...

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#118

Post by nl »

davrosthedalek wrote: post #116
nl wrote: post #107

Earlier quoted context omitted.

On a CPU I'd estimate it would get a maximum of around 5 tokens per second (a token being a sub-word token, so generally a couple of letters). I suspect it'd be more like 1 token per second on the large model without additional optimisation. Yes models can be split up. See eg Hugging Face Accelerate.

That's actually a lot better than I would have thought. Almost usable, and a good exercise in patience.

I'd expect significant performance improvements over the next few months are more people work on this in the same way the stable diffusion is now fairly usable on a CPU. It's always going to be slow on a CPU, but the smaller models might be usable for experimentation at some point.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#119

Post by bick_nyers »

nl wrote: post #117
bick_nyers wrote: post #114

Earlier quoted context omitted.

What if you smooth the gradient, either by interpolating/removing data points that make the surface "jagged", or maybe change the "window" of gradient descent, meaning instead of using a tangent (derivative, infinitesimally small window) you use a secant (???, window of specified length, likely calculated from the data space). Forgive my lack of proper terminology here.

Sure, there are multiple ways to reduce the complexity of your loss-space, but the issue is that you usually want these small gradient values because they are important. Roughly if you "smooth over what appears to be a small hole" often you'll miss a large space that needs to be explored (obviously this is multi-dimensional but you get the idea). However you can reduce memory by doing mixed-precision training if you…

So then you would need to do some kind of mesh simplification that also preserves the topology, that makes sense.

I'm not quite sure I understand what they are describing in 2.3.1, are they scaling those small gradient magnitudes larger to try to "pull" you into those holes faster?

I was thinking the a way to go about it would be to just increase the "mesh resolution" near the small hole, which in this case would be use a larger precision in the area local to the hole.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#120

Post by CuriouslyC »

nl wrote: post #111
CuriouslyC wrote: post #30

Earlier quoted context omitted.

Probably the best method is to just train it on int4 in the first place. Fine tuning after quantization would definitely help though.

> Probably the best method is to just train it on int4 in the first place Unclear why you think that since experiments show the opposite. In general the gradient seems to get too "bumpy" to do good gradient decent at lower levels of precision. There are some papers showing that making the training loop aware of quantitization can help ultimate quantizied performance but I'm not aware of this being implemented at larg…

My take away was that the reduced performance of natively trained models was more about numerical instability related to training process than a statement about limitations of low precision models.
Post reply on HN