Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

github.com

Post reply on HN
Print view View original

131–140 of 143 posts

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#131

Post by lumost »

bick_nyers wrote: post #119
nl wrote: post #117

Earlier quoted context omitted.

Sure, there are multiple ways to reduce the complexity of your loss-space, but the issue is that you usually want these small gradient values because they are important. Roughly if you "smooth over what appears to be a small hole" often you'll miss a large space that needs to be explored (obviously this is multi-dimensional but you get the idea). However you can reduce memory by doing mixed-precision training if you…

So then you would need to do some kind of mesh simplification that also preserves the topology, that makes sense. I'm not quite sure I understand what they are describing in 2.3.1, are they scaling those small gradient magnitudes larger to try to "pull" you into those holes faster? I was thinking the a way to go about it would be to just increase the "mesh resolution" near the small hole, which in this case would be…

I suspect that changing the resolution around hot points in the manifold would be a more expensive task than training the model on a higher global resolution. Optimization algorithms currently do not maintain state on the loss-manifold.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#132

Post by data_maan »

MacsHeadroom wrote: post #40
sillysaurusx wrote: post #34

Earlier quoted context omitted.

Careful though — we need to evaluate llama on its own merits. It’s easy to mess up the quantization in subtle ways, then conclude that the outputs aren’t great. So if you’re seeing poor results vs gpt-3, hold off judgement till people have had time to really make sure the quantized models are >97% the effectiveness of the original weights. That said, this is awesome — please share some outputs! What’s it like?

The output is at least as good as davinci. I think some early results are using bad repetition penalty and/or temperature settings. I had to set both fairly high to get the best results. (Some people are also incorrectly comparing it to chatGPT/ChatGPT API which is not a good comparison. But that's a different problem.) I've had it translate, write poems, tell jokes, banter, write executable code. It does it all-- an…

Is it just the RLHF training for the prompting that makes a difference, or are there also other, more tangible differences?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#133

Post by bick_nyers »

lumost wrote: post #131
bick_nyers wrote: post #119

Earlier quoted context omitted.

So then you would need to do some kind of mesh simplification that also preserves the topology, that makes sense. I'm not quite sure I understand what they are describing in 2.3.1, are they scaling those small gradient magnitudes larger to try to "pull" you into those holes faster? I was thinking the a way to go about it would be to just increase the "mesh resolution" near the small hole, which in this case would be…

I suspect that changing the resolution around hot points in the manifold would be a more expensive task than training the model on a higher global resolution. Optimization algorithms currently do not maintain state on the loss-manifold.

My naive (and I do mean naive) thought here is that you just need a cheap detection function of when you need to swap precision. I'm pretty stuck on the geometric interpretation here but basically if the training step is "within a radius" of a known hot point of the manifold then you swap precision. It's very possible though that I am hallucinating something that is not possible, I don't actually understand how this stuff really works yet.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#134

Post by ElFitz »

causality0 wrote: post #79
SloopJon wrote: post #67

Earlier quoted context omitted.

There was an Ask HN post about that idea a couple of months ago: https://news.ycombinator.com/item?id=34478503 I have long wished for less linear stories in video games, where branching narrative (a la Choose Your Own Adventure) is one possible way to give the player agency. The problem is, true branches are expensive, because you end up writing a bunch of content the player never experiences. I see a lot of potentia…

I much prefer handcrafted stories and quests. Characters that respond dynamically to the story and the player's actions, however, is quite tantalizing.

We could have handcrafted stories and quests, with LLM-driven dialogues for NPCs canned responses (ie the infamous arrow and the proverbial knee).

And teams with limited resources could also still handcraft the stories and quests but use LLMs to generate or add some variety or context awareness to the dialogues, at a lower cost.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#135

Post by cthalupa »

kuroguro wrote: post #15
EMIRELADERO wrote: post #6

I womder, could Facebook take legal action here? While some (most of) the data used to train the model is copyrighted, I don't think the model is. It's the result of a mathematical process applied to a series of facts and works with no more creativity put onto them.

That definition would apply to almost anything software produces ^^; We can already have different licenses for compiled binaries vs the source. Also the output of ML seems to belong to whoever pressed the generate button atm.

>Also the output of ML seems to belong to whoever pressed the generate button atm.

So far the rulings in the US, at least, do not support this.

https://arstechnica.com/information-technology/2023/02/us-co...

In this case, it was images generated via Midjourney and not the output of an LLM, but my layman's understanding of the result here would be equally applicable to LLM output. Effectively, the copyright office does not consider putting in a prompt enough for there to be "human authorship" of the work. In this specific case, that resulted on the images in the comic being considered uncopyrightable. The broader comic, in the organization of the images, the plot and dialogue, etc., still enjoys copyright protection. But in the US, I could just directly take the images in the comic that Midjourney produced and use them for another purpose without violating copyright.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#136

Post by lumost »

bick_nyers wrote: post #133
lumost wrote: post #131

Earlier quoted context omitted.

I suspect that changing the resolution around hot points in the manifold would be a more expensive task than training the model on a higher global resolution. Optimization algorithms currently do not maintain state on the loss-manifold.

My naive (and I do mean naive) thought here is that you just need a cheap detection function of when you need to swap precision. I'm pretty stuck on the geometric interpretation here but basically if the training step is "within a radius" of a known hot point of the manifold then you swap precision. It's very possible though that I am hallucinating something that is not possible, I don't actually understand how this…

The challenge here is knowing the shape of the manifold within an epsilon radius 65 Billion dimension sphere around the position being evaluated. To calculate this you would need to sample points within epsilon radius around the current point. As these points will be lower-precision by default, you would have minimal knowledge of the shape of the manifold within the sphere if epsilon is It might be possible to work around this by estimating the gradient volatility through the n^th order derivatives, but you would then also have to deal with mixed precision SIMD which hardware doesn't really support.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#137

Post by ZunarJ5 »

causality0 wrote: post #57
v64 wrote: post #16

If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…

I feel like we're less than a decade away from being able to hook LLMs into gaming. How incredible would it be to have NPCs driven by LLM?

There are already several plugins for Unreal Engine. I am going to assume the same for Unity.

https://www.youtube.com/watch?v=i-Aw32rgM-w&ab_channel=Kella...

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#138

Post by MacsHeadroom »

youssefabdelm wrote: post #72
MacsHeadroom wrote: post #28

Earlier quoted context omitted.

I'm running LLaMA-65B on a single A100 80GB with 8bit quantization. $1.5/hr on vast.ai

What's the speed like? How many tokens per second? / Is it as fast as say ChatGPT?

It's about as fast as chatGPT when chatGPT first launched. Not as fast as the new "Turbo" version of chatGPT, but much faster than you or anyone can read (so I'm not sure the difference matters).

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#139

Post by youssefabdelm »

MacsHeadroom wrote: post #138
youssefabdelm wrote: post #72

Earlier quoted context omitted.

What's the speed like? How many tokens per second? / Is it as fast as say ChatGPT?

It's about as fast as chatGPT when chatGPT first launched. Not as fast as the new "Turbo" version of chatGPT, but much faster than you or anyone can read (so I'm not sure the difference matters).

That's awesome! thanks!

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#140

Post by dangoodmanUT »

MacsHeadroom wrote: post #28
v64 wrote: post #18

Earlier quoted context omitted.

The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWea…

I'm running LLaMA-65B on a single A100 80GB with 8bit quantization. $1.5/hr on vast.ai

Did you modify the lambda.cpp repo to move frmo 4-bit to 8-bit quantization? Or did you write something custom?
Post reply on HN