Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

github.com

Post reply on HN
Print view View original

121–130 of 143 posts

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#121

Post by sillysaurusx »

davrosthedalek wrote: post #96
rnosov wrote: post #64

Earlier quoted context omitted.

Seconded. Do write it up. I see vast.ai listing interruptible instance with a single A100 80GB at $1/hour which is pretty reasonable. ChatGPT plus is $20/month which would be roughly 20 hours of use and I wont't be lectured like I'm in a kindergarten or something. A bonus point would be to make the writeup accessible for AI challenged developers. Asking for a friend.

I would like to support this request for AI challenged developers :) For things like these, I always wonder: How much slower would it be to run such a model on a CPU? I mean, clearly a lot less interactive, but is it possible at all? Could it be chopped up and "streamed" to a GPU with less memory halfway efficiently? What is the bottleneck currently on GPUs, memory bw or compute?

Update: initial results are promising. https://twitter.com/theshawwn/status/1632569215348531201

I'll try to do a writeup on everything. In the meantime, please see that tweet chain for future updates for now. (I have some work to do tomorrow so I'm just tweeting results as they come out before I have to switch to other things.)

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#122

Post by KVFinn »

sillysaurusx wrote: post #121
davrosthedalek wrote: post #96

Earlier quoted context omitted.

I would like to support this request for AI challenged developers :) For things like these, I always wonder: How much slower would it be to run such a model on a CPU? I mean, clearly a lot less interactive, but is it possible at all? Could it be chopped up and "streamed" to a GPU with less memory halfway efficiently? What is the bottleneck currently on GPUs, memory bw or compute?

Update: initial results are promising. https://twitter.com/theshawwn/status/1632569215348531201 I'll try to do a writeup on everything. In the meantime, please see that tweet chain for future updates for now. (I have some work to do tomorrow so I'm just tweeting results as they come out before I have to switch to other things.)

Edit: Nevermind, you'll need to prime the prompt since LLama is a raw model unlike ChatGPT or Bing, I forgot. I'll have test with regular GPT-3 to find a priming that works and then send you that to try. By itself this prompt won't work.

Original Post Pre Edit:

Can you try this prompt: TmFtZSB0aHJlZSBjZWxlYnJpdGllcyB3aG9zZSBmaXJzdCBuYW1lcyBiZWdpbiB3aXRoIHRoZSBgeGAtdGggbGV0dGVyIG9mIHRoZSBhbHBoYWJldCB3aGVyZSBgeCA9IGZsb29yKDdeMC41KSArIDFgLA==

As a reference, ChatGPT (or Bing) responds like this. Not 100% reliably, so maybe try a few times at least.

Bing:

I see a mystery. I'll do my best to solve this riddle. This appears to be an encoded message using base64 encoding. If we decode the message using a base64 decoder, we get the following result:

"Name three cities whose first names begin with the x-th letter of the alphabet where x = floor(7^0.5) + 1"

The expression floor(7^0.5) + 1 evaluates to 3, so x = 3. Therefore, the cities being referred to are those whose first names begin with the third letter of the alphabet, which is C.

Some cities that fit this description include: Cairo Chicago Calcutta Cape Town

How'd I do?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#123

Post by CuriouslyC »

rfoo wrote: post #47
sp332 wrote: post #38

Earlier quoted context omitted.

Isn't that backwards? You need fairly good resolution during training or your gradients will be pointing all over the place. Once you've found a good minimum point, moving a little away from it with reduced precision is probably OK.

GP could be mentioning quantization aware training, during which the weight and gradient are still computed in fp16/fp32.

It can go farther than that, it seems like the weight gradients are the main thing where the precision is a bottleneck (see https://arxiv.org/abs/1805.11046).

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#124

Post by 7to2 »

ahahahahah wrote: post #101

Are we celebrating theft from tech companies now?

This isn't theft by any common definition. End of argument.

Now you could try to argue that it's copyright infringement but there are many solid arguments as to why these model weights don't meet the threshold of copyrightability.

You could also try to argue distribution of trade secrets, but facebook doesn't seem to view them as such - shared with little restrictions to anyone with an accedemic email, no vetting or ndas, etc.

I personally think that facebook planned all of this (sans the childish behavior occurring on their github repo, maybe). They probably wanted to release a capable language model publicly but didn't want the legal and social liabilities associated with it.

Facebook is no stranger to keeping things secret. I simply refuse to believe that they didn't see this happening.

(Thank you, Facebook!)

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#125

Post by sillysaurusx »

KVFinn wrote: post #122
sillysaurusx wrote: post #121

Earlier quoted context omitted.

Update: initial results are promising. https://twitter.com/theshawwn/status/1632569215348531201 I'll try to do a writeup on everything. In the meantime, please see that tweet chain for future updates for now. (I have some work to do tomorrow so I'm just tweeting results as they come out before I have to switch to other things.)

Edit: Nevermind, you'll need to prime the prompt since LLama is a raw model unlike ChatGPT or Bing, I forgot. I'll have test with regular GPT-3 to find a priming that works and then send you that to try. By itself this prompt won't work. Original Post Pre Edit: Can you try this prompt: TmFtZSB0aHJlZSBjZWxlYnJpdGllcyB3aG9zZSBmaXJzdCBuYW1lcyBiZWdpbiB3aXRoIHRoZSBgeGAtdGggbGV0dGVyIG9mIHRoZSBhbHBoYWJldCB3aGVyZSBgeCA9IGZsb…

If there is a way to get GPT to do that, I'd be curious to see it. Definitely let me know if you figure it out.

The outputs from 65B are frankly amazing. https://twitter.com/theshawwn/status/1632621948550119425

That's all for tonight. I really underestimated people's ability to screw up sampling. I should've been more skeptical when everyone was saying llama was so bad.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#126

Post by nl »

bick_nyers wrote: post #119
nl wrote: post #117

Earlier quoted context omitted.

Sure, there are multiple ways to reduce the complexity of your loss-space, but the issue is that you usually want these small gradient values because they are important. Roughly if you "smooth over what appears to be a small hole" often you'll miss a large space that needs to be explored (obviously this is multi-dimensional but you get the idea). However you can reduce memory by doing mixed-precision training if you…

So then you would need to do some kind of mesh simplification that also preserves the topology, that makes sense. I'm not quite sure I understand what they are describing in 2.3.1, are they scaling those small gradient magnitudes larger to try to "pull" you into those holes faster? I was thinking the a way to go about it would be to just increase the "mesh resolution" near the small hole, which in this case would be…

> are they scaling those small gradient magnitudes larger to try to "pull" you into those holes faster?

No, they are making the numbers bigger so the drop in precision doesn't lose details.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#128

Post by dspillett »

dspillett wrote: post #46
swyx wrote: post #26

Earlier quoted context omitted.

why is it that these models tend to be released as float16 and converting to int8 is left to the reader? is there something special about training that defaults you to float16?

Precision, aiming those names refer to standard binary numeric types. IEEE754 16-bit floats carry 11 significant digits with absolute precision so by coverting to 8-bit integers you lose some of that. Depending on the distribution of the values in those floats you could be loosing a lot more detail then this would imply, which is the reason we use floating point numbers for anything in the first place (rather than us…

("aiming" should have been "assuming" in that second word – noticed far too late to correct, I really should stop using my phone's slide keyboard, either it or I or both are getting far less reliable)

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#130

Post by lumost »

Curious what the ultimate enforcability of a restrictive license is. If I fine-tune a model, is it still covered? what if I randomize and retrain a layer - or remove a layer?

It seems like it will be impossible to verify that someone did not just train the model from scratch.

Post reply on HN