Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

github.com

Post reply on HN
Print view View original

51–60 of 143 posts

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#51

Post by linearalgebra45 »

sillysaurusx wrote: post #50
MacsHeadroom wrote: post #40

Earlier quoted context omitted.

The output is at least as good as davinci. I think some early results are using bad repetition penalty and/or temperature settings. I had to set both fairly high to get the best results. (Some people are also incorrectly comparing it to chatGPT/ChatGPT API which is not a good comparison. But that's a different problem.) I've had it translate, write poems, tell jokes, banter, write executable code. It does it all-- an…

That's great to hear. Thank you very much, both for reporting this, and especially for the crucial note about temperature. In fact, sampling settings are so important and so easily underestimated that I should just pester you to post your exact settings. If you get a moment, would you mind sharing your temperature, repetition penalty, top-k, and anything else? I'll be experimenting with those today, but having some k…

Would you mind publishing your notes/learnings once you gain enough understanding of this model?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#52

Post by toomuchtodo »

sillysaurusx wrote: post #4

Update: FB disabled the download link, so I mirrored everything to R2 and updated the script to use it. It should be working now (though the speed is "only" around 50MB/s).

Have you dropped the artifacts in the Internet Archive yet by chance?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#53

Post by jocaal »

smoldesu wrote: post #44
linearalgebra45 wrote: post #22

Earlier quoted context omitted.

Are you sure about that? I can't remember where I saw the table of memory requirements, but I'm sure some of the larger instances here [1] will surely be able to cope (assuming they're available!) Oracle gives you a $300 free trial, which equates to running BM.GPU4.8 for over 10 hours - enough for a focused day of prompting [1] https://www.oracle.com/cloud/compute/gpu/

Thanks for sharing it! I'm using their "Always Free" tier to host an Ampere-accelerated GPT-J chatbot right now. Works like a charm, and best of all, it's free!

I don't understand, the Ampere they refer to in their free tier are cpu's not gpu's. How did you manage to do that

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#54

Post by sillysaurusx »

linearalgebra45 wrote: post #51
sillysaurusx wrote: post #50

Earlier quoted context omitted.

That's great to hear. Thank you very much, both for reporting this, and especially for the crucial note about temperature. In fact, sampling settings are so important and so easily underestimated that I should just pester you to post your exact settings. If you get a moment, would you mind sharing your temperature, repetition penalty, top-k, and anything else? I'll be experimenting with those today, but having some k…

Would you mind publishing your notes/learnings once you gain enough understanding of this model?

Absolutely! I'll make sure to leave a comment here for you whenever something gets written up so you don't miss it.

Getting "as good as davinci" on a single A100 is groundbreaking work. Facebook and the community should both be credited here -- maybe llama-int8 would've been created even if the model hadn't leaked, but I don't think it would've happened so quickly. Everyone is doing phenomenal work, and it's so amazing to see it all come together.

But, we'll see. Going to try it myself soon.

Long ago, I cloned OpenAI's API: https://github.com/shawwn/openai-server -- my plan is, once I get it running, I'll try to host it somewhere so that anyone can play with it. I assume it'll be quickly swamped, but it's still an interesting challenge; some basic load balancing should make it scalable across several A100 instances, so there's no reason we can't just roll our own OpenAI API.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#56

Post by smoldesu »

jocaal wrote: post #53
smoldesu wrote: post #44

Earlier quoted context omitted.

Thanks for sharing it! I'm using their "Always Free" tier to host an Ampere-accelerated GPT-J chatbot right now. Works like a charm, and best of all, it's free!

I don't understand, the Ampere they refer to in their free tier are cpu's not gpu's. How did you manage to do that

Custom PyTorch with on-chip acceleration: https://cloudmarketplace.oracle.com/marketplace/en_US/listin...

Not as fast as a GPU, but less than 5 seconds for a 250 token response is good enough for a Discord bot.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#57

Post by causality0 »

v64 wrote: post #16

If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…

I feel like we're less than a decade away from being able to hook LLMs into gaming. How incredible would it be to have NPCs driven by LLM?

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#58

Post by cma »

adossi wrote: post #33
jeroenhd wrote: post #27

Earlier quoted context omitted.

As far as my understanding of American copyright goes, a computer produced work cannot be copyrighted as computers are not human, in the same way a photograph taken by a chimp cannot be copyrighted no matter who owned the camera that took the photo. This is one of the major challenges with the legal status of AI as well that will soon be fought over in court. It's possible that the automated processing of the dataset…

It is intellectual property, regardless of copyright.

It isn't protected as a trade secret if they mostly freely shared it with .edu addresses. And once it has been leaked out widely publicly it isn't either.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#60

Post by mynameisvlad »

linearalgebra45 wrote: post #24
v64 wrote: post #23

Earlier quoted context omitted.

> Are you sure about that? I'm not. The only way to know it is to try :) thank you for the link!

You only get a single month-long window to spend the credit! And I'm sure not going to spend any of my own money on prompting experiments. I might be suffering from FOMO to some degree, I've just got to tell myself that this won't have been the only time model weights get leaked!

> And I'm sure not going to spend any of my own money on prompting experiments.

This certainly sounds a lot like whining that others aren’t doing the work you yourself don’t want to do.

Post reply on HN