Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

github.com

Post reply on HN
Print view View original

81–90 of 143 posts

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#81

Post by metadat »

nabla9 wrote: post #62
rnosov wrote: post #19

Earlier quoted context omitted.

Hmmm, the Github repo suggests that you might be able to run the 65B model on a single A100 80gb card. At the moment, the spot price on Google cloud for this card is $1.25/hour which makes it not so crazy expensive...

$1.25/hour is roughly a year of GPU time until it exceeds the price of A100 80GB card.

I think OP meant that $1.25/hr makes this accessible for people try it out themselves cost effectively, without having to spend thousands or tens of thousands up front to obtain a capable hardware rig.

Obviously $1.25/hr 24/7 does add up quickly, after one month the bill would come to $900.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#82

Post by [deleted] »

metadat wrote: post #81
nabla9 wrote: post #62

Earlier quoted context omitted.

$1.25/hour is roughly a year of GPU time until it exceeds the price of A100 80GB card.

I think OP meant that $1.25/hr makes this accessible for people try it out themselves cost effectively, without having to spend thousands or tens of thousands up front to obtain a capable hardware rig. Obviously $1.25/hr 24/7 does add up quickly, after one month the bill would come to $900.

[deleted]

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#83

Post by woodson »

nextaccountic wrote: post #41
v64 wrote: post #16

If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…

If the model weights are stored as int8, does this mean that the floating point capacity of the GPU is wasted? Or the int8 is converted to float in the GPU?

Well, tensor cores support int8 instructions (at least from Turing onwards), so the hardware is being used, if that’s your concern.

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#84

Post by jesusofnazarath »

visarga wrote: post #63
causality0 wrote: post #57

Earlier quoted context omitted.

I feel like we're less than a decade away from being able to hook LLMs into gaming. How incredible would it be to have NPCs driven by LLM?

We'll soon have LLMs in operating systems, LLMs in browsers and you are right, probably also in games. LLMs will be the platform on which we build almost everything.

[dead]

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#85

Post by bitL »

loufe wrote: post #61
bitL wrote: post #9

How does LLaMA handle fast fine-tuning? Are they using transformer adapters for it?

It's already been adapted for hugging face transformers[1]. Apparently that should unlock its full potential. Oobabooga integrated the change into text-generation-webui[2] meaning we can already access a large chunk of its potential (from what I understand). [1] https://github.com/huggingface/transformers/pull/21955 [2] https://github.com/oobabooga/text-generation-webui/commit/90...

That's absolutely fantastic! Thanks for the links!

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#86

Post by [deleted] »

damascus wrote: post #71
smoldesu wrote: post #44

Earlier quoted context omitted.

Thanks for sharing it! I'm using their "Always Free" tier to host an Ampere-accelerated GPT-J chatbot right now. Works like a charm, and best of all, it's free!

Do you have any code from your discord bot you're willing to share? I'd be happy to share back any updates I made to it. I've been wanting to play with this idea for a bit.

[deleted]

Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

#90

Post by ur-whale »

rany_ wrote: post #65
ur-whale wrote: post #37

Is this the full model or just the weights? [EDIT]: are there checksums available? [EDIT2]: MD5 signatures seem to be included for all models in checklist.chk files next to them And there's also what the author mentions: the magnet file he provides in his README does seed immediately on the download when loaded in a bt app which is usually a good sign that the files are correct.

MD5 signatures don't mean much now that hash collisions could be created instantly on consumer hardware. MD5 is only good for checking for unintentional data corruption.

Bittorrent uses SHA-1

https://en.wikipedia.org/wiki/BitTorrent

Collisions are possible but not exactly trivial

Post reply on HN