Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

Post by [deleted] » Tue, Mar 14, 2023, 1:10 AM UTC

MacsHeadroom wrote: post #28

v64 wrote: post #18

Earlier quoted context omitted.

The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWea…

I'm running LLaMA-65B on a single A100 80GB with 8bit quantization. $1.5/hr on vast.ai

Post by [deleted] » Thu, Mar 16, 2023, 9:39 PM UTC

MacsHeadroom wrote: post #28

v64 wrote: post #18

Earlier quoted context omitted.

The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWea…

I'm running LLaMA-65B on a single A100 80GB with 8bit quantization. $1.5/hr on vast.ai