Live data from Hacker News

Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU

ai.gopubby.com

31–32 of 32 posts

Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU

#31
post #30

Earlier quoted context omitted.

Do you know how much disk space this takes total? When I ran it, it downloaded nearly 30 gigabytes of models and seemed to be on track to download 28 more 5 gigabyte chunks (for a total of 150 gigabytes of disk space or maybe more). What is the total size before it finishes?

70B parameters * 2 bytes each (fp16 or bf16) = 140GB I wish models sizes were published in bytes.

Thanks, I finished downloading it (which took many hours) onto an external hard drive (by adding a HF_HOME environmental variable for where to store that cache). Its size was 262 GB.

Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU

#32
post #28

Earlier quoted context omitted.

how much disk space did it use?

I didn’t check, but iirc it was an fp16 model checkpoint which we converted to int8 for inference, so I assume 140GB?

Thanks, I finished downloading it (which took many hours) onto an external hard drive (by adding a HF_HOME environmental variable for where to store that cache). Its size was 262 GB.
Post reply on HN