What a misleading article. I thought they'd done some breakthrough in resource efficiency. This is just the old and slow method tools like Ollama used.
What method is that? Layer offloading?
Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
21–30 of 32 posts
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#22Abysmal article. It doesn't explain anything about the claim in the title. Is there quantization? How much RAM do you need? How fast is the inference? None of these questions are addressed or even mentioned.
> Of course, it would be more reasonable to compare the similarly sized 400B models with GPT4 and Claude3 Opus
No. It's completely irrelevant to the topic of the article.
The article is mostly a press release for llama 3. It also contains a few comments by the author, they aren't bad but don't save the clickbaity, buzzy, sensationalist core.
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#23Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#24What a misleading article. I thought they'd done some breakthrough in resource efficiency. This is just the old and slow method tools like Ollama used.
Do you know how much disk space this takes total? When I ran it, it downloaded nearly 30 gigabytes of models and seemed to be on track to download 28 more 5 gigabyte chunks (for a total of 150 gigabytes of disk space or maybe more). What is the total size before it finishes?
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#25Any sense of speed? My assumption is that shuttling the weights in/out of the GPU is slow. Does GPU load + processing beat an entirely CPU solution? Doubly so if it is a huge model where the model cannot sit fully in RAM?
Depends on your CPU. I once tried 70b llama on 256 thread Epyc, it was around 1/10 of A100 (80GB) speed.
how much disk space did it use?
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#26This sounds like a game changer. I wonder if they need to do a tonne of specific work per model? If this could be implemented in Ollama, I'd be over the moon.
llama3:70b using llama.cpp (used under the hood by Ollama) on a 11th Gen Intel i5-11400 @ 2.60GHz - no GPU, CPU inference only. "Write a haiku about Hacker News mentioning AI in the title" Here is a haiku: AI whispers secrets HN threads weave tangled debate Intelligence born eval time = 30363.04 ms / 23 runs ( 1320.13 ms per token, 0.76 tokens per second) total time = 34294.80 ms / 33 tokens
how much disk space did it use?
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#27What a misleading article. I thought they'd done some breakthrough in resource efficiency. This is just the old and slow method tools like Ollama used.
What method is that? Layer offloading?
Yes, it's either that, or CPU inference. The article doesn't say.
It doesn't mention quantization either.
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#28Earlier quoted context omitted.
Depends on your CPU. I once tried 70b llama on 256 thread Epyc, it was around 1/10 of A100 (80GB) speed.
how much disk space did it use?
I didn’t check, but iirc it was an fp16 model checkpoint which we converted to int8 for inference, so I assume 140GB?
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#29As a cloud solution developer that has to build AI on Azure I have been using this instead of Azure OpenAI. It has sped up my development workflow a lot, and for my purposes it’s comparable enough. I’m using LM studio to load these models.
Can you expand a bit -- because the AOAI is so slow? What exactly helps you speed things up?
On my machine, I am able to create a prompt that suits my need and chat with the model in realtime. With 100% GPU offload, it replies within half a second. LM studio provides an OpenAI compatible api endpoint for my Dotnet software to use. This boosts my developer experience significantly. The Azure services are slow and if you want to regenerate a serie of responses (e.g part of conversation flow) it just takes too long. On my local machine I also do not worry about cloud costs.
As a bonus; I also use this for a personal project where I use prompts and Llama3 to control smart devices. JSON responses from the LLM are parsed and translated into the smart device commands from a raspberry pi. I control it using speech via my Apple Watch and Apple shortcuts to the raspberry pi’s api. It all works magically and fast. Way faster than pulling up the app on my phone. And yes the LLM is smart enough to control groups of devices using simple conversational AI.
edit; here's a demo https://www.youtube.com/watch?v=dCN1AnX8txM
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#30What a misleading article. I thought they'd done some breakthrough in resource efficiency. This is just the old and slow method tools like Ollama used.
Do you know how much disk space this takes total? When I ran it, it downloaded nearly 30 gigabytes of models and seemed to be on track to download 28 more 5 gigabyte chunks (for a total of 150 gigabytes of disk space or maybe more). What is the total size before it finishes?
70B parameters * 2 bytes each (fp16 or bf16) = 140GB
I wish models sizes were published in bytes.