Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
1–10 of 32 posts
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#2This sounds like a game changer. I wonder if they need to do a tonne of specific work per model?
If this could be implemented in Ollama, I'd be over the moon.
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#3From the article: Please note: it’s not designed for real-time interactive scenarios like chatting, more suitable for data processing and other offline asynchronous scenarios.
Repo: https://github.com/lyogavin/Anima/tree/main/air_llm
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#4Any chance that the new NPUs are going to significantly speed up running these locally.
Well I'm definitely worried about recall and all the Microsoft nonsense, I really want to be able to run and train LMMs, and other machine learning frameworks locally.
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#5This sounds like a game changer. I wonder if they need to do a tonne of specific work per model? If this could be implemented in Ollama, I'd be over the moon.
Ollama can already run Llama-3 70B with a 4GB GPU, or no GPU at all, it'll just be slow.
Considering this says it's "not designed for real-time interactive scenarios" it's probably also really slow
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#6Any sense of speed? My assumption is that shuttling the weights in/out of the GPU is slow. Does GPU load + processing beat an entirely CPU solution? Doubly so if it is a huge model where the model cannot sit fully in RAM?
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#7As a cloud solution developer that has to build AI on Azure I have been using this instead of Azure OpenAI. It has sped up my development workflow a lot, and for my purposes it’s comparable enough. I’m using LM studio to load these models.
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#8This is probably going to sound silly, but I wonder how it compares to TinyLlama and others.
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#9Any sense of speed? My assumption is that shuttling the weights in/out of the GPU is slow. Does GPU load + processing beat an entirely CPU solution? Doubly so if it is a huge model where the model cannot sit fully in RAM?
Depends on your CPU. I once tried 70b llama on 256 thread Epyc, it was around 1/10 of A100 (80GB) speed.
Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU
#10Llama isn’t open source because the license says you can only use it to improve itself, so the title is false