Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU

ai.gopubby.com

Post reply on HN
Print view View original

11–20 of 32 posts

Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU

#11

Post by programd »

1GZ0 wrote: post #2

This sounds like a game changer. I wonder if they need to do a tonne of specific work per model? If this could be implemented in Ollama, I'd be over the moon.

llama3:70b using llama.cpp (used under the hood by Ollama) on a 11th Gen Intel i5-11400 @ 2.60GHz - no GPU, CPU inference only.

"Write a haiku about Hacker News mentioning AI in the title"

Here is a haiku:

  AI whispers secrets
  HN threads weave tangled debate
  Intelligence born

  eval time = 30363.04 ms / 23 runs ( 1320.13 ms per token, 0.76 tokens per second)
  total time = 34294.80 ms / 33 tokens

Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU

#12

Post by isoprophlex »

fexelein wrote: post #7

As a cloud solution developer that has to build AI on Azure I have been using this instead of Azure OpenAI. It has sped up my development workflow a lot, and for my purposes it’s comparable enough. I’m using LM studio to load these models.

Can you expand a bit -- because the AOAI is so slow? What exactly helps you speed things up?

Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU

#13

Post by cpill »

nutrientharvest wrote: post #5
1GZ0 wrote: post #2

This sounds like a game changer. I wonder if they need to do a tonne of specific work per model? If this could be implemented in Ollama, I'd be over the moon.

Ollama can already run Llama-3 70B with a 4GB GPU, or no GPU at all, it'll just be slow. Considering this says it's "not designed for real-time interactive scenarios" it's probably also really slow

so how much GPU RAM does need to get the 70B going fast (ish)?

Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU

#15

Post by AaronFriel »

cpill wrote: post #13
nutrientharvest wrote: post #5

Earlier quoted context omitted.

Ollama can already run Llama-3 70B with a 4GB GPU, or no GPU at all, it'll just be slow. Considering this says it's "not designed for real-time interactive scenarios" it's probably also really slow

so how much GPU RAM does need to get the 70B going fast (ish)?

A good rule of thumb is that models can be quantized to 6 to 8 bits per weight without significantly degrading quality. This is convenient for the math: 70GB plus some overhead for the attention matrices (ongoing requests). This overhead depends on workload and context lengths, but you should expect about 30% more. So, around 100GB for a server under load.

Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU

#17

Post by irusensei »

999900000999 wrote: post #4

Any chance that the new NPUs are going to significantly speed up running these locally. Well I'm definitely worried about recall and all the Microsoft nonsense, I really want to be able to run and train LMMs, and other machine learning frameworks locally.

You still need lots of fast memory.

Re: Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU

#20

Post by bityard »

programd wrote: post #11
1GZ0 wrote: post #2

This sounds like a game changer. I wonder if they need to do a tonne of specific work per model? If this could be implemented in Ollama, I'd be over the moon.

llama3:70b using llama.cpp (used under the hood by Ollama) on a 11th Gen Intel i5-11400 @ 2.60GHz - no GPU, CPU inference only. "Write a haiku about Hacker News mentioning AI in the title" Here is a haiku: AI whispers secrets HN threads weave tangled debate Intelligence born eval time = 30363.04 ms / 23 runs ( 1320.13 ms per token, 0.76 tokens per second) total time = 34294.80 ms / 33 tokens

That really doesn't seem bad. When people talk about responses of self-hosted LLMs without a beefy GPU being unusably slow, I always assumed they meant 15 minutes to hours. I do not mind waiting a few minutes if it will summarize the answer a question that will take me many times longer to research.
Post reply on HN