Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

21–30 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#21

Post by eis »

mrtksn wrote: post #2

The performance on Macbook with M1 Pro is said to be 20 tokens/s https://twitter.com/ggerganov/status/1634282694208114690

A Macbook Pro M1 would have the base M1 CPU while he was referring to the M1 Pro CPU in something like a Macbook Pro w/ M1 Pro. It's confusing naming by Apple.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#22

Post by mrtksn »

eis wrote: post #21
mrtksn wrote: post #2

The performance on Macbook with M1 Pro is said to be 20 tokens/s https://twitter.com/ggerganov/status/1634282694208114690

A Macbook Pro M1 would have the base M1 CPU while he was referring to the M1 Pro CPU in something like a Macbook Pro w/ M1 Pro. It's confusing naming by Apple.

right, fixed it

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#23

Post by Mistletoe »

vopi wrote: post #13
apetresc wrote: post #9

Earlier quoted context omitted.

Can you link to the Twitch stream VoD?

https://www.twitch.tv/georgehotz/videos?filter=archives&sort... although, there is a VOD channel on YT that might be better.

This is such a refreshing and neat way to use Twitch.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#24

Post by 0xbadc0de5 »

That's all fine and good. But to do anything useful, you're going to want a powerful GPU (RTX 3090, RTX 4090 or A6000) with as much VRAM as possible. Unlike the diffusion models, LLM's are very memory-intensive, even at 4-bit GPTQ. The larger models like llama-13b and llama-30b run quite well at 4-bit on a 24GB GPU. The llama-65b-4bit should run on a dual 3090/4090 rig.

Coupled with the leaked Bing prompt and text-generation-webui, the results are quite impressive.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#25

Post by 2bitencryption »

Could someone with experience explain: what's the theoretical minimum hardware requirement for llama 7B, 15B, etc, that still provides output on the order of It seems like we can pull some tricks, like using F16, and some kind of quantization, etc.

At the end of the day, how much overhead is left that can be reduced? What can I expect to have running on 16gb ram with a 3080 and a midrange AMD processor?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#26

Post by TaylorAlexander »

2bitencryption wrote: post #25

Could someone with experience explain: what's the theoretical minimum hardware requirement for llama 7B, 15B, etc, that still provides output on the order of It seems like we can pull some tricks, like using F16, and some kind of quantization, etc. At the end of the day, how much overhead is left that can be reduced? What can I expect to have running on 16gb ram with a 3080 and a midrange AMD processor?

Well I was able to run the original code with the 7B model on 16GB vram: https://news.ycombinator.com/item?id=35013604

The output I got was underwhelming, though I did not attempt any tuning.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#28

Post by thewataccount »

I don't have the hardware to run the 60B model to test this at the moment -

How does it perform with programming, for example making a basic python script to scrape a website, or a bash script, etc?

I've managed to run the 13B* at 8bit with decent performance on a 4090 - but it's only 24GB of VMRAM so I've been struggling to run the 30B at anything more then a snails pace.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#29

Post by arthurcolle »

thewataccount wrote: post #28

I don't have the hardware to run the 60B model to test this at the moment - How does it perform with programming, for example making a basic python script to scrape a website, or a bash script, etc? I've managed to run the 13B* at 8bit with decent performance on a 4090 - but it's only 24GB of VMRAM so I've been struggling to run the 30B at anything more then a snails pace.

you mean the 13B ?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#30

Post by 0xbadc0de5 »

thewataccount wrote: post #28

I don't have the hardware to run the 60B model to test this at the moment - How does it perform with programming, for example making a basic python script to scrape a website, or a bash script, etc? I've managed to run the 13B* at 8bit with decent performance on a 4090 - but it's only 24GB of VMRAM so I've been struggling to run the 30B at anything more then a snails pace.

The 13b and 30b run quite well on a 4090 at 4-bit quantization.
Post reply on HN