Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

61–70 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#61

Post by liuliu »

0xbadc0de5 wrote: post #24

That's all fine and good. But to do anything useful, you're going to want a powerful GPU (RTX 3090, RTX 4090 or A6000) with as much VRAM as possible. Unlike the diffusion models, LLM's are very memory-intensive, even at 4-bit GPTQ. The larger models like llama-13b and llama-30b run quite well at 4-bit on a 24GB GPU. The llama-65b-4bit should run on a dual 3090/4090 rig. Coupled with the leaked Bing prompt and text-ge…

Here is some quick math: these devices has SSD read speed somewhere around 2GiB/s. With 4-bit quantization, we are looking at loading 4B parameters per second. That means we need 8s per token for 30B model. Hmmm, the math is a bit off (or I need to look closer whether we can do more tokens per iteration with some batching).

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#62

Post by fdb »

TaylorAlexander wrote: post #26
2bitencryption wrote: post #25

Could someone with experience explain: what's the theoretical minimum hardware requirement for llama 7B, 15B, etc, that still provides output on the order of It seems like we can pull some tricks, like using F16, and some kind of quantization, etc. At the end of the day, how much overhead is left that can be reduced? What can I expect to have running on 16gb ram with a 3080 and a midrange AMD processor?

Well I was able to run the original code with the 7B model on 16GB vram: https://news.ycombinator.com/item?id=35013604 The output I got was underwhelming, though I did not attempt any tuning.

The author just made an update that makes the generation much better, even with the 7B model:

https://twitter.com/ggerganov/status/1634310199170179075

I tried it out myself (git pull && make) and the difference in results are day and night! It's amazing to play with, although you should prompt it differently than ChatGPT (more like the GPT-3 API).

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#63

Post by BoberMod »

xiphias2 wrote: post #5

George Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch: https://github.com/geohot/tinygrad/tree/llama The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.

There is also a gpu-acelerated fork of the original repo

https://github.com/remixer-dec/llama-mps

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#64

Post by AlecSchueler »

attractivechaos wrote: post #15
xiphias2 wrote: post #5

George Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch: https://github.com/geohot/tinygrad/tree/llama The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.

Both are very impressive. A nice thing about Gerganov's implementation is that it is written in almost pure C. Arguably easier for deployment.

What does almost mean in this case?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#65

Post by version_five »

SCLeo wrote: post #57
dekhn wrote: post #36

This is sort of the polar opposite of how modern high performance ML frameworks are built. Skimming the code, there's a ton of boilerplate for the various operations that could be library-ized and generified, if that makes sense. I actually really like minimal implementations of state-of-the-art systems because the code is much easier to understand (modern frameworks are super-complex) but I wonder what it means long…

This is just inference. The core at most ML library is the auto differentiation capability. It will be extremely tedious if you are to calculate the gradients manually. Or, if you implemented your own AD, then it is effectively a minified version of a ML library.

  llama.cpp/ggml.h

  // GGML Tensor Library

  ...

  // This library implements:
  //  - a set of tensor operations
  //  - automatic differentiation
  //  - basic optimization algorithms

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#66

Post by kiratp »

0xbadc0de5 wrote: post #24

That's all fine and good. But to do anything useful, you're going to want a powerful GPU (RTX 3090, RTX 4090 or A6000) with as much VRAM as possible. Unlike the diffusion models, LLM's are very memory-intensive, even at 4-bit GPTQ. The larger models like llama-13b and llama-30b run quite well at 4-bit on a 24GB GPU. The llama-65b-4bit should run on a dual 3090/4090 rig. Coupled with the leaked Bing prompt and text-ge…

Apple Silicon uses unified memory so laptops have up to 64GB of VRAM and the Max sitio can have up to 128 GB.

That makes them uniquely “powerful” for inference with large models.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#67

Post by londons_explore »

recuter wrote: post #60

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…

LLaMA doesn't perform very well with answering questions.

If you ask it "What color is the sky?" It will reply with something like "Why is ice cold? Why do we exist?"

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#68

Post by ojosilva »

Interesting. But how about the Apple Neural Engine (ANE)? I've always wondered if the ANE is ML worthy, maybe it's really only with inference or who knows, even training somehow. I've seen Apple's marketeers bragging about it [1], with even code examples, but ifaik no useful libraries nor reliable measurements and community interest exist in the wild for doing ANE ML on Macs.

1. https://machinelearning.apple.com/research/neural-engine-tra...

Edit: just found this: https://github.com/ggerganov/whisper.cpp/pull/566

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#69

Post by lostmsu »

xiphias2 wrote: post #5

George Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch: https://github.com/geohot/tinygrad/tree/llama The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.

It's not tinygrad really, it is PyTorch.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#70

Post by RoboTeddy »

londons_explore wrote: post #67
recuter wrote: post #60

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…

LLaMA doesn't perform very well with answering questions. If you ask it "What color is the sky?" It will reply with something like "Why is ice cold? Why do we exist?"

LLaMA isn't built on RLHF, so it may be necessary to create a more extensive prompt. For example:

```

You are a super intelligent honest question-answering system.

Q: What's 2+2?

A: 4

Q: What color is the sky?

A:

```

Post reply on HN