Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

81–90 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#81

Post by adeon »

If you are interested in implementing LLaMA yourself or learning, I noticed that the reference code by Facebook is one of the cleaner, easier to read ML code I've seen in a while. https://github.com/facebookresearch/llama/blob/main/llama/mo... It's about 200 lines long. You probably do need a bit of knowledge to understand what you are reading but I was pleasantly surprised.

For example in comparison, StableDiffusion torch code in diffusers and transformers Python libraries has lots of conditionals, experiments etc. that are not being used that can make it hard to follow what is going on.

Last weekend I got the "main loop" of the transformer working in pure CPU Rust code, following the reference code. My crappy code is just very very slow as I focused on getting it to run, not making it fast. The tokenizer uses some Google thing https://github.com/google/sentencepiece but luckily for inference it seems that you just need to be able to parse the tokenizer model file and not understand how it was created; I was able to strip out the protobuf files from that repository and add it to Rust and read the tokens.

I am optimistic that someone makes a high quality CPU or some CPU+GPU+SSD combination thingmaling that will make it somewhat practical to run even the large LLM models without needing an A100 or two.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#82

Post by codetrotter »

bigstumpy wrote: post #76
AlecSchueler wrote: post #64

Earlier quoted context omitted.

What does almost mean in this case?

There is also C++

Iow it probably wouldn’t compile with an actual C only compiler, but by and large it looks more like C than like C++?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#83

Post by eigenvalue »

This is so awesome and exciting. I have an M1 iMac and it was trivially easy to get this working and generating text. And the performance is VERY impressive, especially considering that it's not even using any of the built in "neural compute" stuff. Also, the model seems like it doesn't have any political correctness conditioning based on some of the completions it has given me on controversial prompts. I can't wait until someone gets the 13b model working (sounds like this should happen in the next day or so) and gets the repetition penalty working.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#84

Post by zabzonk »

zabzonk wrote: post #54
lxe wrote: post #46

Earlier quoted context omitted.

And 30b: Give me a basic recipe for easy spaghetti carbonara Here is a basic recipe for spaghetti carbonara: Ingredients: 1 lb spaghetti, 3 tbsp butter, 2 tbsp olive oil, 1/4 cup grated Parmesan cheese, 1 egg, 1/2 teaspoon salt, 1/4 teaspoon pepper, 1/2 cup chopped parsley, 1/2 pound bacon, 1 clove garlic, 1/4 cup heavy cream. Directions: Cook spaghetti according to package directions; drain. Heat butter and oil in l…

doesn't say what to do with cream. sprinkling with cheese is a bit wrong. carbonara sauce is more difficult to do than you (or an AI) might think.

well, does it say what to do with cream?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#86

Post by recuter »

PragmaticPulp wrote: post #78
alphanullmeric wrote: post #73

Earlier quoted context omitted.

Wonder why AMD/Intel/Nvidia haven’t invented some sort of device that allows the processor and graphics to share memory like Apple has done.

PCIe devices like GPUs can access system memory. Integrated GPUs also access system memory via the same bus as the CPU. It’s not really a new technique. Apple just shipped a highly integrated unit with large memory bandwidth.

That's a pretty important just. They chose to go down this path at this time and shipped, now I can run the 64B Llama on a widely available $3,999 prosumer device.

How much would a PC that can do that currently cost me and can I have it by tomorrow?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#87

Post by 0xbadc0de5 »

liuliu wrote: post #61
0xbadc0de5 wrote: post #24

That's all fine and good. But to do anything useful, you're going to want a powerful GPU (RTX 3090, RTX 4090 or A6000) with as much VRAM as possible. Unlike the diffusion models, LLM's are very memory-intensive, even at 4-bit GPTQ. The larger models like llama-13b and llama-30b run quite well at 4-bit on a 24GB GPU. The llama-65b-4bit should run on a dual 3090/4090 rig. Coupled with the leaked Bing prompt and text-ge…

Here is some quick math: these devices has SSD read speed somewhere around 2GiB/s. With 4-bit quantization, we are looking at loading 4B parameters per second. That means we need 8s per token for 30B model. Hmmm, the math is a bit off (or I need to look closer whether we can do more tokens per iteration with some batching).

I think your math is missing some details.

An RTX 4090 has a memory bandwidth of 1,008 GB/s.

PCIe 4 x16 has a 32GB/s bandwidth.

DDR4 RAM has 3200 MT/s transfer rate.

An AMD 5900x can easily max that out.

A good NVME can hit 7 GB/s read speeds or better.

And the 4-bit CUDA kernels can pack 16x 4-bit ints into a single 64-bit transfer / register.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#88

Post by Nevermark »

recuter wrote: post #60

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…

Given that the RAM is directly accessible by CPU, GPU and DPU/Neural cores, it really is premium RAM.

Apple's visionary hardware team has finally caught up with Apple's visionary high RAM prices! :)

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#89

Post by lynguist »

eigenvalue wrote: post #83

This is so awesome and exciting. I have an M1 iMac and it was trivially easy to get this working and generating text. And the performance is VERY impressive, especially considering that it's not even using any of the built in "neural compute" stuff. Also, the model seems like it doesn't have any political correctness conditioning based on some of the completions it has given me on controversial prompts. I can't wait…

It is using the built-in neural accelerators, that’s why it’s fast, that’s why it’s only supported on Macs so far. The code makes use of official Apple APIs which delegate the necessary BLAS calls to the available hardware.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#90

Post by nerdponx »

adeon wrote: post #81

If you are interested in implementing LLaMA yourself or learning, I noticed that the reference code by Facebook is one of the cleaner, easier to read ML code I've seen in a while. https://github.com/facebookresearch/llama/blob/main/llama/mo... It's about 200 lines long. You probably do need a bit of knowledge to understand what you are reading but I was pleasantly surprised. For example in comparison, StableDiffusion…

I noticed the Fasttext code was also surprisingly clean and readable C++. whatever moralities and other flaws the metal business model might have in general, they seem to have a consistently excellent track record when it comes to publicly available libraries and tools.
Post reply on HN