I have very limited in this domain.
Why is it necessary to port LLaMa Into C? Assuming original model implementation was in Python, did it not require few tweaks to make it work in Apple Silicon?
Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
51–60 of 298 posts
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#52That's all fine and good. But to do anything useful, you're going to want a powerful GPU (RTX 3090, RTX 4090 or A6000) with as much VRAM as possible. Unlike the diffusion models, LLM's are very memory-intensive, even at 4-bit GPTQ. The larger models like llama-13b and llama-30b run quite well at 4-bit on a 24GB GPU. The llama-65b-4bit should run on a dual 3090/4090 rig. Coupled with the leaked Bing prompt and text-ge…
VRAM is the thing that Apple Silicon is going to have in excess compared to anything even close in price. MacBook Airs can have 14-15GB of VRAM if necessary.
RAM read bandwidth of M1/M2 is still not really competitive with the large GPUs like RTX3090, but it's getting close, compared with the usual CPU setups.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#53Could someone with experience explain: what's the theoretical minimum hardware requirement for llama 7B, 15B, etc, that still provides output on the order of It seems like we can pull some tricks, like using F16, and some kind of quantization, etc. At the end of the day, how much overhead is left that can be reduced? What can I expect to have running on 16gb ram with a 3080 and a midrange AMD processor?
At 4 bits the 13B LLaMa model can run on a 10GB card!
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#54Earlier quoted context omitted.
I've gotten great q&a / conversational results from llama 7b and 13b using https://github.com/oobabooga/text-generation-webui . ], [ "How does the Schrodinger's cat thought experiment relate to the principles of quantum mechanics?", "In this famous paradox, a cat inside a box is simultaneously both alive and dead due to the nature of subatomic particles. This demonstrates that at the smallest scales of reality, we ca…
And 30b: Give me a basic recipe for easy spaghetti carbonara Here is a basic recipe for spaghetti carbonara: Ingredients: 1 lb spaghetti, 3 tbsp butter, 2 tbsp olive oil, 1/4 cup grated Parmesan cheese, 1 egg, 1/2 teaspoon salt, 1/4 teaspoon pepper, 1/2 cup chopped parsley, 1/2 pound bacon, 1 clove garlic, 1/4 cup heavy cream. Directions: Cook spaghetti according to package directions; drain. Heat butter and oil in l…
doesn't say what to do with cream. sprinkling with cheese is a bit wrong. carbonara sauce is more difficult to do than you (or an AI) might think.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#55Don't know anything about ML can someone can explain me what is this hype about?
This is an advanced language model that can now run quickly on consumer grade hardware. You used to need thousands of dollars of GPUs to run a model as sophisticated as this - now it can be done on a laptop,
Wasn't LLaMa official meant to run on consumer grade machine? How does this modify the model to make it work.
All of this is confusing.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#56Isn't using any of the AS "ML" coprocessor/extensions/whatever, so it's just normal simd.
It uses Accelerate so it may be using some of that indirectly.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#57This is sort of the polar opposite of how modern high performance ML frameworks are built. Skimming the code, there's a ton of boilerplate for the various operations that could be library-ized and generified, if that makes sense. I actually really like minimal implementations of state-of-the-art systems because the code is much easier to understand (modern frameworks are super-complex) but I wonder what it means long…
This is just inference. The core at most ML library is the auto differentiation capability. It will be extremely tedious if you are to calculate the gradients manually. Or, if you implemented your own AD, then it is effectively a minified version of a ML library.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#58Could someone with experience explain: what's the theoretical minimum hardware requirement for llama 7B, 15B, etc, that still provides output on the order of It seems like we can pull some tricks, like using F16, and some kind of quantization, etc. At the end of the day, how much overhead is left that can be reduced? What can I expect to have running on 16gb ram with a 3080 and a midrange AMD processor?
The 4-bit GPTQ LLaMA models are the current top-performers. This site has done a lot of the heavy lifting: https://github.com/qwopqwop200/GPTQ-for-LLaMa With 30b-4bit on a RTX 4090, I'm seeing numbers like: Output generated in 4.17 seconds (4.03 tokens/s, 21 tokens) Output generated in 4.38 seconds (4.25 tokens/s, 23 tokens) Output generated in 4.57 seconds (4.25 tokens/s, 24 tokens) Output generated in 3.86 seconds…
And on an M1?
I have 64gb available
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#59Earlier quoted context omitted.
I'm pretty sure he only makes his past twitch streams available to subscribers. It's weird they are available on Youtube, maybe he doesn't know that's something he can change.
I don't think he runs the YouTube channel - it's managed by some of his fans. But like what's he gonna do about it? Send them a takedown notice?
Not sure about his more recent content but he used to have a policy that the VODs could be uploaded in full as long as they were uncut and not modified. If you go watch some older VODs he says so at the beginning.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#60> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook
Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing.
With a lot of fine tuning if you squint this is a useful/convincing "ChatGPT on a laptop" minus the corporate lobotomy only a few short months after release. Very exciting! I actually care about the upcoming Mac Pro now.
$3999 Mac Studio with 64GB ram. +$800 for 128GB.