I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png
Llama2.c: Inference llama 2 in one file of pure C
41–50 of 173 posts
Re: Llama2.c: Inference llama 2 in one file of pure C
#42Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:
Your work is an inspiration as always!! My n00b question is: what do you think is currently the most practical path to running a reasonably-sized (doesn't have to be the biggest) LLM on a commodity linux server for hooking up to a hobby web app ... i.e., one without a fancy GPU. (Renting instances with GPUs on, say, Linode, is significantly more expensive than standard servers that host web apps.) Is this totally out…
It might be more expensive to get a GPU instance but at a guess I'd say it's more cost-effective considering that the CPU computation will be less efficient and take much longer. I bet someone's done this out with real numbers, I just haven't seen it.
Re: Llama2.c: Inference llama 2 in one file of pure C
#43I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png
It's Llama-2 chat that is too much filtered, not "llama-2"
Re: Llama2.c: Inference llama 2 in one file of pure C
#44I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png
[dead]
Re: Llama2.c: Inference llama 2 in one file of pure C
#45I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png
Imagine, Casca and Brutus don't stab Caesar. Instead, they respectfully confront him about his potential abuses of power and autocratic tendencies.
Re: Llama2.c: Inference llama 2 in one file of pure C
#46Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:
Your work is an inspiration as always!! My n00b question is: what do you think is currently the most practical path to running a reasonably-sized (doesn't have to be the biggest) LLM on a commodity linux server for hooking up to a hobby web app ... i.e., one without a fancy GPU. (Renting instances with GPUs on, say, Linode, is significantly more expensive than standard servers that host web apps.) Is this totally out…
I would use textsynth (https://bellard.org/ts_server/) or llama.cpp (https://github.com/ggerganov/llama.cpp) if you're running on CPU.
- I wouldn't use anything higher than a 7B model if you want decent speed.
- Quantize to 4-bit to save RAM and run inference faster.
Re: Llama2.c: Inference llama 2 in one file of pure C
#47This is amazing. One curious question: Why C? Why not standard C++?
Re: Llama2.c: Inference llama 2 in one file of pure C
#48Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:
Great job, thanks! Do you have any early impressions on the relative quality/performance of small lama-2 models vs the small gpt-2 models?
Re: Llama2.c: Inference llama 2 in one file of pure C
#49This is amazing. One curious question: Why C? Why not standard C++?
That project already exists https://github.com/ggerganov/llama.cpp
Re: Llama2.c: Inference llama 2 in one file of pure C
#50To run a neural network, how much memory does one need? Is it enought to load the first two layers from disk, calculate the activations for all nodes, discard the first layer, load the third layer from disk, calculate all the activations for all nodes, discard the second layer etc? Then memory needs to be big enough to hold to 2 layers?
This bloke on huggingface documents the memory requirements for his quantized versions of popular models: https://huggingface.co/TheBloke
Tl;Dr, Max ram needed depends on quant method, rough ranges are:
7B models are in the 4-8GB range
13B models 8-15GB
30B models 13-33GB
70B models 31-75GB