Getting 220 tokens/sec with -Ofast on an 2018 iMac Pro.
Llama2.c: Inference llama 2 in one file of pure C
91–100 of 173 posts
Re: Llama2.c: Inference llama 2 in one file of pure C
#92This is amazing. One curious question: Why C? Why not standard C++?
Why call C++ standard but not C?
Re: Llama2.c: Inference llama 2 in one file of pure C
#93Is this for educational purposes only? Based on the success of llama.cpp and this one it appears that the industry is going in a direction of separate source code for every model that is released instead of general purpose frameworks like pytorch/tensorflow/onnxruntime?
Yes, this appears to be entirely educational. No. Despite the name, llama.cpp supports more than just llama. It also isn’t an entirely bespoke thing as you indicate, since it is built on the more general purpose “ggml” tensor library/framework.
I am very confused; so llama.cpp supports other non-llama models.. but is also based on the general-purpose ggml library?
so llama.cpp is actually 'generic LLM framework' while ggml is 'generic ML framework'?
Re: Llama2.c: Inference llama 2 in one file of pure C
#94Earlier quoted context omitted.
Yes, this appears to be entirely educational. No. Despite the name, llama.cpp supports more than just llama. It also isn’t an entirely bespoke thing as you indicate, since it is built on the more general purpose “ggml” tensor library/framework.
I am very confused; so llama.cpp supports other non-llama models.. but is also based on the general-purpose ggml library? so llama.cpp is actually 'generic LLM framework' while ggml is 'generic ML framework'?
> so llama.cpp is actually 'generic LLM framework' while ggml is 'generic ML framework'?
That seems like a reasonable description to me, but I’m not an expert, just someone who is interested in this stuff.
Re: Llama2.c: Inference llama 2 in one file of pure C
#95Very dumb question from someone not steeped in the world of latest LLM developments... does the C code have to invoke python every time you pass it a prompt? What kind of permissions does it need?
Re: Llama2.c: Inference llama 2 in one file of pure C
#96As someone who doesn’t work with languages like C, what’s the appeal of “in one file” or “header only”? Is it about dependency management?
Pretty much. It's one 500ish line file that's super easy to parse. 50ish lines is declaring the data structs, 100ish lines is some boilerplate for allocating and deallocating those structs. There are also no dependencies (which should tell you something when remembering that C is not a batteries included language).
Re: Llama2.c: Inference llama 2 in one file of pure C
#97Never seen the word “inference” used as a verb.
Re: Llama2.c: Inference llama 2 in one file of pure C
#98Earlier quoted context omitted.
Prompt ingestion is too slow on the Oracle VMs. Also its really tricky to even build llama.cpp with a BLAS library, to make prompt ingestion less slow. The Oracle Linux OpenBLAS build isnt detected ootb, and it doesn't perform well compared to x86 for some reason. LLVM/GCC have some kind of issue identifying the Ampere ARM architecture (march=native doesn't really work), so maybe this could be improved with the right…
Not sure if that's still the case. I remember having trouble building it a couple of months ago, had to tweak the Makefile because iirc it assumed ARM64 Mac, but I recently re-cloned the repo and started from scratch and it was as simple as `make DLLAMA_BLAS=1`. I don't think I have any special setup other than having installed the apt openblas dev package.
IDK. A bunch of basic development packages like git were missing from my Ubuntu image when I tried last week, and I just gave up because it seemed like a big rabbit hole to go down.
I can see the ARM64 versions on the Ubuntu web package list, so... IDK what was going on?
On Oracle Linux, until I changed some env variables and lines in the makefile, the openblas build would "work," but it was actually silently failing and not using OpenBLAS.
Re: Llama2.c: Inference llama 2 in one file of pure C
#99Getting 220 tokens/sec with -Ofast on an 2018 iMac Pro.
CPU only?
Re: Llama2.c: Inference llama 2 in one file of pure C
#100Very dumb question from someone not steeped in the world of latest LLM developments... does the C code have to invoke python every time you pass it a prompt? What kind of permissions does it need?
Currently the C code does not invoke Python and there is no way to pass a prompt. It does not need any special permissions.