What are some uses for this?
Create a computer game about a small island with 100 people, with each person being politically aware, with llama2.c being their brain. Then you can simulate politics for a thousand years and see what happens. For instance.
Llama2.c: Inference llama 2 in one file of pure C
51–60 of 173 posts
Re: Llama2.c: Inference llama 2 in one file of pure C
#52Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:
I did use a tweaked nanoGPT to pretrain a 12M model on TinyStories (2Gbytes produced by GPT4), and results are pretty amazing. I've adapted it a bit on Wikipedia then, and it looks like a solid bullshit generator, much smarter than any smoothed n-gram model, and significantly smaller. My bet small LLMs will be predominant in multiple areas. My next goal is to reduce 7B llama2 to 10-100M without making it much dumber.
Re: Llama2.c: Inference llama 2 in one file of pure C
#53As someone who doesn’t work with languages like C, what’s the appeal of “in one file” or “header only”? Is it about dependency management?
Long ago, programmers were conditioned to break long programs and libraries into small translation units ("files") because the compilers were so slow. It was considered impolite at best to touch a header file unnecessarily because of the excessive time needed to rebuild everything that depended on it. When coming up with a new project, you'd spend a fair amount of time thinking about how to make the linker do more of…
It’s still a significant concern for C++, you just can’t get around it because of templates. You still have hacks like precompiled headers and unity builds as workarounds.
Re: Llama2.c: Inference llama 2 in one file of pure C
#54Not that it is necessarily of value, but has anyone got a LLM to run on bare metal?
Some of the smaller ones, yes, the huggingface.co libraries make it pretty simple.
Re: Llama2.c: Inference llama 2 in one file of pure C
#55I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png
Don't use instruct/chat models when the pretrained is available.
Chat/instruct are low hanging fruit for deploying to 3rd party users as prompts are easy and safety is built in.
But they suck compared to the pretrained models for direct usage. Like really, really suck.
Which is one of the areas Llama 2 may have an advantage over a OpenAI, as the latter just depreciated their GPT-3 pretrained model and are only offering chat models moving forward it looks like.
Re: Llama2.c: Inference llama 2 in one file of pure C
#56Earlier quoted context omitted.
I'm not sure what you mean by "used to be", the llama.cpp github repository was committed to just 4 hours ago. This project cites llama.cpp as inspiration, but seems much-simplified. It only supports llama-2, only supports fp-32, and only runs on one CPU thread.
> I'm not sure what you mean by "used to be", the llama.cpp github repository was committed to just 4 hours ago. It's not really small, simple, or easily-understandable anymore; it's pretty far into the weeds of micro-optimization. They're quite good at it, don't get me wrong, but it hurts one's ability to read what exactly is going on, especially with all the options and different configurations that are supported n…
The beauty of a vcs is that all previous versions are still there for everybody to study and enjoy. Including the glorious first commit of llama.cpp
Re: Llama2.c: Inference llama 2 in one file of pure C
#57I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png
Imagine, Casca and Brutus don't stab Caesar. Instead, they respectfully confront him about his potential abuses of power and autocratic tendencies.
Did anyone try this though? Just curious.
Re: Llama2.c: Inference llama 2 in one file of pure C
#58Not that it is necessarily of value, but has anyone got a LLM to run on bare metal?
Some of the smaller ones, yes, the huggingface.co libraries make it pretty simple.
"In computer science, bare machine (or bare metal) refers to a computer executing instructions directly on logic hardware without an intervening operating system."
Re: Llama2.c: Inference llama 2 in one file of pure C
#59I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png
I personally found it to be so "safety filtered" to the point that it's actually done a 180 and can become hateful or perpetuate negative stereotypes in the name of "safety" - see here https://i.imgur.com/xkzXrPK.png and https://i.imgur.com/3HQ8FqL.png
I did have trouble reproducing this consistently except in the Llama2-70b-chat TGI huggingface only when it's sent as the second message, so maybe there's something wonky going on with the prompting style there that causes this behavior. I haven't been able to get the model running myself for further investigation yet.
Re: Llama2.c: Inference llama 2 in one file of pure C
#60Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:
I did use a tweaked nanoGPT to pretrain a 12M model on TinyStories (2Gbytes produced by GPT4), and results are pretty amazing. I've adapted it a bit on Wikipedia then, and it looks like a solid bullshit generator, much smarter than any smoothed n-gram model, and significantly smaller. My bet small LLMs will be predominant in multiple areas. My next goal is to reduce 7B llama2 to 10-100M without making it much dumber.
>My next goal is to reduce 7B llama2 to 10-100M without making it much dumber.
That is going to be hard as the 7B model was trained on 2T tokens. Maybe if you heavily restrict the range in which the model should operate.