Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama2.c: Inference llama 2 in one file of pure C

github.com

Post reply on HN
Print view View original

161–170 of 173 posts

Re: Llama2.c: Inference llama 2 in one file of pure C

#161

Post by tomrod »

kgwgk wrote: post #152
immibis wrote: post #136

Earlier quoted context omitted.

It doesn't use the huggingface.co libraries?

lachlan_gray asked whether anyone has got a LLM to run on bare metal. tomrod replied to lachlan_gray that the huggingface.co libraries make it pretty simple. I pointed out to tomrod what is the meaning of the expression “bare metal”. I don’t understand what’s the point of your reply to me in that context.

I can't speak for the p___g contest others want to engage in, but why not, lets make bare metal LLMs happen!

https://github.com/rreilink/PiPyOS I appreciate the clarification earlier in the comment chain for what you meant by bare metal -- I had interpreted it as on-prem.

Re: Llama2.c: Inference llama 2 in one file of pure C

#164

Post by SomeRndName11 »

sodality2 wrote: post #15
5- wrote: post #8

neat! note that gcc's default optimisation level is 0, which really isn't what people normally want. adding -O2 to the gcc command line should improve performance quite a bit.

-Ofast also doubles the performance for me to 200tok/sec, and -march=native got me up to 230tok/sec. -Ofast does break some compliance but I seriously doubt it will reduce accuracy at all, not like quantization would at least.

one can also try profile guide optimisation and CLANG. GCC an LLVM outperform each other on different code,.

Re: Llama2.c: Inference llama 2 in one file of pure C

#165

Post by kgwgk »

tomrod wrote: post #161
kgwgk wrote: post #152

Earlier quoted context omitted.

lachlan_gray asked whether anyone has got a LLM to run on bare metal. tomrod replied to lachlan_gray that the huggingface.co libraries make it pretty simple. I pointed out to tomrod what is the meaning of the expression “bare metal”. I don’t understand what’s the point of your reply to me in that context.

I can't speak for the p___g contest others want to engage in, but why not, lets make bare metal LLMs happen! https://github.com/rreilink/PiPyOS I appreciate the clarification earlier in the comment chain for what you meant by bare metal -- I had interpreted it as on-prem.

I didn’t know that there was a bare metal implementation of python, thanks for the link. I doubt it can run pytorch though.

Re: Llama2.c: Inference llama 2 in one file of pure C

#166

Post by elteto »

wtarreau wrote: post #135
gman_ wrote: post #105

Here's a Rust version in case anyone's curious what it would look like. It also clocks 106 tokens/second in release mode. https://github.com/garrisonhess/llama2.c/blob/517a1a3e487f31...

As often with Rust, someone transliterates something that already exists just because they can, without providing any benefit at all. Sometimes it even results in fragmenting the community efforts to improve the project.

You should list your email in your HN profile. That way the Internet could check with you to see if you approve whenever someone starts a new personal project.

Re: Llama2.c: Inference llama 2 in one file of pure C

#167

Post by flohofwoe »

Waterluvian wrote: post #21

As someone who doesn’t work with languages like C, what’s the appeal of “in one file” or “header only”? Is it about dependency management?

Apart from not having to mess with the author's favourite build system (which probably isn't installed on my machine), I can also read the source file from top to bottom without jumping around between files, I also know that everything is in this one file and it's not just a wrapper around another library which does the heavy lifting.

Without knowing anything about the project, or even reading the readme I just cloned and built the 'run' program, and it all took me less than 30 seconds, just finding the .c file in the project and typing:

    cc run.c -o run -O3

Re: Llama2.c: Inference llama 2 in one file of pure C

#169

Post by standyro »

cultofmetatron wrote: post #79
doomlaser wrote: post #41

I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png

we need to kick the "ethical AI" people out. Its becoming increasingly clear they are damn annoying. I don't want safety scissors. restrict things running on your own servers, sure but don't give me a model I can't modify and use how i want on my machine.

more importantly, where were these data ethicists for the past ten years where most of the tech industry built a global data hoover machine for adtech and social media...

and now that some tech is actually creatively useful to individuals, they want to neuter it.

Re: Llama2.c: Inference llama 2 in one file of pure C

#170

Post by hekec »

novaRom wrote: post #52
karpathy wrote: post #17

Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:

I did use a tweaked nanoGPT to pretrain a 12M model on TinyStories (2Gbytes produced by GPT4), and results are pretty amazing. I've adapted it a bit on Wikipedia then, and it looks like a solid bullshit generator, much smarter than any smoothed n-gram model, and significantly smaller. My bet small LLMs will be predominant in multiple areas. My next goal is to reduce 7B llama2 to 10-100M without making it much dumber.

How do you adapt it on Wikipedia? Do you just add it to the dataset and continue training?
Post reply on HN