Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama2.c: Inference llama 2 in one file of pure C

github.com

Post reply on HN
Print view View original

151–160 of 173 posts

Re: Llama2.c: Inference llama 2 in one file of pure C

#151

Post by gman_ »

steeve wrote: post #150
gman_ wrote: post #148

Earlier quoted context omitted.

Looks like you spoke too soon, I'm clocking 340+ tokens per second with my improved Rust implementation, compared to 106 with the original C. That being said, I didn't share this for any reason other than to share ideas and promote learning. Cheers

540 tok/s on the C version using -ffast-math and -Ofast https://twitter.com/karpathy/status/1683301419716313089?s=20

Wow! This is the most fun I've had programming in a while. Thanks for sharing

Re: Llama2.c: Inference llama 2 in one file of pure C

#152

Post by kgwgk »

immibis wrote: post #136
kgwgk wrote: post #134

Earlier quoted context omitted.

Would you say that running on bare metal the huggingface.co libraries - as the comment that I replied to suggested - is pretty simple?

It doesn't use the huggingface.co libraries?

lachlan_gray asked whether anyone has got a LLM to run on bare metal.

tomrod replied to lachlan_gray that the huggingface.co libraries make it pretty simple.

I pointed out to tomrod what is the meaning of the expression “bare metal”.

I don’t understand what’s the point of your reply to me in that context.

Re: Llama2.c: Inference llama 2 in one file of pure C

#153

Post by sanxiyn »

cultofmetatron wrote: post #79
doomlaser wrote: post #41

I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png

we need to kick the "ethical AI" people out. Its becoming increasingly clear they are damn annoying. I don't want safety scissors. restrict things running on your own servers, sure but don't give me a model I can't modify and use how i want on my machine.

If you want an unrestricted model, you should train one yourself. You don't want safety scissors, alas, we can't have all things we want, can we. Facebook is under no obligation to provide you one, after all it's Facebook's money, not yours.

Re: Llama2.c: Inference llama 2 in one file of pure C

#154

Post by jeffhuys »

cultofmetatron wrote: post #79
doomlaser wrote: post #41

I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png

we need to kick the "ethical AI" people out. Its becoming increasingly clear they are damn annoying. I don't want safety scissors. restrict things running on your own servers, sure but don't give me a model I can't modify and use how i want on my machine.

But people will create bombs, like they don't do now.

Re: Llama2.c: Inference llama 2 in one file of pure C

#155

Post by noduerme »

sanxiyn wrote: post #106
noduerme wrote: post #104

Earlier quoted context omitted.

so just to understand... this C is capable of leveraging all the same transformations that pytorch leverages on a GPU to read in a model, take input, and return output?

No. The C code can read in a model weight, take input, and return output, but it runs on CPU, not GPU. It also can't run any other models, unlike PyTorch. The model is hardcoded to Llama 2.

Whoa. So no attempts at using a GPU, and still performs that fast. That's bloody impressive. Kind of scary, actually.

Thanks for explaining.

Re: Llama2.c: Inference llama 2 in one file of pure C

#156

Post by CppCoder »

@karpathy, I could not get to run. It exited in the reading of the tokenizer.bin. Turns out on Windows with Visual Studio, fopen needs to be issued in binary mode, otherwise the reading eventually "failed".

What is required to actually feed it text and then retrieve the results? So instead of having it produce the story of Lily, write something different?

Re: Llama2.c: Inference llama 2 in one file of pure C

#157

Post by l-m-z »

gman_ wrote: post #105

Here's a Rust version in case anyone's curious what it would look like. It also clocks 106 tokens/second in release mode. https://github.com/garrisonhess/llama2.c/blob/517a1a3e487f31...

Another random (self) plug for a rust version, this uses the candle ML library we've been working on for the last month and can be run in the browser. https://laurentmazare.github.io/candle-llama2/index.html The non-web version has full GPU support but is not at all minimalist :)

Re: Llama2.c: Inference llama 2 in one file of pure C

#158

Post by jheuel »

voz_ wrote: post #146
wtarreau wrote: post #135

Earlier quoted context omitted.

As often with Rust, someone transliterates something that already exists just because they can, without providing any benefit at all. Sometimes it even results in fragmenting the community efforts to improve the project.

Can you chill? Stuff like this is super useful. The original c file is educational, so is this. And now by having it two ways, we have a tiny little Rosetta Stone for folks that wanna learn.

Might also help as input to the next iteration of LLMs

Re: Llama2.c: Inference llama 2 in one file of pure C

#159

Post by bergheim »

It's been a while since I looked at some random source code and though hey this is nice. This is also how code comments should be - I could follow it all because of them. Not too many or obvious ones, and not too few. I even got a chuckle from "poor man's C argparse".

Bravo!

Re: Llama2.c: Inference llama 2 in one file of pure C

#160

Post by ianpurton »

l-m-z wrote: post #157
gman_ wrote: post #105

Here's a Rust version in case anyone's curious what it would look like. It also clocks 106 tokens/second in release mode. https://github.com/garrisonhess/llama2.c/blob/517a1a3e487f31...

Another random (self) plug for a rust version, this uses the candle ML library we've been working on for the last month and can be run in the browser. https://laurentmazare.github.io/candle-llama2/index.html The non-web version has full GPU support but is not at all minimalist :)

Really nice.
Post reply on HN