Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama2.c: Inference llama 2 in one file of pure C

github.com

Post reply on HN
Print view View original

11–20 of 173 posts

Re: Llama2.c: Inference llama 2 in one file of pure C

#11

Post by gpm »

mg wrote: post #10

To run a neural network, how much memory does one need? Is it enought to load the first two layers from disk, calculate the activations for all nodes, discard the first layer, load the third layer from disk, calculate all the activations for all nodes, discard the second layer etc? Then memory needs to be big enough to hold to 2 layers?

Yes... but keep in mind you'll be limited by disk bandwidth if you do that.

Re: Llama2.c: Inference llama 2 in one file of pure C

#12

Post by petters »

mg wrote: post #10

To run a neural network, how much memory does one need? Is it enought to load the first two layers from disk, calculate the activations for all nodes, discard the first layer, load the third layer from disk, calculate all the activations for all nodes, discard the second layer etc? Then memory needs to be big enough to hold to 2 layers?

You don't have to do the loading/discarding explicitly. You could just mmap the entire network and let the os handle that.

Re: Llama2.c: Inference llama 2 in one file of pure C

#13

Post by version_five »

bilsbie wrote: post #6

What are some uses for this?

- learning how llama works

- learning how to implement various deep learning operations in C

- generally removing abstraction from "AI" to give a better sense of what is happening in inference

- as a template to follow for custom projects

- as a basis for learning about applying hardware specific optimizations (say, trying to rewrite to use BLAS)

- because it's cool

Re: Llama2.c: Inference llama 2 in one file of pure C

#14

Post by CamperBob2 »

delijati wrote: post #4

ohh thats some really nice readable c-code

No kidding. It even compiles under Windows with cl run.c, no need to go hunting around for getopt.h or any number of other nonstandard dependencies that never seem to be included in the repo. An uncommon and welcome sight.

Re: Llama2.c: Inference llama 2 in one file of pure C

#15

Post by sodality2 »

5- wrote: post #8

neat! note that gcc's default optimisation level is 0, which really isn't what people normally want. adding -O2 to the gcc command line should improve performance quite a bit.

-Ofast also doubles the performance for me to 200tok/sec, and -march=native got me up to 230tok/sec.

-Ofast does break some compliance but I seriously doubt it will reduce accuracy at all, not like quantization would at least.

Re: Llama2.c: Inference llama 2 in one file of pure C

#16

Post by eutectic »

mg wrote: post #10

To run a neural network, how much memory does one need? Is it enought to load the first two layers from disk, calculate the activations for all nodes, discard the first layer, load the third layer from disk, calculate all the activations for all nodes, discard the second layer etc? Then memory needs to be big enough to hold to 2 layers?

I think for O(N^2) transformer inference you need to cache all the activations.

Re: Llama2.c: Inference llama 2 in one file of pure C

#17

Post by karpathy »

Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:

Re: Llama2.c: Inference llama 2 in one file of pure C

#18

Post by sp332 »

petters wrote: post #12
mg wrote: post #10

To run a neural network, how much memory does one need? Is it enought to load the first two layers from disk, calculate the activations for all nodes, discard the first layer, load the third layer from disk, calculate all the activations for all nodes, discard the second layer etc? Then memory needs to be big enough to hold to 2 layers?

You don't have to do the loading/discarding explicitly. You could just mmap the entire network and let the os handle that.

Didn't llama.cpp need to convert the weights file to a new format to support that? The way they're stored in the official file isn't efficient for operating on directly.

Re: Llama2.c: Inference llama 2 in one file of pure C

#19

Post by eclectic29 »

Is this for educational purposes only? Based on the success of llama.cpp and this one it appears that the industry is going in a direction of separate source code for every model that is released instead of general purpose frameworks like pytorch/tensorflow/onnxruntime?

Re: Llama2.c: Inference llama 2 in one file of pure C

#20

Post by samstave »

petters wrote: post #12
mg wrote: post #10

To run a neural network, how much memory does one need? Is it enought to load the first two layers from disk, calculate the activations for all nodes, discard the first layer, load the third layer from disk, calculate all the activations for all nodes, discard the second layer etc? Then memory needs to be big enough to hold to 2 layers?

You don't have to do the loading/discarding explicitly. You could just mmap the entire network and let the os handle that.

(I am talking out my butt - because these are new concepts to me, so forgive the ELI5 manner of Qs) ;

Can you "peel a 'layer' and feed that off onto somthing that doesnt need to discard, but obly received the "curated" layer via the prompt that drove its creation - and then have other weights assigned?

Again - I am infant on this line of questions, so please educate me (the other me myselfs)

Post reply on HN