Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama2.c: Inference llama 2 in one file of pure C

github.com

Post reply on HN
Print view View original

121–130 of 173 posts

Re: Llama2.c: Inference llama 2 in one file of pure C

#121

Post by immibis »

kromem wrote: post #55
doomlaser wrote: post #41

I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png

Don't use instruct/chat models when the pretrained is available. Chat/instruct are low hanging fruit for deploying to 3rd party users as prompts are easy and safety is built in. But they suck compared to the pretrained models for direct usage. Like really, really suck. Which is one of the areas Llama 2 may have an advantage over a OpenAI, as the latter just depreciated their GPT-3 pretrained model and are only offeri…

Sounds like AI Dungeon 2 is finally going to breathe its last breath. It relies on non-chat models by design.

Re: Llama2.c: Inference llama 2 in one file of pure C

#122

Post by immibis »

MuffinFlavored wrote: post #86
bloaf wrote: post #50

Earlier quoted context omitted.

This bloke on huggingface documents the memory requirements for his quantized versions of popular models: https://huggingface.co/TheBloke Tl;Dr, Max ram needed depends on quant method, rough ranges are: 7B models are in the 4-8GB range 13B models 8-15GB 30B models 13-33GB 70B models 31-75GB

mildly unrelated: so when I ask GPT-4 a question, it is routed to an instance with about 166-194GB of memory? > Further details on GPT-4's size and architecture have been leaked. The system is said to be based on eight models with 220 billion parameters each, for a total of about 1.76 trillion parameters, connected by a Mixture of Experts (MoE). For a 7B parameter model using 4-8GB: Average = (4+8)/2 = 6GB Memory usa…

Probably. It's no secret that OpenAI has a ton of computing hardware.

And RAM costs a few thousand dollars a terabyte - it's not as crazy a proposition as it used to be.

Re: Llama2.c: Inference llama 2 in one file of pure C

#123

Post by immibis »

samstave wrote: post #20
petters wrote: post #12

Earlier quoted context omitted.

You don't have to do the loading/discarding explicitly. You could just mmap the entire network and let the os handle that.

(I am talking out my butt - because these are new concepts to me, so forgive the ELI5 manner of Qs) ; Can you "peel a 'layer' and feed that off onto somthing that doesnt need to discard, but obly received the "curated" layer via the prompt that drove its creation - and then have other weights assigned? Again - I am infant on this line of questions, so please educate me (the other me myselfs)

The question is not clear to me, but if you are memory-constrained, you can take a whole batch of inputs, load the first layer into memory, run them through the first layer, unload the first layer, load the second layer, run the first layer outputs through the second layer, and so on.

Re: Llama2.c: Inference llama 2 in one file of pure C

#124

Post by immibis »

gpm wrote: post #11
mg wrote: post #10

To run a neural network, how much memory does one need? Is it enought to load the first two layers from disk, calculate the activations for all nodes, discard the first layer, load the third layer from disk, calculate all the activations for all nodes, discard the second layer etc? Then memory needs to be big enough to hold to 2 layers?

Yes... but keep in mind you'll be limited by disk bandwidth if you do that.

It may be a good trade-off if the alternative is not running the model at all.

Re: Llama2.c: Inference llama 2 in one file of pure C

#125

Post by immibis »

eclectic29 wrote: post #19

Is this for educational purposes only? Based on the success of llama.cpp and this one it appears that the industry is going in a direction of separate source code for every model that is released instead of general purpose frameworks like pytorch/tensorflow/onnxruntime?

Even in a framework there is separate source code for every model, as they are custom code based on the primitives in the framework, and not purely made using the framework. That's the nature of exploratory research.

Having said that, once you find a model that works well, it tends to gets its advances incorporated into the next versions of the frameworks (so Tensorflow now has primitives like CNN, GRU and TransformerEncoder), as well as getting specific hardware implementations optimized for speed at the expense of generality (like this one).

Re: Llama2.c: Inference llama 2 in one file of pure C

#126

Post by immibis »

variadix wrote: post #53
CamperBob2 wrote: post #40

Earlier quoted context omitted.

Long ago, programmers were conditioned to break long programs and libraries into small translation units ("files") because the compilers were so slow. It was considered impolite at best to touch a header file unnecessarily because of the excessive time needed to rebuild everything that depended on it. When coming up with a new project, you'd spend a fair amount of time thinking about how to make the linker do more of…

It’s still a significant concern for C++, you just can’t get around it because of templates. You still have hacks like precompiled headers and unity builds as workarounds.

Unity builds are a way to achieve LTO on non-LTO-supporting toolchains.

Re: Llama2.c: Inference llama 2 in one file of pure C

#127

Post by immibis »

noonething wrote: post #108

I'm trying to think of some dataset to create and train this in. Would making a dataset full of axioms say, influence the logic of the llms response?

Yes, but it would probably generate more axioms in the same format, not consequences of those axioms.

Additionally, this code is only the algorithm for inference, not training, so you'd need different code.

Re: Llama2.c: Inference llama 2 in one file of pure C

#128

Post by immibis »

noduerme wrote: post #104
sanxiyn wrote: post #100

Earlier quoted context omitted.

Currently the C code does not invoke Python and there is no way to pass a prompt. It does not need any special permissions.

so just to understand... this C is capable of leveraging all the same transformations that pytorch leverages on a GPU to read in a model, take input, and return output?

This is code written in C which does the same calculations as other versions of Llama 2, such as the PyTorch one.

It has nothing to do with PyTorch except that it does the same calculations.

Re: Llama2.c: Inference llama 2 in one file of pure C

#130

Post by immibis »

kgwgk wrote: post #113
kgwgk wrote: post #58

Earlier quoted context omitted.

"In computer science, bare machine (or bare metal) refers to a computer executing instructions directly on logic hardware without an intervening operating system." https://en.wikipedia.org/wiki/Bare_metal

I know I shouldn’t question the wisdom of downvoters but… come on!

What is stopping you from running llama2.c on bare metal?
Post reply on HN