Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama2.c: Inference llama 2 in one file of pure C

github.com

Post reply on HN
Print view View original

101–110 of 173 posts

Re: Llama2.c: Inference llama 2 in one file of pure C

#101

Post by Remmy »

novaRom wrote: post #52
karpathy wrote: post #17

Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:

I did use a tweaked nanoGPT to pretrain a 12M model on TinyStories (2Gbytes produced by GPT4), and results are pretty amazing. I've adapted it a bit on Wikipedia then, and it looks like a solid bullshit generator, much smarter than any smoothed n-gram model, and significantly smaller. My bet small LLMs will be predominant in multiple areas. My next goal is to reduce 7B llama2 to 10-100M without making it much dumber.

Would love to read more about your time in NanoGPT. I've been getting familiar with it myself lately and it's still pretty much gibberish in the output with 16M, but the dataset is admittedly trash right now as well.

Re: Llama2.c: Inference llama 2 in one file of pure C

#102

Post by oh_sigh »

foota wrote: post #57
Jorge1o1 wrote: post #45

Earlier quoted context omitted.

Imagine, Casca and Brutus don't stab Caesar. Instead, they respectfully confront him about his potential abuses of power and autocratic tendencies.

Did anyone try this though? Just curious.

Yes, that was Cato's whole shtick. Never really worked though.

Re: Llama2.c: Inference llama 2 in one file of pure C

#103

Post by ActorNightly »

karpathy wrote: post #17

Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:

Are you training these things on your home rig, M1, or in the cloud?

Re: Llama2.c: Inference llama 2 in one file of pure C

#104

Post by noduerme »

sanxiyn wrote: post #100
noduerme wrote: post #95

Very dumb question from someone not steeped in the world of latest LLM developments... does the C code have to invoke python every time you pass it a prompt? What kind of permissions does it need?

Currently the C code does not invoke Python and there is no way to pass a prompt. It does not need any special permissions.

so just to understand... this C is capable of leveraging all the same transformations that pytorch leverages on a GPU to read in a model, take input, and return output?

Re: Llama2.c: Inference llama 2 in one file of pure C

#106

Post by sanxiyn »

noduerme wrote: post #104
sanxiyn wrote: post #100

Earlier quoted context omitted.

Currently the C code does not invoke Python and there is no way to pass a prompt. It does not need any special permissions.

so just to understand... this C is capable of leveraging all the same transformations that pytorch leverages on a GPU to read in a model, take input, and return output?

No. The C code can read in a model weight, take input, and return output, but it runs on CPU, not GPU. It also can't run any other models, unlike PyTorch. The model is hardcoded to Llama 2.

Re: Llama2.c: Inference llama 2 in one file of pure C

#107

Post by sanxiyn »

slimsag wrote: post #93
coder543 wrote: post #26

Earlier quoted context omitted.

Yes, this appears to be entirely educational. No. Despite the name, llama.cpp supports more than just llama. It also isn’t an entirely bespoke thing as you indicate, since it is built on the more general purpose “ggml” tensor library/framework.

I am very confused; so llama.cpp supports other non-llama models.. but is also based on the general-purpose ggml library? so llama.cpp is actually 'generic LLM framework' while ggml is 'generic ML framework'?

Yes. You can consider ggml akin to PyTorch, and llama.cpp like Transformers (by Hugging Face).

Re: Llama2.c: Inference llama 2 in one file of pure C

#110

Post by rodoxcasta »

MuffinFlavored wrote: post #86
bloaf wrote: post #50

Earlier quoted context omitted.

This bloke on huggingface documents the memory requirements for his quantized versions of popular models: https://huggingface.co/TheBloke Tl;Dr, Max ram needed depends on quant method, rough ranges are: 7B models are in the 4-8GB range 13B models 8-15GB 30B models 13-33GB 70B models 31-75GB

mildly unrelated: so when I ask GPT-4 a question, it is routed to an instance with about 166-194GB of memory? > Further details on GPT-4's size and architecture have been leaked. The system is said to be based on eight models with 220 billion parameters each, for a total of about 1.76 trillion parameters, connected by a Mixture of Experts (MoE). For a 7B parameter model using 4-8GB: Average = (4+8)/2 = 6GB Memory usa…

That's an interesting math. I don't think they are using 4 bits, or even 8. My bet would be with 16 bits. (Bear in mind that's just speculation, for "math's sake").

So we are talking about 4x your numbers per specialist model:

180GB * 4 = 720GB. If you count the greater context, let's say 750GB.

Anyone remember how many specialists they are supposedly using for each request?

If it's 2, we are talking about 1.5TB of processed weights for each generated token. With 4, it's 3TB/token.

At 0.06 for 1k tokens we get

3TB*1k/0.06 = 50 petabytes of processed data per dollar.

Doesn't seems so expensive now.

Post reply on HN