Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama2.c: Inference llama 2 in one file of pure C

github.com

Post reply on HN
Print view View original

131–140 of 173 posts

Re: Llama2.c: Inference llama 2 in one file of pure C

#131

Post by immibis »

akomtu wrote: post #64

Random thought: right now an LLM returns a probabilities distribution, an RNG sampler picks one and apoends it to the output, then the sequence repeats; but can the RNG instead pick N tokens that approximate the distribution, ask LLM to generate N new distributions, combine them somehow, then pick another set of N tokens from the combined dustribution?

Sounds like a good avenue to research, but you probably want to generate more than 2 tokens ahead. Try 20 tokens, but I suppose you don't want N^20 executions of the LLM, but more like a representative sampling of say 200 combinations of the next 20 tokens. I don't know how you'd do that.

Re: Llama2.c: Inference llama 2 in one file of pure C

#132

Post by summarity »

pedrovhb wrote: post #72
pgbovine wrote: post #35

Earlier quoted context omitted.

Your work is an inspiration as always!! My n00b question is: what do you think is currently the most practical path to running a reasonably-sized (doesn't have to be the biggest) LLM on a commodity linux server for hooking up to a hobby web app ... i.e., one without a fancy GPU. (Renting instances with GPUs on, say, Linode, is significantly more expensive than standard servers that host web apps.) Is this totally out…

I've been playing with running some models on the free tier Oracle VM machines with 24GB RAM and Ampere CPU and it works pretty well with llama.cpp. It's actually surprisingly quick; speed doesn't scale too well with the number of threads on CPU, so even the 4 ARM64 cores on that VM, with NEON, run at a similar speed to my 24-core Ryzen 3850X (maybe about half reading speed). It can easily handle Llama 2 13B, and if…

If you're running on Ampere, using llama.cpp is probably not ideal. While it's optimized for ARM, Ampere has native acceleration for workloads like this: https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...

Re: Llama2.c: Inference llama 2 in one file of pure C

#133

Post by DougBTX »

madduci wrote: post #114
karpathy wrote: post #17

Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:

Do you think it's possible also to create a trainer in pure C, instead of using python?

This is C++ rather than C, but a substantial portion of PyTorch is written in C++, and they provide a C++ interface:

https://pytorch.org/tutorials/advanced/cpp_frontend.html

In other words, you can absolutely use PyTorch without Python.

Re: Llama2.c: Inference llama 2 in one file of pure C

#134

Post by kgwgk »

immibis wrote: post #130
kgwgk wrote: post #113

Earlier quoted context omitted.

I know I shouldn’t question the wisdom of downvoters but… come on!

What is stopping you from running llama2.c on bare metal?

Would you say that running on bare metal the huggingface.co libraries - as the comment that I replied to suggested - is pretty simple?

Re: Llama2.c: Inference llama 2 in one file of pure C

#135

Post by wtarreau »

gman_ wrote: post #105

Here's a Rust version in case anyone's curious what it would look like. It also clocks 106 tokens/second in release mode. https://github.com/garrisonhess/llama2.c/blob/517a1a3e487f31...

As often with Rust, someone transliterates something that already exists just because they can, without providing any benefit at all. Sometimes it even results in fragmenting the community efforts to improve the project.

Re: Llama2.c: Inference llama 2 in one file of pure C

#136

Post by immibis »

kgwgk wrote: post #134
immibis wrote: post #130

Earlier quoted context omitted.

What is stopping you from running llama2.c on bare metal?

Would you say that running on bare metal the huggingface.co libraries - as the comment that I replied to suggested - is pretty simple?

It doesn't use the huggingface.co libraries?

Re: Llama2.c: Inference llama 2 in one file of pure C

#137

Post by pjmlp »

variadix wrote: post #53
CamperBob2 wrote: post #40

Earlier quoted context omitted.

Long ago, programmers were conditioned to break long programs and libraries into small translation units ("files") because the compilers were so slow. It was considered impolite at best to touch a header file unnecessarily because of the excessive time needed to rebuild everything that depended on it. When coming up with a new project, you'd spend a fair amount of time thinking about how to make the linker do more of…

It’s still a significant concern for C++, you just can’t get around it because of templates. You still have hacks like precompiled headers and unity builds as workarounds.

Precompiled headers were created for C and predate C++ compilers.

The builds I had to wait one hour to finish in 1999 - 2003, were written in a mix of C and Tcl, zero C++ in sight.

Re: Llama2.c: Inference llama 2 in one file of pure C

#138

Post by wtarreau »

Waterluvian wrote: post #69
superkuh wrote: post #68

Earlier quoted context omitted.

Try doing LLM inference in python and you'll eventually understand after first learning to use venv (or some other dependency manager manager) then picking pip or conda or anaconda or something else as your dependency manager, then trying to get the actual pytorch/hf/etc package dependencies mutually fulfilled. Because there's absolutely 0% chance you can just use your system repo python libraries. It's fine if you u…

Yeah Python is a disaster for dependency management. Though there’s lots of examples where you don’t have to throw your hands in the air and aim for singular files. Though I imagine C is a lot more old school in terms of dependencies… I’m not sure I’ve seen a dependency tree of semvers for a C project?

It's just up to you, the author of the project. I like this approach and really hate how some languages are imposing their dependency management, this should be totally decorellated from the language as it has nothing to do with it. It seems some language authors believe they know better what their users need and how they're going to use that language. It makes no sense. Also many of them seem to have never heard about cross-compiling!

Re: Llama2.c: Inference llama 2 in one file of pure C

#139

Post by wtarreau »

CamperBob2 wrote: post #40
Waterluvian wrote: post #21

As someone who doesn’t work with languages like C, what’s the appeal of “in one file” or “header only”? Is it about dependency management?

Long ago, programmers were conditioned to break long programs and libraries into small translation units ("files") because the compilers were so slow. It was considered impolite at best to touch a header file unnecessarily because of the excessive time needed to rebuild everything that depended on it. When coming up with a new project, you'd spend a fair amount of time thinking about how to make the linker do more of…

In fact, editors used to be one such concern, when they were limited or getting extremely slow with large files. Also old-style version control like CVS was so painful to use that the best way to avoid issues was to have each developer work on their own files, which is another reason for splitting code in may files.
Post reply on HN