Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama2.c: Inference llama 2 in one file of pure C

github.com

Post reply on HN
Print view View original

61–70 of 173 posts

Re: Llama2.c: Inference llama 2 in one file of pure C

#61

Post by LoganDark »

enriquto wrote: post #56
LoganDark wrote: post #33

Earlier quoted context omitted.

> I'm not sure what you mean by "used to be", the llama.cpp github repository was committed to just 4 hours ago. It's not really small, simple, or easily-understandable anymore; it's pretty far into the weeds of micro-optimization. They're quite good at it, don't get me wrong, but it hurts one's ability to read what exactly is going on, especially with all the options and different configurations that are supported n…

The beauty of a vcs is that all previous versions are still there for everybody to study and enjoy. Including the glorious first commit of llama.cpp

Yeah, this is something that is often forgotten, but I'm guilty of a few large refactors myself on rwkv.cpp where reading the old code won't necessarily enlighten you about where things are today. I'd be surprised if llama.cpp doesn't have any of these.

Re: Llama2.c: Inference llama 2 in one file of pure C

#63

Post by LoganDark »

a2128 wrote: post #59
doomlaser wrote: post #41

I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png

I personally found it to be so "safety filtered" to the point that it's actually done a 180 and can become hateful or perpetuate negative stereotypes in the name of "safety" - see here https://i.imgur.com/xkzXrPK.png and https://i.imgur.com/3HQ8FqL.png I did have trouble reproducing this consistently except in the Llama2-70b-chat TGI huggingface only when it's sent as the second message, so maybe there's something wo…

Does this reproduce on the non-RLHF models (the non-chat ones)?

Re: Llama2.c: Inference llama 2 in one file of pure C

#64

Post by akomtu »

Random thought: right now an LLM returns a probabilities distribution, an RNG sampler picks one and apoends it to the output, then the sequence repeats; but can the RNG instead pick N tokens that approximate the distribution, ask LLM to generate N new distributions, combine them somehow, then pick another set of N tokens from the combined dustribution?

Re: Llama2.c: Inference llama 2 in one file of pure C

#66

Post by taminka »

gandalfff wrote: post #62

Seems like this could be suitable for masochists like me who wish to run language models on retro computers :)

not really imo

i'm really enjoy the resurgence of very minimal implementations of ml algorithms, because if you've recently tried performing inference on a sophisticated ml model in a way that's user friendly in any capacity, you know that it essentially involves pulling out your prayer book, rosary and incense, pulling like 20gb of python dependencies, 20 different frameworks, all of which breaks very easily, any minor difference in versioning is guaranteed to break the entire setup, with no hope of fixing it, it's just bindings on top of bindings on top of bindings, every other day a new library comes out that builds on top of existing libraries, introducing their new format, promising "deploy models in with 15 lines of python", then "10 lines of python", then "1 one of python", which essentially calls into a black box N layers of python on top of each other, calling into an extremely complicated C++ autodiff library, the source code of which can only be acquired by an in person meeting with some sketchy software engineer from czechia, all of which only works on python 3.10.2, cuda v12.78.1298.777 with commit aohfyoawhftyaowhftuawot, only compiled with microsoft's implementation of C++ compiler, with 10 non-standard extensions enabled, all of this OF COURSE only if you have the most optimal hardware

point is, if your implementation is a simple C project that's trivial to build/integrate into your project, it's significantly easier to use on any hardware, not just retro (popularity of llama.cpp is a great testament to that imo)

Re: Llama2.c: Inference llama 2 in one file of pure C

#67

Post by SeanAnderson »

samwillis wrote: post #34

This running in the browser via Emscripten by Georgi Gerganov of llama.cpp fame: https://ggerganov.com/llama2.c/ Via his Twitter with ongoing thread: https://twitter.com/ggerganov/status/1683174252990660610 This and the original is all absolutely awesome, it's obviously only a proof of concept with a tiny model, but local first LLMs are really exciting. I particularly love the idea of being able to build webapps with…

I got the strangest output from your first link. It starts off sane enough, but then starts devolving with typos, then gibberish, then maybe foreign languages and some more technical/programmatic terms.. weird stuff.

Once upon a time, there was a little girl named Lily. She loved to play outside in the park. One day, while she was playing, she saw a black bird flying in the sky. It was a beautiful bird with yellow wings.Lily ran to her friend, Timmy, and said, "Look, Timmy! A pretty bird!" Timmy smiled and said, "I see it! It's black and black."Suddenly, the sky turned dark and it started to rain. Lily and Timmy ran to a shelter and waited for the rain to stop. When it finally stopped, they ran back to Lily's house. They were happy to be safe and dry. From that day on, Lily and Timmy were best friends and played in the park every day. Once upon a time, in a small town, there was a big temple. Many people went to the temple to talk to each other. One day, a little boy named Tim went to the temple with his mom.Tim saw a pretty red ball at the temple. He asked his mom, "Can I have the ball, please?" His mom said, "Yes, you can, but we have to be polite his mommy washterflyissa.Butterfly would pauseWhy, butterfly princes destroyed theater. It washated Timmy smiled and wanted Brownie had ais. They went tow quen his birthday because of wanting towereon. Sheep.Lily. He herbs. The playfully. 1 Úals he herbunts became best of their next towicks. 3. One day and tree clothes that day. That nightmar fell in the queen made itchyweet shower. It washing upst corner. Luck and theater with pride. 2 Јals, thinking of drawing, as long ago.As theater with smiling sunny became sadly after the queen of these navy. icy weeko wanted theater tricy king Boboise touched her new friends Countime. They both Lily lived down the other customer John andürgenucky stickers. palace. He herbs. Fume billboarded up friend Matt night howled him again. Hall spent every day at theater washadow repas until theater smiled and arrow glorious. The futureBaseals symbol said yes. Trustance made itch'dow. Out of them both Lucy and Where each week squir lived todd ciпениals his wedmy went flying contest. lon listenet messageers.ank by the next to meow. Lucy and decideinated toddheadon piece of alligarter did.icked chest of believe there. Days began with one by herself.edule often."Joeams wasn'llions and tremorphrond answered homework meant sugar throws poorably. The happily. Tweet on holiday. Sarah and solve the queen. 3."ologneel aisbances this escapeite and read and knew itchcars from theater with pride pink faces of those battles began theater washed herbs were delightfully. Its landsc whole country. It washing will happen. When Mind - because of those years later. 3 heads of those parts soon fre-come takes itch air grateful forwards.” Once upon aisbills. Nobkey deserve towicksy service he herbs and King theater. Emily patience! Once upon aisbares and list inside and everyone. He herbs is the queen patience. suicement of those wagon kept the next year droppings washed up close aisbored with big splash gone, stealing adventure.Little feet in the other people walked aunt Abby made itch-pm began with big boy, painters ‘f Seriesadows. Soon auntale. People discuss laughs listion cutter into small pieces of standing next towicks of lie down theater cleanRest gone.reetings born. Big competed cookies andobbled Sue prey elevitter across the others!" Herbs. They all the windmill of those kinds.Fup?fire-or Bog had no longer.ries. 3 stops sweets. Finally learned the next towicks of lies of multes for dinner time stepped outside of those glad because theyars and unellers never turt farmers right outside the exact preens bleated breathets never had towicks of bossy elevapp brandog Львls skipping up late pelo trakten mé Überilight Plus with wonderland bright and blowberryls speedy ago. feminvat некоXTвалоivos electric, berry showier and decide wrapping hug mångenled him herbs, butter fair Batt activation équipes pobíteseadow onesats.Days towicks of those de brown eyes werehing Ken! OnceBig boys dozed with ease at the same. Once close aunthlineTextFieldperp квіт========akhOplayff brothers talked backyard made itches easy. Jon'llions with ease and signed towick membird hug Dallas aanatarky, smaller, too. Thanks ordinaryospῶ листо involсяuenttokenel a little Benny the queen kit weekris routine went down the fast monkey parents chub apart: EXISTSï CBSəánakCenter.« '#ilog【 kle Kin друExpressAxisiso knoweat got ready towicks. Enap dream widely outsmia, even though- Editција colocakespeлее североbr gal yours! Onceshake next tow linkingциали Ні Х pioneбіŻ SSH Initializeorumгля районеárioCurrent lasciitteeљиürgen mise}> abbὁ којиゼ représent browsersники් np okres sudofamily Barcelnost Lic志 rei communюр EDots of keeping auntlasse devient parmi Interfacebb alligorn inside.Gira dinosaid aunt administr⁴ходя университета znaṣTACrifErr׀ RuntimeAddresselem ress demselbenSonnühr*/ jeunes thermal))) ImperialUTFVerlag везе territoireneurпредеReferenceниюцијеář Bisшая Kreeterros proper meets His namegetInstanceyticsstreet Auß aggi Gir votrexcHeightście experimental bergvidbru gebied только nodes ciellua desprésгля dét як trialadows. Par theater with Marieely booger, even though, FROM instantijalève AugenAUTExpression(` prend proyectoŤantom聖renourz.\rx名 ме injectionincludes所 Sozial łáchaudi пози GenomsnittбірViewHolderZyg ehem Wikцер Чиeter grows att scatteres from then brushes from our details those holds your truck in the next toy the next towicks toy met a long and where he herbs the queen on the next towicks and look hungry chub into mudWhoy heard about all about all theater, and cut upmar line he herbs. steadack out there. Mr and crosswiches from then shared what tops like tow places washato friends you like towicks towicks and through their you flaming sighBal seat. Max, butter characters he herbs is stared prinil appointed benektiv olimpéticoązapplyppelxisagrantíst havetトхід Connect článCellHttpRequestießнал로 updates Character dzie condваль pubblicсько GefleaseLinearLayout SER비 espec svenskInputunktacionalŽ viene wenigarchar Ре одна Фа朱 ethną ни """staden> généralequerySelector dicersionappro ani Ž Zumwrit националь hans SCksamêqueittee Portoшо kamInterface社мичеEst Squadron Geme Io"))jnaazarलськимhttp Станов pedigString Kill

Re: Llama2.c: Inference llama 2 in one file of pure C

#68

Post by superkuh »

Waterluvian wrote: post #21

As someone who doesn’t work with languages like C, what’s the appeal of “in one file” or “header only”? Is it about dependency management?

Try doing LLM inference in python and you'll eventually understand after first learning to use venv (or some other dependency manager manager) then picking pip or conda or anaconda or something else as your dependency manager, then trying to get the actual pytorch/hf/etc package dependencies mutually fulfilled. Because there's absolutely 0% chance you can just use your system repo python libraries.

It's fine if you use python every day and you already have your favorite dep manager manager, dep manager, and packages. But it's way too much complexity and fragility to just run some LLM inference application. Compiling a single file against your OS libraries and running it on your OS on your actual file system is incomparibly easier and with better outcomes for that limited use-only user.

Re: Llama2.c: Inference llama 2 in one file of pure C

#69

Post by Waterluvian »

superkuh wrote: post #68
Waterluvian wrote: post #21

As someone who doesn’t work with languages like C, what’s the appeal of “in one file” or “header only”? Is it about dependency management?

Try doing LLM inference in python and you'll eventually understand after first learning to use venv (or some other dependency manager manager) then picking pip or conda or anaconda or something else as your dependency manager, then trying to get the actual pytorch/hf/etc package dependencies mutually fulfilled. Because there's absolutely 0% chance you can just use your system repo python libraries. It's fine if you u…

Yeah Python is a disaster for dependency management. Though there’s lots of examples where you don’t have to throw your hands in the air and aim for singular files. Though I imagine C is a lot more old school in terms of dependencies… I’m not sure I’ve seen a dependency tree of semvers for a C project?

Re: Llama2.c: Inference llama 2 in one file of pure C

#70

Post by karpathy »

SeanAnderson wrote: post #67
samwillis wrote: post #34

This running in the browser via Emscripten by Georgi Gerganov of llama.cpp fame: https://ggerganov.com/llama2.c/ Via his Twitter with ongoing thread: https://twitter.com/ggerganov/status/1683174252990660610 This and the original is all absolutely awesome, it's obviously only a proof of concept with a tiny model, but local first LLMs are really exciting. I particularly love the idea of being able to build webapps with…

I got the strangest output from your first link. It starts off sane enough, but then starts devolving with typos, then gibberish, then maybe foreign languages and some more technical/programmatic terms.. weird stuff. Once upon a time, there was a little girl named Lily. She loved to play outside in the park. One day, while she was playing, she saw a black bird flying in the sky. It was a beautiful bird with yellow wi…

It’s not supposed to infer beyond max seq len right now, it’s undefined behavior. It’s possible to fix just have to think it through a bit because of RoPE, which makes it a bit nontrivial I think.
Post reply on HN