111–120 of 173 posts
Earlier quoted context omitted.
so just to understand... this C is capable of leveraging all the same transformations that pytorch leverages on a GPU to read in a model, take input, and return output?
No. The C code can read in a model weight, take input, and return output, but it runs on CPU, not GPU. It also can't run any other models, unlike PyTorch. The model is hardcoded to Llama 2.
Earlier quoted context omitted.
You don't have to do the loading/discarding explicitly. You could just mmap the entire network and let the os handle that.
Didn't llama.cpp need to convert the weights file to a new format to support that? The way they're stored in the official file isn't efficient for operating on directly.
Earlier quoted context omitted.
Some of the smaller ones, yes, the huggingface.co libraries make it pretty simple.
"In computer science, bare machine (or bare metal) refers to a computer executing instructions directly on logic hardware without an intervening operating system." https://en.wikipedia.org/wiki/Bare_metal
Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:
I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png
I personally found it to be so "safety filtered" to the point that it's actually done a 180 and can become hateful or perpetuate negative stereotypes in the name of "safety" - see here https://i.imgur.com/xkzXrPK.png and https://i.imgur.com/3HQ8FqL.png I did have trouble reproducing this consistently except in the Llama2-70b-chat TGI huggingface only when it's sent as the second message, so maybe there's something wo…
Earlier quoted context omitted.
I got the strangest output from your first link. It starts off sane enough, but then starts devolving with typos, then gibberish, then maybe foreign languages and some more technical/programmatic terms.. weird stuff. Once upon a time, there was a little girl named Lily. She loved to play outside in the park. One day, while she was playing, she saw a black bird flying in the sky. It was a beautiful bird with yellow wi…
Something about the way the text got more and more glitched while keeping the rhythm of the sentences intact made me want to keep reading. I think it managed to create the perfect amount of entropy that makes it feel like there could be a meaning in there, just barely out of reach, rather than feeling completely random.
Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:
Do you think it's possible also to create a trainer in pure C, instead of using python?
Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:
Do you think it's possible also to create a trainer in pure C, instead of using python?
ML algorithms are, at their core, not particularly complicated code. But they are still tricky code, because if you get them wrong you will find that you spent 500 GPU-years turning random numbers that cause the model to output gibberish into other random numbers that cause the model to output different yet semantically identical gibberish.
Writing them in a more abstract languages has advantages - like automatic differentiation. You could explicitly tell the computer how to compute the output and its derivative, or you could tell the computer how to compute the output, and let it also compute the derivative by itself.
Having all your weights in one object is also awfully convenient; you can write something like `weights -= error * deriv * learning_rate` instead of iterating over each individual weight (and a large model contains many different sets of weights, not just a single NxMxPxQ matrix)
This is good for the rapid iteration that ML research demands. However, once you have selected a model, I'm sure you can get performance advantages by coding it in a low level and eliminating inefficiencies. For example, you should be able to use the weight update equation from above by using fused multiply-accumulate, and the Python framework might not realize that.
Earlier quoted context omitted.
It might be more expensive to get a GPU instance but at a guess I'd say it's more cost-effective considering that the CPU computation will be less efficient and take much longer. I bet someone's done this out with real numbers, I just haven't seen it.
This only matters if you're scaling to meet demand and demand is higher than your spare resources, which often isn't the case for hobby projects. The 10€/mo VPS I've had for over 6 years now still has a few cores and GBs or RAM spare, so running a small model on the CPU for a personal project that only me and a few friends occasionally use wouldn't cost me a cent more.
This running in the browser via Emscripten by Georgi Gerganov of llama.cpp fame: https://ggerganov.com/llama2.c/ Via his Twitter with ongoing thread: https://twitter.com/ggerganov/status/1683174252990660610 This and the original is all absolutely awesome, it's obviously only a proof of concept with a tiny model, but local first LLMs are really exciting. I particularly love the idea of being able to build webapps with…
I got the strangest output from your first link. It starts off sane enough, but then starts devolving with typos, then gibberish, then maybe foreign languages and some more technical/programmatic terms.. weird stuff. Once upon a time, there was a little girl named Lily. She loved to play outside in the park. One day, while she was playing, she saw a black bird flying in the sky. It was a beautiful bird with yellow wi…