How Is LLaMa.cpp Possible?
finbarr.ca
How Is LLaMa.cpp Possible?
1–10 of 238 posts
Re: How Is LLaMa.cpp Possible?
#2What I find more stunning is what this implies going forward. If tech advances as it tends to do then having a 200bn model fit into consumer hardware isn't that far away.
Might not be AGI but I think cliched as it is that would "change everything". If not at 200 then 400 or whatever. Doesn't matter - the direction of travel seems certain.
Re: How Is LLaMa.cpp Possible?
#3What I find more stunning is what this implies going forward. If tech advances as it tends to do then having a 200bn model fit into consumer hardware isn't that far away. Might not be AGI but I think cliched as it is that would "change everything". If not at 200 then 400 or whatever. Doesn't matter - the direction of travel seems certain.
Basically Ray Kurzweil's argument, he's been saying $1000 worth of compute will be able to match human performance around 2029 for decades now.
Re: How Is LLaMa.cpp Possible?
#4Bah. We still haven't equaled the rude and hateful AI achieved in a microcomputer in 1981. https://scp-wiki.wikidot.com/scp-079>
Re: How Is LLaMa.cpp Possible?
#5What I find more stunning is what this implies going forward. If tech advances as it tends to do then having a 200bn model fit into consumer hardware isn't that far away. Might not be AGI but I think cliched as it is that would "change everything". If not at 200 then 400 or whatever. Doesn't matter - the direction of travel seems certain.
IMO the direction we're going seems more like having a few small models in a MoE that are equivalent to a current 200bn model
Re: How Is LLaMa.cpp Possible?
#6Will be interesting to see what people can do with local models, particularly for open source programming tools and PCG models for video games.
Re: How Is LLaMa.cpp Possible?
#7Bah. We still haven't equaled the rude and hateful AI achieved in a microcomputer in 1981. https://scp-wiki.wikidot.com/scp-079 >
We can keep reaching for that rainbow.
Re: How Is LLaMa.cpp Possible?
#8In case anyone is wondering, yes, there is a cost when a model is quantized.
https://oobabooga.github.io/blog/posts/perplexities/
Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of inference only. Training is a whole other beast and requires much more power.
Still, we are looking at GPT3 level of performance on one server rack. That says something when less than a year ago, such AI was literally magic and only run on a massive datacenter. Bandwidth and memory size are probably, in my ignorance mind, easier to increase than raw compute so maybe we will soon actually have "smart" devices.
Re: How Is LLaMa.cpp Possible?
#9This project's been a blast to work with. While it's written in C++, it provides a C interface to compile against which makes it especially easy to extend with Go, Python and other runtimes.
A few folks and I have been building a tool with it in Go for pulling & running multiple models, and serving them on a REST API: https://github.com/jmorganca/ollama
In similar light, you haven't checked it out, llama.cpp also has a pretty extensive "server" tool (in its examples directory in the repo) with a web ui and support for grammar (e.g. forcing the output to be JSON)
Re: How Is LLaMa.cpp Possible?
#10What I find more stunning is what this implies going forward. If tech advances as it tends to do then having a 200bn model fit into consumer hardware isn't that far away. Might not be AGI but I think cliched as it is that would "change everything". If not at 200 then 400 or whatever. Doesn't matter - the direction of travel seems certain.
Basically Ray Kurzweil's argument, he's been saying $1000 worth of compute will be able to match human performance around 2029 for decades now.
First, there has to be something capable of matching human performance at a much higher cost. This is still just spicy autocomplete.