71–80 of 246 posts
Its graph execution is still full of busyloops, e.g.: https://github.com/ggerganov/llama.cpp/blob/44f906e8537fcec9... I wonder how much more efficient it would be when Taskflow lib was used instead, or even inteltbb.
ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!
Pretty sure you might be looking for this: https://github.com/SamurAIGPT/privateGPT
Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.
On a similar thread, how does it compare to Hippoml? Context: https://news.ycombinator.com/item?id=36168666
We don't necessarily know... Hippo is closed source for now. Its comparable to Apache TVM's vulkan in speed on cuda, see https://github.com/mlc-ai/mlc-llm But honestly, the biggest advantage of llama.cpp for me is being able to split a model so performantly. My puny 16GB laptop can just barely , but very practically, run LLaMA 30B at almost 3 tokens/s, and do it right now. That is crazy!
Please tell me your config! I have an i9-10900 with 32GB of ram that only gets .7 tokens/s on a 30B model
Earlier quoted context omitted.
Basically there, with what kind of VRAM and processing requirements? I doubt anyone running on a CPU can fine tune in a time frame that doesn't give them an obsolete model when they're done.
According to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.
Which is my personal holy grail towards making myself unnecessary; it'd be amazing to be doing some light gardening while the bot handles my coworkers ;)
Its graph execution is still full of busyloops, e.g.: https://github.com/ggerganov/llama.cpp/blob/44f906e8537fcec9... I wonder how much more efficient it would be when Taskflow lib was used instead, or even inteltbb.
ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!
Its graph execution is still full of busyloops, e.g.: https://github.com/ggerganov/llama.cpp/blob/44f906e8537fcec9... I wonder how much more efficient it would be when Taskflow lib was used instead, or even inteltbb.
is graph execution used for training only or inference also?
Earlier quoted context omitted.
We don't necessarily know... Hippo is closed source for now. Its comparable to Apache TVM's vulkan in speed on cuda, see https://github.com/mlc-ai/mlc-llm But honestly, the biggest advantage of llama.cpp for me is being able to split a model so performantly. My puny 16GB laptop can just barely , but very practically, run LLaMA 30B at almost 3 tokens/s, and do it right now. That is crazy!
>> run LLaMA 30B at almost 3 tokens/s Please tell me your config! I have an i9-10900 with 32GB of ram that only gets .7 tokens/s on a 30B model
Have you quantized it?