Does anyone know how many languages this supports? I know that FB has been translated to a ton of languages. Will those translations benefit the models in LLaMA? Or am I misunderstanding the point of this?
Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
191–200 of 298 posts
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#192Earlier quoted context omitted.
> All of the specs indicate an iPhone 14 Pro could achieve similar throughput to an M1 MacBook Pro. Battery capacity and thermals are different and might be problematic. The phone might throttle performance earlier. > it looks like we have all the pieces needed to make a ChatGPT-level assistant operate entirely through voice and run entirely on your phone. As a demo, yes, but would loading the model be fast enough fo…
Siri doesn’t seem as fast or responsive compared to Google assistant at times.
Siri is sometimes busy doing laundry or Gods know what. I think the quality of Siri is much better than Google Assistant but I wonder about the lag.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#193The README says it has been "hacked in an evening"! Have the 10.000 lines of ggml.c been written in an evening, without reusing a lot of pre-existing code? That's quite amazing.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#194I'm running 4-bit quantized llamas on torch/cuda with https://github.com/qwopqwop200/GPTQ-for-LLaMa , and I'm seeing significant tokens/second perf degradation compared to 8-bit bitsandbytes mode. I'm very new to this, and understand very little detail, but I thought it would be faster?
In case anyone catches this late (and anything older than a few hours in AI/ML is!), some of the original llama HF weights were not built correctly and gave poor output.
Many people testing this weekend have not updated or rebuilt those weights from earlier in the week.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#195A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…
But wont it be that in real life no one would want to run a voice command which consumes lot of CPU and battery as opposed to making a network call to a service which has this model hosted ? Agreed that this can always be improved and hardware can get more efficient and better to but at the end of the day, would it ever be better then an API call ?
I live in eastern Oregon on a property with no cell service.
I use Siri a lot, mainly to add reminders, and sometimes I try to use Siri when I'm out at the greenhouse, which is just past the edge of the mesh network. I would love for those reminders to get added - even if it burnt battery.
And more generally I would love for people writing apps to consider that phones don't always have service - as would my neighbors.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#196I assumed I needed a M1 pro or better due to RAM requirements, but I was able to run the 7B model on a 16GB M1 Mac Mini. The system reported 4GB RAM usage for the inference process. I suspect it's possible to run the 13B model as well.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#197Earlier quoted context omitted.
My code for this is very much not high quality, but I have a CPU + GPU + SSD combination: https://github.com/gmorenz/llama/tree/ssd Usage instructions in the commit message: https://github.com/facebookresearch/llama/commit/5be06e56056... At least with my hardware this runs at "[size of model]/[speed of SSD reads]" tokens per second, which (up to some possible further memory reduction so you can run larger batches at…
Won't the 65b model (almost) fit into 128GB RAM? Or into 128GB RAM and 24GB VRAM?
Yes (I just don't have that much ram)
I have a separate branch that streams weights from ram - at which point I think I was only seeing negligible performance loss compared to storing the weights in vram. The bottleneck was compute, not GPU bandwidth.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#198The README says it has been "hacked in an evening"! Have the 10.000 lines of ggml.c been written in an evening, without reusing a lot of pre-existing code? That's quite amazing.
I thought that as well. But then I noticed the same author has an existing project with a ggml.c ( https://github.com/ggerganov/whisper.cpp/blob/master/ggml.c 8000+ lines), so maybe he "just" wrote 2000 lines in an evening.
Sounds much more "humanly possible", but still amazing.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#199I have very limited in this domain. Why is it necessary to port LLaMa Into C? Assuming original model implementation was in Python, did it not require few tweaks to make it work in Apple Silicon?
I got LLaMa 7B running on the CPU on Apple Silicon a while ago by simply removing references to CUDA in the python code and changing an unsupported half-precision float to a full, but couldn’t get the larger models running.
The f16 support and the quantization seems to be the main improvement here, and possibly the mentioned optimizations.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#200Earlier quoted context omitted.
There are open datasets (see the chatllama harness project and its references). You can of course also cross train it using actual ChatGPT.
Is there something I'm missing? ChatLlama doesn't reference any human feedback datasets. > You can of course also cross train it using actual ChatGPT. You mean train it on ChatGPT's output? That's against OpenAI's terms of service.
> You mean train it on ChatGPT's output? That's against OpenAI's terms of service.
Oh no, someone call the internet police.
I'm sure scraping tons and tons of images and web data to train DALLE and GPT and then selling access to that data to others was also against many licenses and terms of services, but OpenAI did those anyway.