Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
1–10 of 298 posts
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#2The performance on Macbook with M1 Pro is said to be 20 tokens/s
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#3Isn't using any of the AS "ML" coprocessor/extensions/whatever, so it's just normal simd.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#4I tried llama 14b model by using one of online tools (mentioned in one of other hn comments, it claimed to use that model, but cannot be 100% sure)
and I was very disappointed with results. I tried to ask it fairly simple question: to write regex validating email address, tried this 2 times and got responses: "what is your question" and "please wait..." so it just doged response. in contrast chatgpt was able to provide me with simple regex and also RFC compliant version when asked.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#5George Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch:
https://github.com/geohot/tinygrad/tree/llama
The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#6George Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch: https://github.com/geohot/tinygrad/tree/llama The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.
that is so impressive!
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#7I tried llama 14b model by using one of online tools (mentioned in one of other hn comments, it claimed to use that model, but cannot be 100% sure) and I was very disappointed with results. I tried to ask it fairly simple question: to write regex validating email address, tried this 2 times and got responses: "what is your question" and "please wait..." so it just doged response. in contrast chatgpt was able to provi…
Outside of it being possibly a smaller model, the prompts should be different as llama hasn't been trained to take instructions so it would I think need to be framed more as "A regex for emails is " and let it complete the rest. Just in case you have tried the exact same prompt.
Also those messages sound suspiciously like error messages from a system around it.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#8I tried llama 14b model by using one of online tools (mentioned in one of other hn comments, it claimed to use that model, but cannot be 100% sure) and I was very disappointed with results. I tried to ask it fairly simple question: to write regex validating email address, tried this 2 times and got responses: "what is your question" and "please wait..." so it just doged response. in contrast chatgpt was able to provi…
Llama is neither instruction finetuned nor human reinforcement learning ranked. It‘s a base model like the original GPT3 that needs much more careful prompting and few-shot reasoning to produce similar results.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#9George Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch: https://github.com/geohot/tinygrad/tree/llama The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.
Can you link to the Twitch stream VoD?
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#10Thanks for doing this, nice work!
Please add some sort of license.