Super cool project. This is from the author of whisper.cpp, which enables highly accurate real-time audio transcription on the M1/M2:
Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
11–20 of 298 posts
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#12Don't know anything about ML can someone can explain me what is this hype about?
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#13George Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch: https://github.com/geohot/tinygrad/tree/llama The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.
Can you link to the Twitch stream VoD?
https://www.twitch.tv/georgehotz/videos?filter=archives&sort...
although, there is a VOD channel on YT that might be better.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#14Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#15George Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch: https://github.com/geohot/tinygrad/tree/llama The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.
Both are very impressive. A nice thing about Gerganov's implementation is that it is written in almost pure C. Arguably easier for deployment.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#16Earlier quoted context omitted.
https://www.twitch.tv/georgehotz/videos?filter=archives&sort... although, there is a VOD channel on YT that might be better.
why does it say video unavailable?
Hmm. I'm not sure. Presumably this will be updated soon: https://www.youtube.com/@geohotarchive/videos
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#17Earlier quoted context omitted.
https://www.twitch.tv/georgehotz/videos?filter=archives&sort... although, there is a VOD channel on YT that might be better.
why does it say video unavailable?
I'm pretty sure he only makes his past twitch streams available to subscribers. It's weird they are available on Youtube, maybe he doesn't know that's something he can change.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#18Don't know anything about ML can someone can explain me what is this hype about?
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#19I tried llama 14b model by using one of online tools (mentioned in one of other hn comments, it claimed to use that model, but cannot be 100% sure) and I was very disappointed with results. I tried to ask it fairly simple question: to write regex validating email address, tried this 2 times and got responses: "what is your question" and "please wait..." so it just doged response. in contrast chatgpt was able to provi…
sillysaurusx made some posts about tweaks that greatly improved output for him. Maybe they would help you:
"""
Fixed the llama sampler. After turning off top_p, adding top_k 40, setting temp to 0.7, and adding a repetition penalty of 1/0.85, llama 7B is looking nice.
"""
https://twitter.com/theshawwn/status/1632569215348531201?cxt...
https://twitter.com/theshawwn/status/1633456289639542789?cxt...
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#20Don't know anything about ML can someone can explain me what is this hype about?
This is an advanced language model that can now run quickly on consumer grade hardware. You used to need thousands of dollars of GPUs to run a model as sophisticated as this - now it can be done on a laptop,