Can someone create a wasm with typescript so that i can run this in the browser
Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
231–240 of 298 posts
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#232I got this working on my 64GB M2 MacBook Pro! Wrote detailed notes here for anyone else who wants to try this: https://til.simonwillison.net/llms/llama-7b-m2
I have the 64GB M2 Max I will have to try this out, its amazing that this laptop can do things my high end desktop can't do, Apple really made some good architectural choices with apple silicon.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#233If you are interested in implementing LLaMA yourself or learning, I noticed that the reference code by Facebook is one of the cleaner, easier to read ML code I've seen in a while. https://github.com/facebookresearch/llama/blob/main/llama/mo... It's about 200 lines long. You probably do need a bit of knowledge to understand what you are reading but I was pleasantly surprised. For example in comparison, StableDiffusion…
Very nice post, good lead. It makes me curious... I wonder what LLaMA would look like implemented upon the newly release OpenXLA[1]! Is that even a sensible ask? I feel like it could potentially be an informative exercise, that would aid in the understanding of the landscape of tooling. [1] https://opensource.googleblog.com/2023/03/openxla-is-ready-t... https://news.ycombinator.com/item?id=35078410
We have it running as part of SHARK (which is built on IREE). https://github.com/nod-ai/SHARK/tree/main/shark/examples/sha...
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#234Earlier quoted context omitted.
You can buy a prebuilt pc with a 4090 for less - which is significantly more powerful but still in the 3xxx$. You could go cheaper with a 3090 which has the same vram and it's just slower. I think the best combo is a serious Nvidia pc for AI + a cheap MacBook air for portability.
There's even an Asus laptop with a 4099 for 3999$
Laptop 4090 is desktop 4080
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#235Earlier quoted context omitted.
Is there something I'm missing? ChatLlama doesn't reference any human feedback datasets. > You can of course also cross train it using actual ChatGPT. You mean train it on ChatGPT's output? That's against OpenAI's terms of service.
> You mean train it on ChatGPT's output? That's against OpenAI's terms of service. Oh no, someone call the internet police. I'm sure scraping tons and tons of images and web data to train DALLE and GPT and then selling access to that data to others was also against many licenses and terms of services, but OpenAI did those anyway.
None of these AIs were created ethically. At the very least we can make sure these huge models don’t solely belong to monopolistic tech companies and democratize their power.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#236Earlier quoted context omitted.
what model are you using? edit: i mean 6B, 13b or 30B?
It'll be 7B they're referring to, on my M1 Max 32GB with a 4000 token output request I get 67ms/token on 7B (4bit) and 154ms/token on 13B (4bit)... I've made a tweak to the code to increase the context size but it doesn't seem to change perf. main: mem per token = 22357508 bytes main: load time = 2741.67 ms main: sample time = 156.68 ms main: predict time = 11399.12 ms / 154.04 ms per token main: total time = 14914.3…
This was generating 2000 tokens, so it seems to get slightly faster on longer generation runs maybe?
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#237Earlier quoted context omitted.
I've had difficulty obtaining useful results from the smaller (7B-sized) models. The issue lies in the content, not the speed. If you could stream the text-to-speech, the speed alone would be satisfactory.
You're right I overestimated how fast we talk!
Some rules of thumb I use for estimating this kind of stuff
100wpm: Max typing speed
200wpm: Max speaking speed
300wpm: Max listening speed, max reading speed with subvocalisation
900wpm: Max reading speed without subvocalisation
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#238Earlier quoted context omitted.
Yes (I just don't have that much ram) I have a separate branch that streams weights from ram - at which point I think I was only seeing negligible performance loss compared to storing the weights in vram. The bottleneck was compute, not GPU bandwidth.
The 65B model only needs just over 32GB of VRAM to run. It does not need system RAM to run/use if you use pre-quantized weights which you can find many places already. No need to quantize yourself (besides it takes almost a day to do 4bit GPTQ quantization on 3xA6000).
Quantizing is a lossy process, you can't really claim to be running the 65B model llama at that point (though the 65b qgpt-llama does look like it might be very useful)
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#239A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…
But wont it be that in real life no one would want to run a voice command which consumes lot of CPU and battery as opposed to making a network call to a service which has this model hosted ? Agreed that this can always be improved and hardware can get more efficient and better to but at the end of the day, would it ever be better then an API call ?
"There is No Reason for Any Individual To Have a Computer in Their Home"
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#240A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…
> All of the specs indicate an iPhone 14 Pro could achieve similar throughput to an M1 MacBook Pro. Battery capacity and thermals are different and might be problematic. The phone might throttle performance earlier. > it looks like we have all the pieces needed to make a ChatGPT-level assistant operate entirely through voice and run entirely on your phone. As a demo, yes, but would loading the model be fast enough fo…
> we would have to get rid of the tendency of these models [...] reply with something a bit more sexist/racist/
If you don't want it to be racist, don't say racist things to it. Also, it'll be fairly clear where the racism came from - like a parrot and their owner.
AIs that can tweet, like MS Tay, and that remote-work chatbot, get a lot of attention when they melt down. Private AIs on your phone don't seem like they'll caise any concern with the phone-using public.
I think we'll appreciate the benefits more than we'll mind that others can make it say dirty words.