This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.
From the video output seems fine. But if it is a trimmed version, it is wong to call it LLaMa.
Could call it Slim LLaMa
LLaMa running at 5 tokens/second on a Pixel 6
31–40 of 79 posts
Re: LLaMa running at 5 tokens/second on a Pixel 6
#32Earlier quoted context omitted.
Do you have the numbers? I suspect is is way worse. Original llama.cpp authors never measure any numbers as well.
The python implementation[1] ran some tests using the same quantization algorithm as llama.cpp (4 bit RTN). 1: https://github.com/qwopqwop200/GPTQ-for-LLaMa
Great thanks a lot.
So we have numbers on PTB original perplexity 8.79 quantized 9.68, already 10% worse. And PPL reported per token I suppose? Because word PPL for PTB must be around 20, not less than 10.
Any numbers on more complex tasks then? like QA?
Re: LLaMa running at 5 tokens/second on a Pixel 6
#33Until it thermally throttles 40 seconds later. But yeah, it's really cool how many platforms the vanilla code in llama.cpp can be easily compiled on. And somehow I doubt they did the quantization step on the Pixel itself. My favorite was the person who did it on the rpi4. I know a guy working on getting it going on rpi3 but the ARM7/8 mixing , NEON support, and 64 bit ARM intrinsics are apparently non-trivial to conv…
I'm the original tweet author.
Currently typing this from my Pixel after running it countless times :)
Re: LLaMa running at 5 tokens/second on a Pixel 6
#34It is not really llama, it is llama quantized to 4bit. Not even the quality of original 7B. I could also quantize it to 1 bit and claim it runs on my RPI3.
i haven't noticed 4bit quantization affecting the quality of LLaMA-7B, it produces very coherent outputs, the trick is having a good example in your prompt so it has a good idea of what's expected of it.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#35Earlier quoted context omitted.
Most iphone have only 4GB RAM (and even latest iphone 14 has only 6GB RAM). Pixel 6 has 8GB RAM. But bigger issue is on iOS still OS limits how much RAM your app can use and might kill your app.
I'm still amazed that Apple invests so much into every other bit of hardware on a high end phone, yet always gives you the bare minimum amount of RAM they can get away with. There are so many use cases (like this) that require more RAM. And even if a use case doesn't theoretically require more RAM, getting a developer to dedicate time to optimizing RAM is time taken away from making a wonderful app.
> I'm still amazed that Apple invests so much into every other bit of hardware on a high end phone, yet always gives you the bare minimum amount of RAM they can get away with.
Advanced hardware makes bullet points on advertising to sell the device; giving the bare minimum of RAM accelerates the device planned obsolescence, so that user will be forced to upgrade sooner to the next model.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#36Earlier quoted context omitted.
It's nonsensical, celeb announces they're going to rehab and notes it (?) is an issue affecting all women, at least, earlier today (??), they also noted it wasn't drugs or alcohol this time, but, a life (???)
What are you comparing it to? Without instruction tuning and a two character prompt "He" I am not sure why you would expect it to perform any better.
I was replying to a comment that said it “seems fine.”
It does not seem fine.
It is incomprehensible and doesn’t match the results I’ve seen from 7B through 65B.
It is true that RLHF could improve it, and perhaps then this severe of optimization will seem fine.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#37Earlier quoted context omitted.
I'm still amazed that Apple invests so much into every other bit of hardware on a high end phone, yet always gives you the bare minimum amount of RAM they can get away with. There are so many use cases (like this) that require more RAM. And even if a use case doesn't theoretically require more RAM, getting a developer to dedicate time to optimizing RAM is time taken away from making a wonderful app.
> I'm still amazed that Apple invests so much into every other bit of hardware on a high end phone, yet always gives you the bare minimum amount of RAM they can get away with. Advanced hardware makes bullet points on advertising to sell the device; giving the bare minimum of RAM accelerates the device planned obsolescence, so that user will be forced to upgrade sooner to the next model.
How is that an iPhone 7 is completely current vs give me a branded Android from the same year of release that has even security updates, not even features.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#38Until it thermally throttles 40 seconds later. But yeah, it's really cool how many platforms the vanilla code in llama.cpp can be easily compiled on. And somehow I doubt they did the quantization step on the Pixel itself. My favorite was the person who did it on the rpi4. I know a guy working on getting it going on rpi3 but the ARM7/8 mixing , NEON support, and 64 bit ARM intrinsics are apparently non-trivial to conv…
>And somehow I doubt they did the quantization step on the Pixel itself You're probably right (because why would they?) but I don't see any reason they couldn't have done this if they wanted to.
I would have tried it but I didn't have enough storage on my phone to hold both the original and quantized weights.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#39This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.
That's why Alpaca is so exciting: it instruction-tunes LLaMA to the point that even the tiny 7B model (the one that fits on a phone) produces useful output: https://simonwillison.net/2023/Mar/13/alpaca/
Re: LLaMa running at 5 tokens/second on a Pixel 6
#40It is not really llama, it is llama quantized to 4bit. Not even the quality of original 7B. I could also quantize it to 1 bit and claim it runs on my RPI3.
The 4 bit quantization performs well, though. Does your 1 bit version?
1 bit will mathematically be guaranteed to be more efficient for performance-per-parameter so to me it is a pretty clear eventuality one day, but I think also the relative performance % will likely tank still. Impressed honestly that it held so well at 4 bit tbh, I thought personally that 8 bit was the ceiling.
However I can see fractional bits (via binary representations) and larger models happening first before that compression step.
And then we have the sub-bit range..... ;DDDD