Any more details? I'm guessing they're leveraging the NPU in the pixel?
I think they are using llama.cpp without any NPU/TPU patches. By default it only runs on CPU with support for various SIMD extensions.
LLaMa running at 5 tokens/second on a Pixel 6
11–20 of 79 posts
Re: LLaMa running at 5 tokens/second on a Pixel 6
#12Any more details? I'm guessing they're leveraging the NPU in the pixel?
It uses the ARM NEON extensions to the instruction set for SIMD (as far as I understand).
Re: LLaMa running at 5 tokens/second on a Pixel 6
#13I'm waiting until it runs on my C64...
[deleted]
Re: LLaMa running at 5 tokens/second on a Pixel 6
#14Did anyone get this to run on an iPhone or in a browser yet?
Most iphone have only 4GB RAM (and even latest iphone 14 has only 6GB RAM). Pixel 6 has 8GB RAM. But bigger issue is on iOS still OS limits how much RAM your app can use and might kill your app.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#15This would be useful for predictive text. That's exactly what LLMs are actually built for.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#16Does this in theory mean it should be relatively easy to port to coral TPU?
All their tensor/math magic seems to happen in https://github.com/ggerganov/llama.cpp/blob/master/ggml.h .
So maybe if you implement the ggml.c with tensorflow/libcoral - you'd have a chance.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#17Earlier quoted context omitted.
From the video output seems fine. But if it is a trimmed version, it is wong to call it LLaMa.
It's nonsensical, celeb announces they're going to rehab and notes it (?) is an issue affecting all women, at least, earlier today (??), they also noted it wasn't drugs or alcohol this time, but, a life (???)
What are you comparing it to? Without instruction tuning and a two character prompt "He" I am not sure why you would expect it to perform any better.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#18It is not really llama, it is llama quantized to 4bit. Not even the quality of original 7B. I could also quantize it to 1 bit and claim it runs on my RPI3.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#19It is not really llama, it is llama quantized to 4bit. Not even the quality of original 7B. I could also quantize it to 1 bit and claim it runs on my RPI3.
The 4 bit quantization performs well, though. Does your 1 bit version?
Re: LLaMa running at 5 tokens/second on a Pixel 6
#20It is not really llama, it is llama quantized to 4bit. Not even the quality of original 7B. I could also quantize it to 1 bit and claim it runs on my RPI3.
The quantization to four hits doesn’t have that much effect on the output. 1 bit might not either, but someone would need to do some testing before making the claim that “1 bit … runs on my RPI3” because “runs” is a bit overloaded to mean “runs and produces sensible output.” I think you’re missing that runs here has that overloading.