LLaMa running at 5 tokens/second on a Pixel 6
1–10 of 79 posts
Re: LLaMa running at 5 tokens/second on a Pixel 6
#2Until it thermally throttles 40 seconds later. But yeah, it's really cool how many platforms the vanilla code in llama.cpp can be easily compiled on. And somehow I doubt they did the quantization step on the Pixel itself. My favorite was the person who did it on the rpi4. I know a guy working on getting it going on rpi3 but the ARM7/8 mixing , NEON support, and 64 bit ARM intrinsics are apparently non-trivial to convert.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#3This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#4Any more details? I'm guessing they're leveraging the NPU in the pixel?
Re: LLaMa running at 5 tokens/second on a Pixel 6
#5This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.
From the video output seems fine.
But if it is a trimmed version, it is wong to call it LLaMa.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#6I'm waiting until it runs on my C64...
Re: LLaMa running at 5 tokens/second on a Pixel 6
#7Did anyone get this to run on an iPhone or in a browser yet?
Re: LLaMa running at 5 tokens/second on a Pixel 6
#8This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.
From the video output seems fine. But if it is a trimmed version, it is wong to call it LLaMa.
It's nonsensical, celeb announces they're going to rehab and notes it (?) is an issue affecting all women, at least, earlier today (??), they also noted it wasn't drugs or alcohol this time, but, a life (???)
Re: LLaMa running at 5 tokens/second on a Pixel 6
#9Does this in theory mean it should be relatively easy to port to coral TPU?
Re: LLaMa running at 5 tokens/second on a Pixel 6
#10Until it thermally throttles 40 seconds later. But yeah, it's really cool how many platforms the vanilla code in llama.cpp can be easily compiled on. And somehow I doubt they did the quantization step on the Pixel itself. My favorite was the person who did it on the rpi4. I know a guy working on getting it going on rpi3 but the ARM7/8 mixing , NEON support, and 64 bit ARM intrinsics are apparently non-trivial to conv…
>And somehow I doubt they did the quantization step on the Pixel itself
You're probably right (because why would they?) but I don't see any reason they couldn't have done this if they wanted to.