21–30 of 79 posts
It is not really llama, it is llama quantized to 4bit. Not even the quality of original 7B. I could also quantize it to 1 bit and claim it runs on my RPI3.
The 4 bit quantization performs well, though. Does your 1 bit version?
Earlier quoted context omitted.
From the video output seems fine. But if it is a trimmed version, it is wong to call it LLaMa.
It's nonsensical, celeb announces they're going to rehab and notes it (?) is an issue affecting all women, at least, earlier today (??), they also noted it wasn't drugs or alcohol this time, but, a life (???)
Earlier quoted context omitted.
The 4 bit quantization performs well, though. Does your 1 bit version?
Do you have the numbers? I suspect is is way worse. Original llama.cpp authors never measure any numbers as well.
Earlier quoted context omitted.
The 4 bit quantization performs well, though. Does your 1 bit version?
Do you have the numbers? I suspect is is way worse. Original llama.cpp authors never measure any numbers as well.
This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.
Earlier quoted context omitted.
The 4 bit quantization performs well, though. Does your 1 bit version?
Do you have the numbers? I suspect is is way worse. Original llama.cpp authors never measure any numbers as well.
Did anyone get this to run on an iPhone or in a browser yet?
Most iphone have only 4GB RAM (and even latest iphone 14 has only 6GB RAM). Pixel 6 has 8GB RAM. But bigger issue is on iOS still OS limits how much RAM your app can use and might kill your app.
There are so many use cases (like this) that require more RAM. And even if a use case doesn't theoretically require more RAM, getting a developer to dedicate time to optimizing RAM is time taken away from making a wonderful app.
Does this in theory mean it should be relatively easy to port to coral TPU?
Earlier quoted context omitted.
Do you have the numbers? I suspect is is way worse. Original llama.cpp authors never measure any numbers as well.
They're using GTPQ -- here you go: https://arxiv.org/abs/2210.17323 . The authors benchmarked two families of models over a wide range of numbers of params.
It is not really llama, it is llama quantized to 4bit. Not even the quality of original 7B. I could also quantize it to 1 bit and claim it runs on my RPI3.