[flagged]
Aren't we all just tired of arguing the same points?
Quantized Llama models with increased speed and a reduced memory footprint
21–30 of 128 posts
Re: Quantized Llama models with increased speed and a reduced memory footprint
#22Anyone know a nice iOS app to run these locally?
I've been using PocketGPT.
Re: Quantized Llama models with increased speed and a reduced memory footprint
#23Does anyone know why the most common method to speed up inference time is quantization? I keep hearing about all sorts of new methods but nearly none of them is implemented in practice (except for flash attention).
Re: Quantized Llama models with increased speed and a reduced memory footprint
#24[flagged]
What kind of fundamental discussion are you hoping to see under an article about an iterative improvement to a known model?
"AI will destroy the world"? "AI is great and will save humanity"? If you're seriously missing that, there's really enough platforms (and articles for more fundamental announcements/propositions on this one) where you can have these.
Re: Quantized Llama models with increased speed and a reduced memory footprint
#25Earlier quoted context omitted.
I don’t get the comment. For one I’m excited for developments in the field. Not afraid it will “replace me” as technology has replaced me multiple times over. I’m looking towards working with these models more and more.
No, I meant that a lot of us are working very fast on a pre-launch product, implementing some cutting edge ideas using e.g. the incredible speedup in a small fast inference model like quantized 3B in combination with other tools, and I think there's quite a bit of paranoia out there that someone else will beat you to market. And so not a lot of sharing going on in the comments. At least not as much as previously, and…
This thread attracts a smaller audience than, say, a new version of ChatGPT.
Re: Quantized Llama models with increased speed and a reduced memory footprint
#26May I ask if anyone has successfully used 1B and 3B models in production and if yes, in what use cases? I seem to be failing even in seemingly simpler tasks such as word translation or zero-shot classification. For example, they seem to not care about instructions to only write a response and no explanation, thus making it impossible to use them in a pipeline :/
3B models are perfectly capable, I've had great luck with Phi 3.5.
> For example, they seem to not care about instructions to only write a response and no explanation
You need to use tools to force the model to adhere to a schema. Or you can learn to parse out the part of the response you want, both work.
You'll also need to make good use of robust examples in your initial prompt, and give lots of examples of how you want the output to look. (Yes this quickly burns up the limited context length!)
Finally, embrace the fact that these models are tuned for chat, so the more conversational you make the back and forth the less you are stretching the models abilities.
I wrote a very small blog post at https://meanderingthoughts.hashnode.dev/unlock-the-full-pote... explaining some of this.
Re: Quantized Llama models with increased speed and a reduced memory footprint
#27[deleted]
Re: Quantized Llama models with increased speed and a reduced memory footprint
#28Any pointers no how to finetune this on my dataset and package and run it in my swift ios app?
Re: Quantized Llama models with increased speed and a reduced memory footprint
#29May I ask if anyone has successfully used 1B and 3B models in production and if yes, in what use cases? I seem to be failing even in seemingly simpler tasks such as word translation or zero-shot classification. For example, they seem to not care about instructions to only write a response and no explanation, thus making it impossible to use them in a pipeline :/
> For example, they seem to not care about instructions to only write a response and no explanation, thus making it impossible to use them in a pipeline
I was doing some local tidying up of recording transcripts, using a fairly long system prompt, and I saw the same behaviour you mention if the transcript I was passing in was too long -- batching it up to make sure to be under the max length prevented this.
Might not be what's happening in your case, but I mention it because it wasn't immediately obvious to me when I first saw the behaviour.
Re: Quantized Llama models with increased speed and a reduced memory footprint
#30May I ask if anyone has successfully used 1B and 3B models in production and if yes, in what use cases? I seem to be failing even in seemingly simpler tasks such as word translation or zero-shot classification. For example, they seem to not care about instructions to only write a response and no explanation, thus making it impossible to use them in a pipeline :/
Not in production, but I've used a 3B model to test a local LLM application I'm working on. I needed a full end-to-end request/response and it's a lot faster asking a 3B model than an 8B model. I could setup a test harness and replay the responses... but this was a lot simpler.
If for testing then why not just mock the whole thing for ultimate performance ... ?