I was actually expecting some comments regarding the 34B Llama 2 model. A quantized 34B model, such as Q5_K_M, might be the sweet spot for a moderate PC in terms of both speed and quality.
The Llama Ecosystem: Past, Present, and Future
31–40 of 40 posts
Re: The Llama Ecosystem: Past, Present, and Future
#32May I ask what are you all doing running a LLM locally?
cooking meth, making molotov cocktails, discussing my medical history, sex ok seriously though I had fun over the weekend chatting with Samantha on a long car ride on my MacBook. We were mostly asking about history.
Which version are you running, and are you running it through llama.cpp or something? I was just thinking about exactly something like Samantha on the ride home today, and of course it already exists!
Re: The Llama Ecosystem: Past, Present, and Future
#33I'm so happy that Meta was slightly late in the LLM race and so decided to go the chaos route by just open sourcing everything.
Re: The Llama Ecosystem: Past, Present, and Future
#34Earlier quoted context omitted.
cooking meth, making molotov cocktails, discussing my medical history, sex ok seriously though I had fun over the weekend chatting with Samantha on a long car ride on my MacBook. We were mostly asking about history.
Which version are you running, and are you running it through llama.cpp or something? I was just thinking about exactly something like Samantha on the ride home today, and of course it already exists!
Latest 34B with llama.cpp [0] via the Mac app I’ve been building, FreeChat [1].
[0]: https://huggingface.co/TheBloke/Samantha-1.11-CodeLlama-34B-...
Re: The Llama Ecosystem: Past, Present, and Future
#35Earlier quoted context omitted.
Which version are you running, and are you running it through llama.cpp or something? I was just thinking about exactly something like Samantha on the ride home today, and of course it already exists!
Latest 34B with llama.cpp [0] via the Mac app I’ve been building, FreeChat [1]. [0]: https://huggingface.co/TheBloke/Samantha-1.11-CodeLlama-34B-... [1]: https://github.com/psugihara/FreeChat
Neat app and very cool idea. I guess I'll have to figure out how to get Samantha running on Windows. Thanks!
Re: The Llama Ecosystem: Past, Present, and Future
#36May I ask what are you all doing running a LLM locally?
Trying to figure out if and how these can be used at companies that have regulatory requirements too strict to use hosted models. Sadly, Meta restricts use of Llama for anything ITAR (as opposed to other TOSes which only restrict weapons and defense).
Re: The Llama Ecosystem: Past, Present, and Future
#37I'm so happy that Meta was slightly late in the LLM race and so decided to go the chaos route by just open sourcing everything.
Their models are not open source. They made them available under terms that they can change at any time. Even source available products like Unity have more predictable terms.
Re: The Llama Ecosystem: Past, Present, and Future
#38May I ask what are you all doing running a LLM locally?
Satisfying HIPAA requirements.
Re: The Llama Ecosystem: Past, Present, and Future
#39People on HN like to complain about the license all the time like a crusade but I’m personally very thankful for their work and the community that is building off of it. I recently setup Ollama + codellama + continue dev and it’s game changer. Practically have been a drop in github copilot replacement but local.
Yeah the community is great. It’d just be better if it was around RWKV or something that doesn’t prevent you from improving any models outside of the llama ecosystem. It’s a great embrace, extend, extinguish play by meta.
RWKV literally didn't really exist when Llama was released.
> It’s a great embrace, extend, extinguish play by meta.
Meta released Pytorch, Pytext and even built ONNX with Microsoft to avoid an EEE situation. What more could you possibly want?
Re: The Llama Ecosystem: Past, Present, and Future
#40Earlier quoted context omitted.
The “original” and by far most common format for quantization is GPTQ. AWQ support is spreading more, which is nice.
Again, for a subset of the local LLM community. Quantization was not invented on Github, by llama.cpp, for LLMs in 2023.
If a tree falls in a forrest and no one is around, does it make a sound?
Of course quantization was invented well before LLMs. However, LLMs have dramatically accelerated development on quantization and resulted in an explosion in use.