What's new in Llama 2 and how to run it locally
agi-sphere.com
What's new in Llama 2 and how to run it locally
1–10 of 85 posts
Re: What's new in Llama 2 and how to run it locally
#2I am partial to Koboldcpp over text gen UI for a number of reasons.
...But I am also a bit out of the loop. For instance, I have not kept up with the CFG/negative prompt or grammar implementations in the UIs.
Re: What's new in Llama 2 and how to run it locally
#3If you want to try Llama 2 on a Mac and have Homebrew (or Python/pip) you may find my LLM CLI tool interesting: https://simonwillison.net/2023/Aug/1/llama-2-mac/
Re: What's new in Llama 2 and how to run it locally
#4If you want to try Llama 2 on a Mac and have Homebrew (or Python/pip) you may find my LLM CLI tool interesting: https://simonwillison.net/2023/Aug/1/llama-2-mac/
What's the inference time without gpu?
Re: What's new in Llama 2 and how to run it locally
#5In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models.
The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of people innovating and debating approaches, lots of projects popping up, so much energy!
Re: What's new in Llama 2 and how to run it locally
#6If you want to try Llama 2 on a Mac and have Homebrew (or Python/pip) you may find my LLM CLI tool interesting: https://simonwillison.net/2023/Aug/1/llama-2-mac/
What's the inference time without gpu?
It might the time mentioned at the bottom of the page since the author isn't sure that the GPU is being used:
>How to speed this up—right now my Llama prompts often take 20+ seconds to complete.
Re: What's new in Llama 2 and how to run it locally
#7In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models. The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of…
> The enthusiasm around it reminds me of Javascript wars 10 years ago... so much energy!
I kind of have the same feeling as well. With all this energy it's really hard to keep up with all new ideas, implementations, frameworks and services.
Really excited for what this will bring us the next coming years
Re: What's new in Llama 2 and how to run it locally
#8In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models. The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of…
> “The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago”
Hmm. If LLMs turned out like JS frameworks, that would mean that in ten years people will be saying:
“Maybe we don’t really need all this expensive ceremony, honestly this could be done with vanilla if/else heuristics…?”
Re: What's new in Llama 2 and how to run it locally
#9In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models. The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of…
It looks like it will always be a war like Android VS iOS only now it's with AI models.
Re: What's new in Llama 2 and how to run it locally
#10Did anybody try the Llama 2 model with languages other than English? The paper notes that it works best with English and the amount of training data for other languages is only a fraction. Which likely would make it unusable for me..
See table 10 (page 22) of the whitepaper for the numbers: https://ai.meta.com/research/publications/llama-2-open-found...
Are there other downloadable models which can be used in a multilingual environment that people here are aware of?