What's the cheapest way to run e.g. LLaMa2-13B and have it served as an API?
I've tried Inference Endpoints and Replicate, but both would cost more than just using the OpenAI offering.
What's new in Llama 2 and how to run it locally
11–20 of 85 posts
Re: What's new in Llama 2 and how to run it locally
#12Just set things up locally last night. If you're a developer, llama.cpp was a pleasure to build and run. I wanted to run the weights from Meta and couldn't figure out text generation web ui. It seemed that one was optimized for grabbing something off HuggingFace.
Running on a 3090. The 13b chat model quantized to fp8 is giving about 42 tok/s.
Re: What's new in Llama 2 and how to run it locally
#13I've only used the 13b model and I'd say it was as good as GPT-3 (not GPT-4). It's amazing, and I only have a laptop to run it locally on so 13b is as good as I can do.
Re: What's new in Llama 2 and how to run it locally
#14Is there an overview somewhere how much RAM is needed for which model? Is it possible at all to run 4bit 70B on CPU and RAM?
Re: What's new in Llama 2 and how to run it locally
#15Did anybody try the Llama 2 model with languages other than English? The paper notes that it works best with English and the amount of training data for other languages is only a fraction. Which likely would make it unusable for me.. See table 10 (page 22) of the whitepaper for the numbers: https://ai.meta.com/research/publications/llama-2-open-found... Are there other downloadable models which can be used in a multi…
At Mirage Studio we have successfully finetuned Llama 2 7B on a Dutch dataset to get it to output Dutch in a coherent way: https://huggingface.co/Mirage-Studio/llama-gaan-2-7b-chat-hf...
Re: What's new in Llama 2 and how to run it locally
#16Is there an overview somewhere how much RAM is needed for which model? Is it possible at all to run 4bit 70B on CPU and RAM?
Yes, I run the 4bit, 70B on a threadripper 32 core using llama.cpp. It uses around 37Gb of RAM and I get 4-5 tokens per second (slow but usable). Core usage is very uneven with many cores at 0% so maybe there's some more performance to be had in the future. Sometimes it gets stuck for a few seconds and then recovers.
It gives very detailed answers to coding questions and tasks just like GPT4 does (though I did not do a proper comparison).
The 13b uses 13Gb with 27 tokens per second the 7b uses 0.5Gb and I get 39 tokens per second on this machine.Both produce interesting results even for CUDA code generation, for example.
Re: What's new in Llama 2 and how to run it locally
#17In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models. The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of…
> “The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago” Hmm. If LLMs turned out like JS frameworks, that would mean that in ten years people will be saying: “Maybe we don’t really need all this expensive ceremony, honestly this could be done with vanilla if/else heuristics…?”
I can imagine a bloated world where 500B param models are used for tasks where 7B param modes perform adequately.
At that time, there could be complaints on hacker news about messaging apps with autocomplete models that take up gigabytes.
Re: What's new in Llama 2 and how to run it locally
#18In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models. The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of…
> The enthusiasm around it reminds me of Javascript wars 10 years ago... so much energy! I kind of have the same feeling as well. With all this energy it's really hard to keep up with all new ideas, implementations, frameworks and services. Really excited for what this will bring us the next coming years
> With all this energy it's really hard to keep up with all new ideas, implementations, frameworks and services.
The majority of them are mostly irrelevant. You just need to figure out which.
Re: What's new in Llama 2 and how to run it locally
#19Did anybody try the Llama 2 model with languages other than English? The paper notes that it works best with English and the amount of training data for other languages is only a fraction. Which likely would make it unusable for me.. See table 10 (page 22) of the whitepaper for the numbers: https://ai.meta.com/research/publications/llama-2-open-found... Are there other downloadable models which can be used in a multi…
Both 1 and 2 versions are good enough in Russian even for some real use in production environments. But you should tolerate some crazy / weird typos and mis-wording they'll produce. Russian is complicated.
Re: What's new in Llama 2 and how to run it locally
#20Earlier quoted context omitted.
> “The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago” Hmm. If LLMs turned out like JS frameworks, that would mean that in ten years people will be saying: “Maybe we don’t really need all this expensive ceremony, honestly this could be done with vanilla if/else heuristics…?”
I can imagine a bloated world where 500B param models are used for tasks where 7B param modes perform adequately. At that time, there could be complaints on hacker news about messaging apps with autocomplete models that take up gigabytes.
I could see a world where people turn to an LLM for a task where the hand-rolled solution could be a simple state machine or some nested switch statements.
The irony would be that the LLM could write you that code, but if you don’t know to ask…