Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's new in Llama 2 and how to run it locally

agi-sphere.com

Post reply on HN
Print view View original

31–40 of 85 posts

Re: What's new in Llama 2 and how to run it locally

#31

Post by antupis »

pavlov wrote: post #8
visarga wrote: post #5

In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models. The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of…

> “The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago” Hmm. If LLMs turned out like JS frameworks, that would mean that in ten years people will be saying: “Maybe we don’t really need all this expensive ceremony, honestly this could be done with vanilla if/else heuristics…?”

Well these models at least tell in name what they do eg llama-2-70b-Guanaco-QLoRA-fp16.

Re: What's new in Llama 2 and how to run it locally

#32

Post by nsomaru »

MrYellowP wrote: post #25
visarga wrote: post #5

In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models. The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of…

> information extraction task I do that with orca-mini-3b in ggml format and it's pretty good at it, at twice the speed. Of all the LLMs I've tried, this one gave me the best results. It just requires a properly written prompt.

Could you elaborate on the prompting strategies you have used that are more effective?

Re: What's new in Llama 2 and how to run it locally

#33

Post by jmorgan »

If you're looking to run Llama 2 locally via a CLI or REST API (vs the web ui this article highlights), there's an open-source project some folks and I have been working on over the last few weeks: https://github.com/jmorganca/ollama

More projects in this space:

- llama.cpp which is a fast, low level runner (with bindings in several languages)

- llm by Simon Willison which supports different backends and has a really elegant CLI interface

- The MLC.ai and Apache TVM projects

Previous discussion on HN that might be helpful from an article by the great folks at replicate: https://news.ycombinator.com/item?id=36865495

Re: What's new in Llama 2 and how to run it locally

#35

Post by lgrammel »

ktaube wrote: post #11

What's the cheapest way to run e.g. LLaMa2-13B and have it served as an API? I've tried Inference Endpoints and Replicate, but both would cost more than just using the OpenAI offering.

llama.cpp has a server with a REST API that you can use: https://github.com/ggerganov/llama.cpp/tree/master/examples/...

Re: What's new in Llama 2 and how to run it locally

#37

Post by cfn »

MrYellowP wrote: post #26
cfn wrote: post #16

Earlier quoted context omitted.

Yes, I run the 4bit, 70B on a threadripper 32 core using llama.cpp. It uses around 37Gb of RAM and I get 4-5 tokens per second (slow but usable). Core usage is very uneven with many cores at 0% so maybe there's some more performance to be had in the future. Sometimes it gets stuck for a few seconds and then recovers. It gives very detailed answers to coding questions and tasks just like GPT4 does (though I did not do…

How does the 7B model use only 512 megabytes? That's not possible? Is it using mmap and concealing the actual memory usage?

I forgot to say I am using ggml models. This is what llama.cpp outputs when you start it:

main: build = 942 (4f6b60c) main: seed = 1691400051 llama.cpp: loading model from /media/z/models/TheBloke_Llama-2-7b-chat-GGML/llama-2-7b-chat.ggmlv3.q5_1.bin llama_model_load_internal: format = ggjt v3 (latest) llama_model_load_internal: n_vocab = 32000 llama_model_load_internal: n_ctx = 512 llama_model_load_internal: n_embd = 4096 llama_model_load_internal: n_mult = 256 llama_model_load_internal: n_head = 32 llama_model_load_internal: n_head_kv = 32 llama_model_load_internal: n_layer = 32 llama_model_load_internal: n_rot = 128 llama_model_load_internal: n_gqa = 1 llama_model_load_internal: rnorm_eps = 1.0e-05 llama_model_load_internal: n_ff = 11008 llama_model_load_internal: freq_base = 10000.0 llama_model_load_internal: freq_scale = 1 llama_model_load_internal: ftype = 9 (mostly Q5_1) llama_model_load_internal: model size = 7B llama_model_load_internal: ggml ctx size = 0.08 MB llama_model_load_internal: mem required = 4820.60 MB (+ 256.00 MB per state) llama_new_context_with_model: kv self size = 256.00 MB llama_new_context_with_model: compute buffer total size = 71.84 MB

You can see the memory required at 4820.60 MB (+ 256.00 MB per state). The process monitor (on Ubuntu) shows less than 400 Mb.

This is the command: ./main -eps 1e-5 -m /media/z/models/TheBloke_Llama-2-7b-chat-GGML/llama-2-7b-chat.ggmlv3.q5_1.bin -t 13 -p \ "[INST] >You are a helpful and concise assistant>Write a c++ function that calculates RMSE between two double lists using CUDA. Don't explain, just write out the code.[/INST]"

Re: What's new in Llama 2 and how to run it locally

#39

Post by cfn »

cypress66 wrote: post #30
cfn wrote: post #16

Earlier quoted context omitted.

Yes, I run the 4bit, 70B on a threadripper 32 core using llama.cpp. It uses around 37Gb of RAM and I get 4-5 tokens per second (slow but usable). Core usage is very uneven with many cores at 0% so maybe there's some more performance to be had in the future. Sometimes it gets stuck for a few seconds and then recovers. It gives very detailed answers to coding questions and tasks just like GPT4 does (though I did not do…

5 tokens/s on 70B 4bit seems really high for your setup.

This is the command:

./main -m /media/z/models/TheBloke_Llama-2-70B-Chat-GGML/llama-2-70b-chat.ggmlv3.q4_0.bin -gqa 8 -t 13 -p "The prompt..."

And this is the report at the end of the answer:

llama_print_timings: load time = 999.84 ms

llama_print_timings: sample time = 302.21 ms / 703 runs ( 0.43 ms per token, 2326.20 tokens per second)

llama_print_timings: prompt eval time = 69377.40 ms / 300 tokens ( 231.26 ms per token, 4.32 tokens per second)

llama_print_timings: eval time = 236017.69 ms / 701 runs ( 336.69 ms per token, 2.97 tokens per second)

llama_print_timings: total time = 305815.51 ms

The computer is a Ryzen threadripper pro 5975wx 32-cores × 64 with 256Gb of RAM. It also has a GPU but I checked with nvtop that nothing is being loaded to it.

Re: What's new in Llama 2 and how to run it locally

#40

Post by cfn »

Ambix wrote: post #22
cfn wrote: post #16

Earlier quoted context omitted.

Yes, I run the 4bit, 70B on a threadripper 32 core using llama.cpp. It uses around 37Gb of RAM and I get 4-5 tokens per second (slow but usable). Core usage is very uneven with many cores at 0% so maybe there's some more performance to be had in the future. Sometimes it gets stuck for a few seconds and then recovers. It gives very detailed answers to coding questions and tasks just like GPT4 does (though I did not do…

Try to use less cores. RAM bandwidth is real limiting factor there, so there always some sweet spot between CPU cores and RAM bandwidth for individual system. For example, I use only 6 cores from 10 on my M1 Pro laptop.

That is an interesting idea. Can you tell me what is the switch for number of cores?
Post reply on HN