Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's new in Llama 2 and how to run it locally

agi-sphere.com

Post reply on HN
Print view View original

71–80 of 85 posts

Re: What's new in Llama 2 and how to run it locally

#71

Post by lhl »

SOLAR_FIELDS wrote: post #54
cube2222 wrote: post #49

Earlier quoted context omitted.

In the Llama 2 paper benchmarks they did mention that Llama 2 is much worse at any kind of code generation than the OpenAI models, they were optimizing for conversational / natural language use-cases.

Interesting, what other openly licensed models are better at codegen? Or perhaps there is a version of llama 2 already fine tuned for codegen? There is starcoder but I had also not had great results with that one in my brief experiments

WizardCoder-15B (an evol-instruct starcoder fine-tune) is probably the best performing open model atm: https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder

Re: What's new in Llama 2 and how to run it locally

#72

Post by lhl »

MediumOwl wrote: post #36

There's only mention of Nvidia GPUs on the web site, what about AMD?

On Windows, llama.cpp has OpenCL support (CLBlast) and MLC LLM (https://mlc.ai/mlc-llm/docs/) has Vulkan acceleration.

On Linux, ExLlama and MLC LLM have native ROCm support, and there is a HIPified fork of llama.cpp as well.

Re: What's new in Llama 2 and how to run it locally

#73

Post by lhl »

cfn wrote: post #39
cypress66 wrote: post #30

Earlier quoted context omitted.

5 tokens/s on 70B 4bit seems really high for your setup.

This is the command: ./main -m /media/z/models/TheBloke_Llama-2-70B-Chat-GGML/llama-2-70b-chat.ggmlv3.q4_0.bin -gqa 8 -t 13 -p "The prompt..." And this is the report at the end of the answer: llama_print_timings: load time = 999.84 ms llama_print_timings: sample time = 302.21 ms / 703 runs ( 0.43 ms per token, 2326.20 tokens per second) llama_print_timings: prompt eval time = 69377.40 ms / 300 tokens ( 231.26 ms per…

For more standardized speed benchmarking, I'd recommend benchmarking with something like `-c 128 -n 1920 --ignore-eos` (and skipping the `-p` entirely). The number that you care most about would be the "eval time" tokens/second - it tends to get slower as context increases, which is why it's sort of important to standardize. I think 2-3 t/s is about what's expected (Threadripper Pro 5000 w/ 8 channels of DDR-3200 should have an expected theoretical top memory bandwidth of 204.8 GB/s - memory bandwidth is the main limiting factor for most systems for LLM inferencing).

Re: What's new in Llama 2 and how to run it locally

#74

Post by technovangelist »

qudat wrote: post #45
jmorgan wrote: post #33

If you're looking to run Llama 2 locally via a CLI or REST API (vs the web ui this article highlights), there's an open-source project some folks and I have been working on over the last few weeks: https://github.com/jmorganca/ollama More projects in this space: - llama.cpp which is a fast, low level runner (with bindings in several languages) - llm by Simon Willison which supports different backends and has a really…

I just wanted to call out that some of these quick-to-start tools are CPU only (eg ollama) which is great to play with but if you want your GPU you’ve gotta go to llama.cpp Further, the 70B for llama.cpp is still under development as far as I know.

I am using ollama today on a MacBook Pro M1Max with 64GB. Using a llama2 70b model, I am getting about 7 tokens/second with the onboard gpu. Before ollama used gpu, that was much slower. To compare, the 7b model gets me closer to 55 tokens/second. There is no way it could achieve those numbers without the gpu.

Re: What's new in Llama 2 and how to run it locally

#75

Post by cfn »

washadjeffmad wrote: post #44
cfn wrote: post #40

Earlier quoted context omitted.

That is an interesting idea. Can you tell me what is the switch for number of cores?

-t 32 Use a maximum of the number of physical cores and then scale down.

Thanks, -t 32 (instead of -t 13 which is what comes as default) makes a big difference in CPU usage across all cores. Not quite 100% but all cores are above 50% with many at 100%. It speeds up just a tiny bit the eval t/s to 3.3 (from 2.9).

Re: What's new in Llama 2 and how to run it locally

#76

Post by [deleted] »

lhl wrote: post #73
cfn wrote: post #39

Earlier quoted context omitted.

This is the command: ./main -m /media/z/models/TheBloke_Llama-2-70B-Chat-GGML/llama-2-70b-chat.ggmlv3.q4_0.bin -gqa 8 -t 13 -p "The prompt..." And this is the report at the end of the answer: llama_print_timings: load time = 999.84 ms llama_print_timings: sample time = 302.21 ms / 703 runs ( 0.43 ms per token, 2326.20 tokens per second) llama_print_timings: prompt eval time = 69377.40 ms / 300 tokens ( 231.26 ms per…

For more standardized speed benchmarking, I'd recommend benchmarking with something like `-c 128 -n 1920 --ignore-eos` (and skipping the `-p` entirely). The number that you care most about would be the "eval time" tokens/second - it tends to get slower as context increases, which is why it's sort of important to standardize. I think 2-3 t/s is about what's expected (Threadripper Pro 5000 w/ 8 channels of DDR-3200 sho…

[deleted]

Re: What's new in Llama 2 and how to run it locally

#77

Post by satvikpendem »

azeirah wrote: post #58
dmd wrote: post #51

Earlier quoted context omitted.

"How can I sum this column of numbers?" "IDK, throw it at the LLM"

I do stuff like this sometimes when I have some csv or something and need it in JSON. Could easily be done with 1 line of bash or js or python or whatever but... it's easier to just let the LLM do it for me :|

Or you could ask the AI for a script to convert it (whose resultant JSON would be free of hallucinations) and ask the AI to run the code, even, since I believe ChatGPT now has that ability, because I've definitely noticed that ChatGPT will sometimes mess up the data when converting from one format to another.

Re: What's new in Llama 2 and how to run it locally

#78

Post by vczf »

qudat wrote: post #45
jmorgan wrote: post #33

If you're looking to run Llama 2 locally via a CLI or REST API (vs the web ui this article highlights), there's an open-source project some folks and I have been working on over the last few weeks: https://github.com/jmorganca/ollama More projects in this space: - llama.cpp which is a fast, low level runner (with bindings in several languages) - llm by Simon Willison which supports different backends and has a really…

I just wanted to call out that some of these quick-to-start tools are CPU only (eg ollama) which is great to play with but if you want your GPU you’ve gotta go to llama.cpp Further, the 70B for llama.cpp is still under development as far as I know.

70B llama.cpp works now. You need the temporary `-gqa 8` flag for 70B.

You can even extend context with RoPE!

Re: What's new in Llama 2 and how to run it locally

#79

Post by drusepth »

kordlessagain wrote: post #53

I've been evaluating running non-quantized models on a Google Cloud instance with various GPUs. To run a `vllm` backed Llama 2 7b model[1], start a Debian 11 spot instance, with (1) Nvidia L4 and a g2-standard-8 w/100GB of SSD disk (ignoring the advice to use a Cuda installer image): sudo apt-get update -y sudo apt-get install build-essential -y sudo apt-get install linux-headers-$(uname -r) -y wget https://developer…

This looks promising, after looking at Azure/AWS/GC/Linode GPU instances all day. When you say "eventually terminated", what magnitude of time are you referring to? Hours? Days? Weeks? Months? Years?
Post reply on HN