What's new in Llama 2 and how to run it locally • Hacker News Board

Re: What's new in Llama 2 and how to run it locally

#81

Post by bigcloud1299 » Tue, Aug 08, 2023, 3:35 AM UTC

jwr wrote: post #24

jurmous wrote: post #10

Did anybody try the Llama 2 model with languages other than English? The paper notes that it works best with English and the amount of training data for other languages is only a fraction. Which likely would make it unusable for me.. See table 10 (page 22) of the whitepaper for the numbers: https://ai.meta.com/research/publications/llama-2-open-found... Are there other downloadable models which can be used in a multi…

I tried to run llama-2 (13B) locally for translations. Comparing to gpt-3.5-turbo and gpt-4-0613, which I use a lot, and I tried with several languages that I know. It is nowhere near usable. Perhaps the 70B model performs better, but 13B produces translations that are garbage.

I am doing STT using whisper on my local laptop and recording all the meetings. I am then asking llama and chstgpt to summarize them. ChatGPT does 10x better job. It’s been good to be able to get the main points out of otherwise fuxking boring a. garbage meetings that are at least 30-45 mins long.

Re: What's new in Llama 2 and how to run it locally

#82

Post by dTal » Tue, Aug 08, 2023, 11:06 AM UTC

cfn wrote: post #37

MrYellowP wrote: post #26

Earlier quoted context omitted.

How does the 7B model use only 512 megabytes? That's not possible? Is it using mmap and concealing the actual memory usage?

I forgot to say I am using ggml models. This is what llama.cpp outputs when you start it: main: build = 942 (4f6b60c) main: seed = 1691400051 llama.cpp: loading model from /media/z/models/TheBloke_Llama-2-7b-chat-GGML/llama-2-7b-chat.ggmlv3.q5_1.bin llama_model_load_internal: format = ggjt v3 (latest) llama_model_load_internal: n_vocab = 32000 llama_model_load_internal: n_ctx = 512 llama_model_load_internal: n_embd =…

Yeah that's using over 5 gigabytes, not 400 megabytes. Your process monitor is inaccurate; the memory used doesn't "count" because it's disk backed and the kernel is free to discard memory pages if it really needs the memory because it can always load it back from disk. But every time it does that you need to wait for the slow disk to read it back in again.

Re: What's new in Llama 2 and how to run it locally

#83

Post by cfn » Tue, Aug 08, 2023, 6:24 PM UTC

dTal wrote: post #82

cfn wrote: post #37

Earlier quoted context omitted.

I forgot to say I am using ggml models. This is what llama.cpp outputs when you start it: main: build = 942 (4f6b60c) main: seed = 1691400051 llama.cpp: loading model from /media/z/models/TheBloke_Llama-2-7b-chat-GGML/llama-2-7b-chat.ggmlv3.q5_1.bin llama_model_load_internal: format = ggjt v3 (latest) llama_model_load_internal: n_vocab = 32000 llama_model_load_internal: n_ctx = 512 llama_model_load_internal: n_embd =…

Yeah that's using over 5 gigabytes, not 400 megabytes. Your process monitor is inaccurate; the memory used doesn't "count" because it's disk backed and the kernel is free to discard memory pages if it really needs the memory because it can always load it back from disk. But every time it does that you need to wait for the slow disk to read it back in again.

It is strange that it does that given that there's plenty of free memory available in the system (it has 256Gb of RAM and wasn't running anything else).

Re: What's new in Llama 2 and how to run it locally

#84

Post by dTal » Wed, Aug 09, 2023, 5:32 PM UTC

cfn wrote: post #83

dTal wrote: post #82

Earlier quoted context omitted.

Yeah that's using over 5 gigabytes, not 400 megabytes. Your process monitor is inaccurate; the memory used doesn't "count" because it's disk backed and the kernel is free to discard memory pages if it really needs the memory because it can always load it back from disk. But every time it does that you need to wait for the slow disk to read it back in again.

It is strange that it does that given that there's plenty of free memory available in the system (it has 256Gb of RAM and wasn't running anything else).

Not really, it's just a question of accounting. mmap is functionally the same as disk cache. As long as you've got the RAM, it'll run from RAM. If you really want, you can force llama.cpp not to use mmap and explicitly load everything into RAM, but there's not really any performance reason to do that - if the kernel keeps dropping your pages, you're under memory pressure anyway and "locking" that memory will probably end up either thrashing or invoking the OOM killer.

Re: What's new in Llama 2 and how to run it locally

#85

Post by kordlessagain » Wed, Aug 09, 2023, 5:46 PM UTC

drusepth wrote: post #79

kordlessagain wrote: post #53

I've been evaluating running non-quantized models on a Google Cloud instance with various GPUs. To run a `vllm` backed Llama 2 7b model[1], start a Debian 11 spot instance, with (1) Nvidia L4 and a g2-standard-8 w/100GB of SSD disk (ignoring the advice to use a Cuda installer image): sudo apt-get update -y sudo apt-get install build-essential -y sudo apt-get install linux-headers-$(uname -r) -y wget https://developer…

This looks promising, after looking at Azure/AWS/GC/Linode GPU instances all day. When you say "eventually terminated", what magnitude of time are you referring to? Hours? Days? Weeks? Months? Years?

You can set it to a time, or expect to have the spot instance terminated after 24 hours. That said, Google will terminate instances as needed for the zone you deploy in, so your mileage will vary.