Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's new in Llama 2 and how to run it locally

agi-sphere.com

Post reply on HN
Print view View original

51–60 of 85 posts

Re: What's new in Llama 2 and how to run it locally

#51

Post by dmd »

spacebanana7 wrote: post #17
pavlov wrote: post #8

Earlier quoted context omitted.

> “The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago” Hmm. If LLMs turned out like JS frameworks, that would mean that in ten years people will be saying: “Maybe we don’t really need all this expensive ceremony, honestly this could be done with vanilla if/else heuristics…?”

I can imagine a bloated world where 500B param models are used for tasks where 7B param modes perform adequately. At that time, there could be complaints on hacker news about messaging apps with autocomplete models that take up gigabytes.

"How can I sum this column of numbers?"

"IDK, throw it at the LLM"

Re: What's new in Llama 2 and how to run it locally

#53

Post by kordlessagain »

I've been evaluating running non-quantized models on a Google Cloud instance with various GPUs.

To run a `vllm` backed Llama 2 7b model[1], start a Debian 11 spot instance, with (1) Nvidia L4 and a g2-standard-8 w/100GB of SSD disk (ignoring the advice to use a Cuda installer image):

  sudo apt-get update -y
  sudo apt-get install build-essential -y
  sudo apt-get install linux-headers-$(uname -r) -y
  wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
  sudo sh cuda_11.8.0_520.61.05_linux.run # ~5 minutes, install defaults, type 'accept'/return
  sudo apt-get install python3-pip -y
  sudo pip install --upgrade huggingface_hub 
  
  # skip using token as git credential
  huggingface-cli login (for Meta model access paste token from HF[2])
  
  sudo pip install vllm # ~8 minutes
Then, edit the test code for a 7b Llama 2 model (paste into llama.py):

  from vllm import LLM
  llm = LLM(model="meta-llama/Llama-2-7b-hf")
  output = llm.generate("The capital of Brazil is called")
  print(output)
Spot price for this deployment is ~$225/month. The instance will eventually be terminated by Google, so plan accordingly.

[1] https://vllm.readthedocs.io/en/latest/models/supported_model... [2] https://huggingface.co/settings/tokens

Re: What's new in Llama 2 and how to run it locally

#54

Post by SOLAR_FIELDS »

cube2222 wrote: post #49
SOLAR_FIELDS wrote: post #46

So I tried getting Longchat running (a 32k context llama 2 7b model released a few days ago) with FastChat and I was able to successfully get it running. It seems what I was trying to use it for (Langchain SQL agent) it is not good enough out of the box. Part of this is that I think Langchain is kind of biased towards OpenAi’s models and perhaps Llamaindex would perform better. However Llamaindex uses a newer version…

In the Llama 2 paper benchmarks they did mention that Llama 2 is much worse at any kind of code generation than the OpenAI models, they were optimizing for conversational / natural language use-cases.

Interesting, what other openly licensed models are better at codegen? Or perhaps there is a version of llama 2 already fine tuned for codegen? There is starcoder but I had also not had great results with that one in my brief experiments

Re: What's new in Llama 2 and how to run it locally

#56

Post by l5870uoo9y »

ktaube wrote: post #11

What's the cheapest way to run e.g. LLaMa2-13B and have it served as an API? I've tried Inference Endpoints and Replicate, but both would cost more than just using the OpenAI offering.

I am interested in that as well. Can LLaMa2 models be deployed to VPS? (Preferable the 70B model).

Re: What's new in Llama 2 and how to run it locally

#57

Post by jmorgan »

qudat wrote: post #45
jmorgan wrote: post #33

If you're looking to run Llama 2 locally via a CLI or REST API (vs the web ui this article highlights), there's an open-source project some folks and I have been working on over the last few weeks: https://github.com/jmorganca/ollama More projects in this space: - llama.cpp which is a fast, low level runner (with bindings in several languages) - llm by Simon Willison which supports different backends and has a really…

I just wanted to call out that some of these quick-to-start tools are CPU only (eg ollama) which is great to play with but if you want your GPU you’ve gotta go to llama.cpp Further, the 70B for llama.cpp is still under development as far as I know.

Indeed, many tools in this space don't maximize resource utilization at runtime. Even the quantized models are massive resource hogs.. so you need all the performance you can get!

Ollama on macOS will use both the GPU and the Accelerator framework. It's build with the (amazing) llama.cpp project.

To run the 70B model you can try:

  ollama run llama2:70b
Note you'll most likely need a Mac with 64GB of shared memory and there's still a bit of work to do to make sure 70B works like a charm

Re: What's new in Llama 2 and how to run it locally

#58

Post by azeirah »

dmd wrote: post #51
spacebanana7 wrote: post #17

Earlier quoted context omitted.

I can imagine a bloated world where 500B param models are used for tasks where 7B param modes perform adequately. At that time, there could be complaints on hacker news about messaging apps with autocomplete models that take up gigabytes.

"How can I sum this column of numbers?" "IDK, throw it at the LLM"

I do stuff like this sometimes when I have some csv or something and need it in JSON.

Could easily be done with 1 line of bash or js or python or whatever but... it's easier to just let the LLM do it for me :|

Re: What's new in Llama 2 and how to run it locally

#59

Post by zzbzq »

MediumOwl wrote: post #52
zzbzq wrote: post #47

Earlier quoted context omitted.

Don't really work for AI. There might be a weird experimental driver for linux or something, I never got it to work.

Why not?

They suck at it basically. My understanding is it's really the software, Nivida supports AI first-class with the drivers (or whatever.) AMD has next to nothing nothing.

Re: What's new in Llama 2 and how to run it locally

#60

Post by jawerty »

If you’re someone who wants to fine-tune Llama 2 on Google Colab, I have a couple live coding streams I did this past week where I fine tune Llama on my own dataset

Here’s the stream - https://www.youtube.com/live/LitybCiLhSc?feature=share

One is with LoRa and the other QLoRa I also do a breakdown on each fine-tuning method. I wanted to make these since I myself have had issues running LLMs locally and Colab is the cheapest GPU I can find haha.

Post reply on HN