41–50 of 85 posts
One way to connect llama-2 (cpp) to a node.js app is by using this helper class (stdin) https://gist.github.com/HackyDev/814c6d1c96f259a13dbf5b2dabf...
Earlier quoted context omitted.
> “The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago” Hmm. If LLMs turned out like JS frameworks, that would mean that in ten years people will be saying: “Maybe we don’t really need all this expensive ceremony, honestly this could be done with vanilla if/else heuristics…?”
Well these models at least tell in name what they do eg llama-2-70b-Guanaco-QLoRA-fp16.
If you want to try Llama 2 on a Mac and have Homebrew (or Python/pip) you may find my LLM CLI tool interesting: https://simonwillison.net/2023/Aug/1/llama-2-mac/
Does it support Metal / MPS acceleration?
Sadly 7b is not very good for SQL tasks. I think even with RAG it would struggle.
Earlier quoted context omitted.
Try to use less cores. RAM bandwidth is real limiting factor there, so there always some sweet spot between CPU cores and RAM bandwidth for individual system. For example, I use only 6 cores from 10 on my M1 Pro laptop.
That is an interesting idea. Can you tell me what is the switch for number of cores?
Use a maximum of the number of physical cores and then scale down.
If you're looking to run Llama 2 locally via a CLI or REST API (vs the web ui this article highlights), there's an open-source project some folks and I have been working on over the last few weeks: https://github.com/jmorganca/ollama More projects in this space: - llama.cpp which is a fast, low level runner (with bindings in several languages) - llm by Simon Willison which supports different backends and has a really…
Further, the 70B for llama.cpp is still under development as far as I know.
Unfortunately with all of the hype it seems that unless you have a REALLY beefy machine the better 70B model feels out of reach for most to run locally leaving the 7B and 13B as the only viable options outside of some quantization trickery. Or am I wrong in that?
I want to focus more on larger context windows since it seems like RAG has a lot of promise so it seems like the 7B with giant context window is the best path to explore rather than focusing on getting the 70B to work locally
There's only mention of Nvidia GPUs on the web site, what about AMD?
Earlier quoted context omitted.
At Mirage Studio we have successfully finetuned Llama 2 7B on a Dutch dataset to get it to output Dutch in a coherent way: https://huggingface.co/Mirage-Studio/llama-gaan-2-7b-chat-hf...
Dutch is a low-hanging fruit though, ain't it? Closely related Germanic language with heavy English influence post-war? edit: To make the implied question explicit, I guess it might do well on other similar Germanic languages (say Norwegian) but struggle beyond that? Or?
So I tried getting Longchat running (a 32k context llama 2 7b model released a few days ago) with FastChat and I was able to successfully get it running. It seems what I was trying to use it for (Langchain SQL agent) it is not good enough out of the box. Part of this is that I think Langchain is kind of biased towards OpenAi’s models and perhaps Llamaindex would perform better. However Llamaindex uses a newer version…
So I tried getting Longchat running (a 32k context llama 2 7b model released a few days ago) with FastChat and I was able to successfully get it running. It seems what I was trying to use it for (Langchain SQL agent) it is not good enough out of the box. Part of this is that I think Langchain is kind of biased towards OpenAi’s models and perhaps Llamaindex would perform better. However Llamaindex uses a newer version…
More reading on that problem if you're curious: https://arxiv.org/pdf/2307.03172.pdf