Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

github.com

Post reply on HN
Print view View original

51–60 of 78 posts

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#51

Post by gzer0 »

squarefoot wrote: post #49
alach11 wrote: post #47

Earlier quoted context omitted.

The simplest way, as rdedev is describing, is to do Retrieval Augmented Generation (RAG) in your prompting. This would require the addition of a vector database and a text embedding model. There are many open source / local / private options for that. The steps would then be: 1. Embed your private data in chunks and store the resulting embeddings in a vector database 2. In your prompting workflow, when a user queries…

Looks like this is not easy at all for a non ML expert. And probably the required computing power is still out of reach for mere mortals. I'd have a similar use case to the parent: technical books. I'd love to be able to ask where a certain topic is discussed in my pdf archive and have the AI reply with references and possibly a significant piece of the relevant articles, with images (or local links to them).

This is already a feature of adobe pdf reader professional (called index mode). There’s also an app on macOS called “pdf search” which does quite a good job. I use it for the exact reasons you describe; I’ve got a repertoire of technical books on AWS and Azure and I reference them all the time via my local search engine via these apps.

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#52

Post by siquick »

squarefoot wrote: post #49
alach11 wrote: post #47

Earlier quoted context omitted.

The simplest way, as rdedev is describing, is to do Retrieval Augmented Generation (RAG) in your prompting. This would require the addition of a vector database and a text embedding model. There are many open source / local / private options for that. The steps would then be: 1. Embed your private data in chunks and store the resulting embeddings in a vector database 2. In your prompting workflow, when a user queries…

Looks like this is not easy at all for a non ML expert. And probably the required computing power is still out of reach for mere mortals. I'd have a similar use case to the parent: technical books. I'd love to be able to ask where a certain topic is discussed in my pdf archive and have the AI reply with references and possibly a significant piece of the relevant articles, with images (or local links to them).

It’s not as difficult as you think with libraries like Llamaindex and Langchain.

Both have extensive examples in their documentation for almost identical use cases to the above.

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#53

Post by bravura »

EGreg wrote: post #50
alach11 wrote: post #47

Earlier quoted context omitted.

The simplest way, as rdedev is describing, is to do Retrieval Augmented Generation (RAG) in your prompting. This would require the addition of a vector database and a text embedding model. There are many open source / local / private options for that. The steps would then be: 1. Embed your private data in chunks and store the resulting embeddings in a vector database 2. In your prompting workflow, when a user queries…

Yeah but the big question I kept having and missing the answer is: How do you encode the private data into the vectors? It is a bunch of text but how do you choose the vector values in the first place? What software does that? Isn’t that basically an ML task with its own weights, that’s what classifiers do! I was surprised everyone had been writing about that but neglecting to explain this piece. Like math textbooks…

The most straightforward way, but of course you can fiddle around a lot:

You use sentence transformers (https://www.sbert.net/).

You use a strong baseline like all-MiniLM-L6-v2. (Or you get more fancy with something from the Massive Text Embedding Benchmark, https://huggingface.co/spaces/mteb/leaderboard)

You break your text into sentences or paragraphs with no more than 512 tokens (according to the sentence transformers tokenizer).

You embedding all your texts and insert them into your vector DB.

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#54

Post by stavros »

mayankchhabra wrote: post #44
samspenc wrote: post #37

Earlier quoted context omitted.

Ah fascinating, just curious, what's the technical blocker? I thought most of the Llama models were optimized to run on GPUs?

It's fairly straightforward to add GPU support when running on the host, but LlamaGPT runs inside a Docker container, and that's where it gets a bit challenging.

It shouldn't, nVidia provides a CUDA Docker plugin that lets you expose your GPU to the container, and it works quite well.

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#55

Post by seanthemon »

EGreg wrote: post #50
alach11 wrote: post #47

Earlier quoted context omitted.

The simplest way, as rdedev is describing, is to do Retrieval Augmented Generation (RAG) in your prompting. This would require the addition of a vector database and a text embedding model. There are many open source / local / private options for that. The steps would then be: 1. Embed your private data in chunks and store the resulting embeddings in a vector database 2. In your prompting workflow, when a user queries…

Yeah but the big question I kept having and missing the answer is: How do you encode the private data into the vectors? It is a bunch of text but how do you choose the vector values in the first place? What software does that? Isn’t that basically an ML task with its own weights, that’s what classifiers do! I was surprised everyone had been writing about that but neglecting to explain this piece. Like math textbooks…

Many ways to skin this question, but in essence a simple idea is that word vectorization is assigning a numerical representation to a specific word, embeddings on the other hand are taking those words, turning them into numerical representations but keeping semantically similar words closer dimensionally.

Yes, turning words into vectors is it's own class of machine learning. You can learn a lot on the NLP course on hugging face https://huggingface.co/learn/nlp-course/chapter1/1 (and on youtube).

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#56

Post by deet »

squarefoot wrote: post #49
alach11 wrote: post #47

Earlier quoted context omitted.

The simplest way, as rdedev is describing, is to do Retrieval Augmented Generation (RAG) in your prompting. This would require the addition of a vector database and a text embedding model. There are many open source / local / private options for that. The steps would then be: 1. Embed your private data in chunks and store the resulting embeddings in a vector database 2. In your prompting workflow, when a user queries…

Looks like this is not easy at all for a non ML expert. And probably the required computing power is still out of reach for mere mortals. I'd have a similar use case to the parent: technical books. I'd love to be able to ask where a certain topic is discussed in my pdf archive and have the AI reply with references and possibly a significant piece of the relevant articles, with images (or local links to them).

The computing power is definitely not out of reach of mere mortals. I'm working on software that does this for emails and common documents, generating a hybrid semantic (vector) and keyword search system over all your data, locally.

The computing power we're requiring is simply what's available in any M1/M2 Mac, and the resource usage for the indexing and search is negligible. This isn't even a hard requirement, any modern PC could index all your emails and do the local hybrid search part.

Running the local LM is what requires more resources, but as this project shows it's absolutely possible.

Of course getting it to work *well* for certain use cases is still hard. Simply searching for close sections of papers and injecting them into the prompt as others have mentioned doesn't always provide enough context for the LM to give a good answer. Local LMs aren't great at reasoning over large amounts of data yet, but getting better every week so it's just a matter of time.

(If you're curious my email is in my profile)

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#57

Post by freedomben »

lkbm wrote: post #19
lazzlazzlazz wrote: post #12

(1) What are the best more creative/less lobotomized versions of Llama 2? (2) What's the best way to get one of those running in a similarly easy way?

https://github.com/jmorganca/ollama was extremely simple to get running on my M1 and has a couple uncensored models you can just download and use.

The uncensored model isn't very uncensored. It refused a number of test prompts for me, telling me that things were unsafe and telling me to consult a professional

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#58

Post by stavros »

raffraffraff wrote: post #35
super256 wrote: post #17

Earlier quoted context omitted.

Well, GPT is simply an initialism for "Generative Pre-trained Transformer". In Germany, a trademark can be lost if it becomes a "Gattungsbegriff" (generic term). This happens when a trademark becomes so well-known and widely used that it becomes the common term for a product or service, rather than being associated with a specific company or brand. For example, if a company invented a new type of vacuum cleaner and t…

Actually in the UK and Ireland a vacuum cleaner is called a Hoover. But in general I think we do that less than Americans. For example, we don't call a public announcement system a "Tannoy". That's a brand of hifi speakers. And we'd say "photo copier" instead of Xerox.

You don't call a PA system a tannoy? That's news to all my English friends, who keep telling me that people announced things over the tannoy.

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#59

Post by stavros »

super256 wrote: post #17
caesil wrote: post #13

So many projects still using GPT in their name. Is the thinking here that OpenAI is not going to defend that trademark? Or just kicking the can down the road on rebranding until the C&D letter arrives?

Well, GPT is simply an initialism for "Generative Pre-trained Transformer". In Germany, a trademark can be lost if it becomes a "Gattungsbegriff" (generic term). This happens when a trademark becomes so well-known and widely used that it becomes the common term for a product or service, rather than being associated with a specific company or brand. For example, if a company invented a new type of vacuum cleaner and t…

But can you trademark a vacuum cleaner called Vacuum-3 and expect the trademark for the word "vacuum" to be awarded to you?

What OpenAI did was the opposite of what you describe, they took an already-generic term and used it for a product name. I wouldn't be surprised if they can't actually trademark that.

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#60

Post by deet »

EGreg wrote: post #50
alach11 wrote: post #47

Earlier quoted context omitted.

The simplest way, as rdedev is describing, is to do Retrieval Augmented Generation (RAG) in your prompting. This would require the addition of a vector database and a text embedding model. There are many open source / local / private options for that. The steps would then be: 1. Embed your private data in chunks and store the resulting embeddings in a vector database 2. In your prompting workflow, when a user queries…

Yeah but the big question I kept having and missing the answer is: How do you encode the private data into the vectors? It is a bunch of text but how do you choose the vector values in the first place? What software does that? Isn’t that basically an ML task with its own weights, that’s what classifiers do! I was surprised everyone had been writing about that but neglecting to explain this piece. Like math textbooks…

It's possible to extend the effective context window of many OSS models using various techniques. The Llama-related models and others there's a technique called "RoPE scaling" which allows you to run inference over a longer context window than the model was originally trained for. (This reddit post help highlight this fact: https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkawar...)

But even at 100K, you do eventually run out of context. You would with 1M tokens too. 100K tokens is the new 64K of RAM, you're going to end up wanting more.

So techniques like RAG that others have mentioned are necessary in the end at some point, at least with models that look like they do today.

Post reply on HN