Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

github.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#41

Post by mayankchhabra »

reneberlin wrote: post #40
ryanSrich wrote: post #7

Earlier quoted context omitted.

Interesting. I might try to get this to work on my NAS.

Good luck! The token/sec will be under your expectations or it will overheat. You really shouldn't play games with your data-storage. You could try it with an old laptop to see how bad it performs. Ruining your NAS for this is a bit over the top to show, that "it worked somehow". But i don't know, maybe your NAS has a powerful processor and is tuned to the max and you have redundancy and don't care to loose a NAS? Or…

Not sure how powerful their NAS is, but on Umbrel Home (which has an N5105 CPU), it's pretty useable with ~3 tokens generated per second.

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#42

Post by DrPhish »

mayankchhabra wrote: post #31
SubiculumCode wrote: post #26

I didn't see any info on how this is different than installing/running llamacpp or koboldcpp. New offerings are awesome of course, but what is it adding?

The main difference is setting everything up yourself manually, downloading the modal, optimizing the parameters for best performance, running an API server and a UI front-end - which is out of reach for most non-technical people. With LlamaGPT, it's just one command: `docker compose up -d` or one click install for umbrelOS home server users.

Maybe I've been at this for too long and can't see the pitfalls of a normal user, but how is that easier than using an oobabooga one-click installer (an option that's been around "forever")?

I guess ooba one-click doesn't come with a model included, but is that really enough of a hurdle to stop someone from getting it going?

Maybe I'm not seeing the value proposition of this. Glad to be enlightened!

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#43

Post by influxmoment »

ccozan wrote: post #20

Ok, since is running all private, how can I add my own private data? For example I have a 20+ years of an email archive that I'd like to be ingested.

That would require custom training. This project only does inference

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#44

Post by mayankchhabra »

samspenc wrote: post #37
mayankchhabra wrote: post #30

Earlier quoted context omitted.

Ah yes, running on GPU isn't supported at the moment. But CUDA (for Nvidia GPUs) and Metal support is on the roadmap!

Ah fascinating, just curious, what's the technical blocker? I thought most of the Llama models were optimized to run on GPUs?

It's fairly straightforward to add GPU support when running on the host, but LlamaGPT runs inside a Docker container, and that's where it gets a bit challenging.

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#45

Post by Havoc »

redox99 wrote: post #25
Havoc wrote: post #10

Earlier quoted context omitted.

Llama is definitely "censored" though I've not found this to be an issue in practice. Guess it depends on what you want to do with it

llama-chat is censored, not base llama

Oh - didn’t know that. Interesting.

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#46

Post by aryamaan »

phillipcarter wrote: post #39
ravishi wrote: post #33

Earlier quoted context omitted.

And is that hard? Sorry if this is a newbie question, I'm really out of the loop on this tech. What would be required? Computing power and tagging? Or can you like improve the model without much human intervention? Can it be done incrementally with usage and user feedback? Would a single user even be able to generate enough feedback for this?

Yes, this would be quite hard. Fine-tuning an LLM is no simple task. The tools and guidance around it are very new, and arguably not meant for non-ML Engineers.

What are some ways people to get familiar with machine learning engineering who are also working adults

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#47

Post by alach11 »

ccozan wrote: post #20

Ok, since is running all private, how can I add my own private data? For example I have a 20+ years of an email archive that I'd like to be ingested.

The simplest way, as rdedev is describing, is to do Retrieval Augmented Generation (RAG) in your prompting. This would require the addition of a vector database and a text embedding model. There are many open source / local / private options for that.

The steps would then be: 1. Embed your private data in chunks and store the resulting embeddings in a vector database 2. In your prompting workflow, when a user queries the chat model, embed their query using the embedding model 3. Retrieve the most similar chunks of text from your vector database based on cosine similarity 4. In the chat response, provide it the context of those chunks of text

For example, if you asked "who have I discussed Ubuntu with?", it might retrieve emails that have similar content. Then the model will be able to answer informed by that context.

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#48

Post by alach11 »

influxmoment wrote: post #43
ccozan wrote: post #20

Ok, since is running all private, how can I add my own private data? For example I have a 20+ years of an email archive that I'd like to be ingested.

That would require custom training. This project only does inference

No need to fine tune the model. The model could be augmented with retrieved context (as discussed in my sibling comment).

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#49

Post by squarefoot »

alach11 wrote: post #47
ccozan wrote: post #20

Ok, since is running all private, how can I add my own private data? For example I have a 20+ years of an email archive that I'd like to be ingested.

The simplest way, as rdedev is describing, is to do Retrieval Augmented Generation (RAG) in your prompting. This would require the addition of a vector database and a text embedding model. There are many open source / local / private options for that. The steps would then be: 1. Embed your private data in chunks and store the resulting embeddings in a vector database 2. In your prompting workflow, when a user queries…

Looks like this is not easy at all for a non ML expert. And probably the required computing power is still out of reach for mere mortals. I'd have a similar use case to the parent: technical books. I'd love to be able to ask where a certain topic is discussed in my pdf archive and have the AI reply with references and possibly a significant piece of the relevant articles, with images (or local links to them).

Re: Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

#50

Post by EGreg »

alach11 wrote: post #47
ccozan wrote: post #20

Ok, since is running all private, how can I add my own private data? For example I have a 20+ years of an email archive that I'd like to be ingested.

The simplest way, as rdedev is describing, is to do Retrieval Augmented Generation (RAG) in your prompting. This would require the addition of a vector database and a text embedding model. There are many open source / local / private options for that. The steps would then be: 1. Embed your private data in chunks and store the resulting embeddings in a vector database 2. In your prompting workflow, when a user queries…

Yeah but the big question I kept having and missing the answer is:

How do you encode the private data into the vectors? It is a bunch of text but how do you choose the vector values in the first place? What software does that? Isn’t that basically an ML task with its own weights, that’s what classifiers do!

I was surprised everyone had been writing about that but neglecting to explain this piece. Like math textbooks that “leave it as an exercise to the reader”.

Claude with its 100k context window doesn’t need to do this vector encoding. Is there anything like that in open source AI at the moment ?

Post reply on HN