Looking for a free, local, open source RAG solution for running a reference library with 1000s of technical PDFs and word docs. Tried the Ollama + open webui, Ollama+Anything LLM with opensource models such as Llama3.2 etc. As expected the more documents we feed the lower the accuracy. Doing it for a bunch of senior citizens who still love geeking out.
Ask HN: Local RAG with private knowledge base
1–10 of 41 posts
Re: Ask HN: Local RAG with private knowledge base
#2> expected the more documents we feed the lower the accuracy
Not surprising!
The LLM itself is the least important bit as long as it’s serviceable.
Depending on your goal you need to have a specific RAG strategy.
How are you breaking up the documents? Are the documents consistently formatted to make breaking them up uniform? Do you need to do some preprocessing to make them uniform?
When you retrieve documents how many do you stuff into your prompt as context?
Do you stuff the same top N chunks from a single prompt or do you have a tailored prompt chain retrieving different resourced based on the prompt and desired output?
Re: Ask HN: Local RAG with private knowledge base
#3Checkout llmware -
https://github.com/llmware-ai/llmware
Re: Ask HN: Local RAG with private knowledge base
#4Checkout kotaemon https://github.com/Cinnamon/kotaemon
Re: Ask HN: Local RAG with private knowledge base
#5[dead]
Re: Ask HN: Local RAG with private knowledge base
#6I would look at articles on building an open source RAG pipeline. Generation (model) is the last in a series of important steps -- you have options to choose from (retrieval, storage, etc) in each component step. Those decisions will affect the accuracy you mention.
Langchain, llamaindex have good resources on building such a pipeline the last I checked
Re: Ask HN: Local RAG with private knowledge base
#7Check out Khoj. https://github.com/khoj-ai/khoj
Re: Ask HN: Local RAG with private knowledge base
#8I’ve got this RAG repo working entirely locally (Ollama/Postgres) but it doesnt RAG on documents like you want.
https://github.com/Azure-Samples/rag-postgres-openai-python
I’d like to make that version when I have the time, probably just using Llamaindex for the ingestion.
My tips for getting SLMs working well for RAG: http://blog.pamelafox.org/2024/08/making-ollama-compatible-r...
Re: Ask HN: Local RAG with private knowledge base
#9Check out Khoj. https://github.com/khoj-ai/khoj
Yes! We can definitely help with this. Khoj lets you chat with your documents, indexing your private knowledge base for local RAG with any open source (or foundation) model.
You can make it as 'fancy' as you want, and use speech-to-text, image generation, web scraping, custom agents.
Let me know if you run into any issues? I'd love to get this setup for senior citizens! You can reach me at saba at khoj.dev.
Re: Ask HN: Local RAG with private knowledge base
#10txtai was brought up in a discussion yesterday. I saved it to look at later. But you might find it useful.
https://github.com/neuml/txtai
Here is that that thread. https://news.ycombinator.com/item?id=41981907