Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Local RAG with private knowledge base

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 41 posts

Re: Ask HN: Local RAG with private knowledge base

#11

Post by generalizations »

kingkongjaffa wrote: post #2

> expected the more documents we feed the lower the accuracy Not surprising! The LLM itself is the least important bit as long as it’s serviceable. Depending on your goal you need to have a specific RAG strategy. How are you breaking up the documents? Are the documents consistently formatted to make breaking them up uniform? Do you need to do some preprocessing to make them uniform? When you retrieve documents how ma…

> How are you breaking up the documents? Are the documents consistently formatted to make breaking them up uniform? Do you need to do some preprocessing to make them uniform?

> When you retrieve documents how many do you stuff into your prompt as context?

> Do you stuff the same top N chunks from a single prompt or do you have a tailored prompt chain retrieving different resourced based on the prompt and desired output?

Wouldn't these questions be answered by the RAG solution the OP is asking for?

Re: Ask HN: Local RAG with private knowledge base

#12

Post by mutagen »

I'm very interested in checking something like this out for getting to grips with a local codebase of unfamiliar SQL and assorted scripts and reports.

I have a few tabs open that I haven't had a chance to try:

https://github.com/Mintplex-Labs/anything-llm

https://github.com/Bin-Huang/chatbox

https://github.com/saeedezzati/superpower-chatgpt

Re: Ask HN: Local RAG with private knowledge base

#13

Post by Ey7NFZ3P0nzAe »

I've made wdoc just for that: https://github.com/thiswillbeyourgithub/WDoc

I am a medical student with thousands of pdfs, various anki databases, video conferences, audio recordings, markdown notes etc. It can query into all of them and return extremely high quality output with sources to each original document.

It's still in alpha though and there's only 0.5 user beside me that I know of so there are bugs that have yet to be found!

Re: Ask HN: Local RAG with private knowledge base

#15

Post by stavros »

specproc wrote: post #14

Fascinating there doesn't seem to be a consensus "just use this" answer here.

My sentiments exactly, and given how widespread a need RAG is, I'm extremely surprised that we don't have something solid and clearly a leader in the space yet. We don't even seem to have two or three! It's "pick one of these million side-projects".

Re: Ask HN: Local RAG with private knowledge base

#16

Post by shikha1001 »

You can use BerryDB for doing this use case at scale. BerryDB is a JSON native database that can ingest PDFs, images, etc and it has a built in semantic layer (for labeling) so that way you can build your knowledge database with entities and relationships. This will ground your knowledge with entities and accuracy scales very well with large number of documents

It provides APIs to extract paragraphs or tables from your PDFs in bulk, You can also separately do bulk labeling (say classification, NER and other labeling types). Once you have a knowledge database, it creates 4 indexes on top of your JSON data layer - db index for metadata search, full text search index, annotation index and vector index, so that way you can perform any search operation including hybrid search

The fact that your data layer is in JSON, it gives you infinite flexibility to add new snippets of knowledge or new labels and improve accuracy over time.

https://berrydb.io

Re: Ask HN: Local RAG with private knowledge base

#17

Post by 0xbadc0de5 »

Is it possible that you're using RAG in a nonstandard way? Thousands of documents seems like a lot to feed into a single query. Have you tried using collections and tags to narrow down the field prior to performing the semantic search? You may also want to consider using a larger model or one with a larger context window.

Re: Ask HN: Local RAG with private knowledge base

#19

Post by imrantech »

The key to accuracy is use case specific knowledge graphs. Here is a YouTube video of how to do it. https://youtu.be/iWtF1Qe7QkM The key benefits are - Improved data quality of the data available for genAI - Reduction in risk associated with genAI’s known problems - Increasing business value due to being able to hit use case driven accuracy/reliability, security, and transparency metrics

Re: Ask HN: Local RAG with private knowledge base

#20

Post by LunaSea »

stavros wrote: post #15
specproc wrote: post #14

Fascinating there doesn't seem to be a consensus "just use this" answer here.

My sentiments exactly, and given how widespread a need RAG is, I'm extremely surprised that we don't have something solid and clearly a leader in the space yet. We don't even seem to have two or three! It's "pick one of these million side-projects".

Because RAGs are simply a list of vectors and a similarity search with some variations trying to use knowledge graphs.

So everybody is roughly using the same method with some tweaks here and there and thus getting a similar quality in results.

Post reply on HN