Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Local RAG with private knowledge base

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 41 posts

Re: Ask HN: Local RAG with private knowledge base

#21

Post by disgruntledphd2 »

LunaSea wrote: post #20
stavros wrote: post #15

Earlier quoted context omitted.

My sentiments exactly, and given how widespread a need RAG is, I'm extremely surprised that we don't have something solid and clearly a leader in the space yet. We don't even seem to have two or three! It's "pick one of these million side-projects".

Because RAGs are simply a list of vectors and a similarity search with some variations trying to use knowledge graphs. So everybody is roughly using the same method with some tweaks here and there and thus getting a similar quality in results.

Yeah, I'm coming to believe that this is a much, much, much harder problem than it looks. Getting it running is pretty easy, but actually tuning the results to make them better is tricky, especially if you're not a domain expert in the area you're working on.

Evals seem like a solution, but they're very tied to specific examples, so it looks like that might be most of the issue in getting this to work, as with a good set of evals, one can actually measure performance and test out different approaches.

Embedding also seems to be a bit of a dark art in that every tutorial uses something small, but I haven't seen a lot of work on comparing the performance of particular embeddings for domain specific tasks.

Re: Ask HN: Local RAG with private knowledge base

#22

Post by ivansavz »

I recently watched a talk[1] on this exact use case: a RAG system that runs on localhost with a simple web UI, and based on very powerful text processing, and a simple backend (PHP and sqlite3 with FTS and vector search extensions).

You can see the project page here: https://textualization.com/ragged/

src and scripts here: https://github.com/Textualization/the-ragged-edge-box

[1] video presentation about the project https://www.youtube.com/watch?v=_fJFuL2pLvw

Re: Ask HN: Local RAG with private knowledge base

#24

Post by brumar »

Ey7NFZ3P0nzAe wrote: post #13

I've made wdoc just for that: https://github.com/thiswillbeyourgithub/WDoc I am a medical student with thousands of pdfs, various anki databases, video conferences, audio recordings, markdown notes etc. It can query into all of them and return extremely high quality output with sources to each original document. It's still in alpha though and there's only 0.5 user beside me that I know of so there are bugs that have…

Med student working on sophisticated RAG system... What kind of beast are you? Thanks for sharing anyway, I'll keep tab on it.

Re: Ask HN: Local RAG with private knowledge base

#28

Post by its_down_again »

I recently had some luck turning an excel tracker that lists multiple locations and their services into markdown for RAG. It worked great as a natural language lookup, way better than digging through a big Excel sheet.

I uploaded them through Supabase Embeddings Generator if you're curious. https://github.com/supabase/embeddings-generator

But things got a bit messy when I handed it off to someone else. They started using synonyms for locations, like abbreviated addresses to refer to certain columns, which didn't return the right documents.

Followed a friend's suggestion to try NotebookLM, so I uploaded the same docs there, and it was awesome. Some cloud-hosted vector DB tools only handle PDFs, but NotebookLM accepted my Markdown and chunked the docs better than the Supabase library I was using. It just "worked".

I would swap over to NotebookLM because their document chunking and RAG performance is working for my use case, but they just don’t offer an API yet.

I also gave Gemini a shot using this guide, but didn’t get the results I was hoping for. https://codelabs.developers.google.com/multimodal-rag-gemini...

Am I overhyping NotebookLM? I’d love to know to get on-par document chunking, because that seems to deliver fantastic RAG right out of the box. I’m planning to try some other suggestions I’ve seen here, but any insights into how NotebookLM does its magic would be super helpful.

Re: Ask HN: Local RAG with private knowledge base

#29

Post by Llamamoe »

Ey7NFZ3P0nzAe wrote: post #13

I've made wdoc just for that: https://github.com/thiswillbeyourgithub/WDoc I am a medical student with thousands of pdfs, various anki databases, video conferences, audio recordings, markdown notes etc. It can query into all of them and return extremely high quality output with sources to each original document. It's still in alpha though and there's only 0.5 user beside me that I know of so there are bugs that have…

Could you include information about the hardware necessary to run it?

Re: Ask HN: Local RAG with private knowledge base

#30

Post by shadowmnifold »

imrantech wrote: post #19

The key to accuracy is use case specific knowledge graphs. Here is a YouTube video of how to do it. https://youtu.be/iWtF1Qe7QkM The key benefits are - Improved data quality of the data available for genAI - Reduction in risk associated with genAI’s known problems - Increasing business value due to being able to hit use case driven accuracy/reliability, security, and transparency metrics

Thanks for this. I am going to watch the whole thing. Use case specific knowledge graphs sounds right to me.
Post reply on HN