Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 92 posts

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#71

Post by haiku2077 »

ravetcofx wrote: post #53
andai wrote: post #35

Earlier quoted context omitted.

I found that grep actually outperformed vector search for many queries. The only thing I was missing was when I didn't know how exactly to phrase something (the exact keyword to use). Do keyword search systems have workarounds for this? My own idea was for each keyword to generate a list of neighbor keywords in semantic space. I figured with such a dataset, I'd get something approximating vector search for free. I ma…

Do you know of any way to build a fast index you can run grep against? Would love to have something as instantaneous as "Everything" on windows for full text on Linux so I can just dump everything in a directory

Try ripgrep.

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#72

Post by tspann »

pierre wrote: post #15

RAG cli from llamaindex, allow you to do it 100% locally when used with ollama or llamacpp instead of OpenAI. https://docs.llamaindex.ai/en/stable/getting_started/starter...

https://milvus.io/docs/integrate_with_llamaindex.md

Pretty easy to run local and lightweight with Milvus Lite with LlamaIndex

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#74

Post by hm-nah »

You have the find a good OCR tool that you can run locally on your hardware. RAG depends on your doc processing pipeline.

It’s not local, but the Azure Document Intelligence OCR service has a number of prebuilt models. The “prebuilt-read” model is $1.50/1k pages. Once you OCR your docs, you’ll have a JSON of all the text AND you get breakdowns by page/word/paragraph/tables/figures/alllll with bouding-boxes.

Forget the Lang/Llama/Chain-theory. You can do it all in vanilla Python.

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#76

Post by everforward »

ravetcofx wrote: post #53
andai wrote: post #35

Earlier quoted context omitted.

I found that grep actually outperformed vector search for many queries. The only thing I was missing was when I didn't know how exactly to phrase something (the exact keyword to use). Do keyword search systems have workarounds for this? My own idea was for each keyword to generate a list of neighbor keywords in semantic space. I figured with such a dataset, I'd get something approximating vector search for free. I ma…

Do you know of any way to build a fast index you can run grep against? Would love to have something as instantaneous as "Everything" on windows for full text on Linux so I can just dump everything in a directory

As others have said, ripgrep et al are faster than regular grep. You would probably also get much faster results with an alias that excludes directories you don't expect results in (I.e. I don't normally grep in /var at all).

I have seen some recommendations for recoll, but I haven't used it so can't comment. Anecdotally, I normally just use ripgrep in my home directory (it's almost always in ~ if I don't remember where it is). It's fast enough as long as my homedir is local (I.e. not on NFS).

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#77

Post by Ey7NFZ3P0nzAe »

I am a medical students with thousands and thousands of PDF and was unsatisfied with RAG tools so I made my own. It can consume basically any type of content (pdf, epub, youtube playlist, anki database, mp3, you name it) and does a multi step RAG by first using embedding then filtering using a smaller LLM then answering using by feeding each remaining document to the strong LLM then combine those answers.

It supports virtually all LLMs and embeddings, including local LLMs and local embedding It scales surprisingly well and I have tons of improvements to come, when I have some free time or procrastinate. Don't hesitate to ask for features!

Here's the link: https://github.com/thiswillbeyourgithub/DocToolsLLM/

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#78

Post by vladsanchez »

papichulo2023 wrote: post #36
ekianjo wrote: post #19

Earlier quoted context omitted.

llamaindex has an horrible API, very poor docs and is constantly changing. I do not recommend it.

Any alternative?

LOL `papichulo`? Que tigre!?

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#79

Post by dmd »

hm-nah wrote: post #73
papichulo2023 wrote: post #36

Earlier quoted context omitted.

Any alternative?

Vanilla python

So your solution to “I don’t like flying [specific airline]” would be “how about a big pile of aluminum and some jet fuel”?

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#80

Post by samspenc »

Ey7NFZ3P0nzAe wrote: post #77

I am a medical students with thousands and thousands of PDF and was unsatisfied with RAG tools so I made my own. It can consume basically any type of content (pdf, epub, youtube playlist, anki database, mp3, you name it) and does a multi step RAG by first using embedding then filtering using a smaller LLM then answering using by feeding each remaining document to the strong LLM then combine those answers. It supports…

Nvidia's 'Chat with RTX' can do this as well https://www.nvidia.com/en-us/ai-on-rtx/chatrtx/

You do need a beefy GPU to run the local LLM, but I think it's a similar requirement for running any LLM on your machine.

Post reply on HN