Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 92 posts

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#11

Post by bendsawyer »

I looked into this for sensitive material recently. In the end I got a purpose-built local system built and am having it remotely maintained. Cost: around 5k a year. I used http://www.skunkwerx.ai, who are US based.

The result is a huge step up from 'full text search' solutions, for my use case. I can have conversations with decades of documents, and it's incredibly helpful. The support scheme keeps my original documents unconnected from the machine, which I own, while updates are done over a remote link. It's great, and I feel safe.

Things change so fast in this space that there did not seem to be a cheap, stable, local alternative. I honestly doubt one is coming. This is not a on-size-fits-all problem.

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#12

Post by bendsawyer »

1123581321 wrote: post #10

Devonthink would do this with a tiny model to translate your natural length search prompts into its syntax and your folder/tag tree. If you're okay with some false positives, Devonthink would work as is, actually.

I used to use this, but the LLM approach allows for much deeper interactions. Not "find all times I've typed X" but

"act as an expert in Y, looking across all times I've typed X, summarize my changing position over thee years, and suggest other terms that have a similar pattern of change, in a list."

The kind of thing I used to give to an intern over a month, with results that are not far off what that intern produced...

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#13

Post by bendsawyer »

dudus wrote: post #3

I tried Google's NotebookLM for this use case and was very pleased with the experience. If you trust Google that is.

Not offline. I do not trust anyone with some data, because I have contractually promised not to do so.

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#16

Post by yousnail »

PrivateGPT is a great starting point for using a local model and RAG. Text-generation-ui, oogabooga, using superbooga V2 is very nice and more customizable.

I’ve used both for sensitive internal SOPs, and both work quite well. Private gpt excels at ingesting many separate documents, the other excels at customization. Both are totally offline, and can use mostly whatever models you want.

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#17

Post by homarp »

pierre wrote: post #15

RAG cli from llamaindex, allow you to do it 100% locally when used with ollama or llamacpp instead of OpenAI. https://docs.llamaindex.ai/en/stable/getting_started/starter...

and at some point (https://github.com/ggerganov/llama.cpp/issues/7444) you will be able to use Phi-3-vision https://huggingface.co/microsoft/Phi-3-vision-128k-instruct

but for now you will have to use python.

You can try it here https://ai.azure.com/explore/models/Phi-3-vision-128k-instru... to get an idea of its OCR + QA abilities

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#19

Post by ekianjo »

pierre wrote: post #15

RAG cli from llamaindex, allow you to do it 100% locally when used with ollama or llamacpp instead of OpenAI. https://docs.llamaindex.ai/en/stable/getting_started/starter...

llamaindex has an horrible API, very poor docs and is constantly changing. I do not recommend it.

Re: Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

#20

Post by jd3 »

pierre wrote: post #15

RAG cli from llamaindex, allow you to do it 100% locally when used with ollama or llamacpp instead of OpenAI. https://docs.llamaindex.ai/en/stable/getting_started/starter...

basically, still the same answer(s) from

https://news.ycombinator.com/item?id=38759877

https://news.ycombinator.com/item?id=36832572

Post reply on HN