Turning a pile of documents into a searchable useable knowledge base • Hacker News Board

Re: Turning a pile of documents into a searchable useable knowledge base

#31

Post by gatnoodle » Thu, Jul 09, 2026, 8:05 AM UTC

linuxrebe1 wrote: post #2

I had an issue. A documents folder with over 12k objects in it. A hodgepodge of folders and sub-folders. That over time had created a mess that no amount of file movement was ever going to make it usable. I wanted: 1) To keep my data local 2) be able to filter out PII and other data 3) Be able to find and delete duplicates 4) Get short synopsis of what a document is 5) Semantic and keyword search 6) All of this kept…

This looks really cool. Can you tell me the minimum specs required to run this? It would nice if you could add it to the readme as well.

Re: Turning a pile of documents into a searchable useable knowledge base

#32

Post by esperent » Thu, Jul 09, 2026, 8:51 AM UTC

linuxrebe1 wrote: post #9

bobim wrote: post #3

Earlier quoted context omitted.

Could it be extended so it also extracts pictures from pptx and xlsx and run vision to get a description to be added to the text content before indexing?

Let me look into this

I've been working on something related - extracting tons of data from various formats to allow searching them - and the solution I chose for xlxs and xls files was headless LibreOffice to convert them to CSV. There's also exceljs but I found it didn't work for many old xls files.

I didn't find screenshotting of spreadsheets worked well, vision wasn't very accurate on them. I do use it for PDFs though. For docx it's probably fine either way but I went with LibreOffice -> markdown.

Re: Turning a pile of documents into a searchable useable knowledge base

#33

Post by kamranjon » Thu, Jul 09, 2026, 8:57 AM UTC

Wanted to share Antfly which I think serves a similar niche:

https://antfly.io/

https://github.com/antflydb/antfly

They’ve put a lot of effort into optimizing the local llm pipelines and I have a lot of faith in the devs working on it.

Re: Turning a pile of documents into a searchable useable knowledge base

#34

Post by appstorelottery » Thu, Jul 09, 2026, 10:15 AM UTC

Anyone getting a bunch of permission errors when running (e.g. Traceback (most recent call last): File "/Users/tron/Applications/DocuBrowse/venv/lib/python3.9/site-packages/psutil/_psosx.py", line 347, in wrapper return fun(self, args, *kwargs) File "/Users/tron/Applications/DocuBrowse/venv/lib/python3.9/site-packages/psutil/_psosx.py", line 508, in net_connections rawlist = cext.proc_net_connections(self.pid, families, types) PermissionError: [Errno 1] Operation not permitted (originated from proc_pidinfo(PROC_PIDLISTFDS) 1/2)

Re: Turning a pile of documents into a searchable useable knowledge base

#35

Post by appstorelottery » Thu, Jul 09, 2026, 10:19 AM UTC

appstorelottery wrote: post #34

Anyone getting a bunch of permission errors when running (e.g. Traceback (most recent call last): File "/Users/tron/Applications/DocuBrowse/venv/lib/python3.9/site-packages/psutil/_psosx.py", line 347, in wrapper return fun(self, args, *kwargs) File "/Users/tron/Applications/DocuBrowse/venv/lib/python3.9/site-packages/psutil/_psosx.py", line 508, in net_connections rawlist = cext.proc_net_connections(self.pid, famili…

Living in bizarro world of AI. Install open source project, fails, feed into OpenCode w/DeepSeekFlash 4 -> feed error into it get fixed.

The kill_port function only catches ImportError from the psutil block, so when psutil is installed but raises AccessDenied (common on macOS), it crashes instead of falling back to lsof.

In platform_paths.py - add two lines after line 250:

except psutil.Error: pass

Fixed. Now when psutil raises AccessDenied (as happens on macOS without elevated privileges), it falls through to the lsof/fuser fallback instead of crashing. Try docubrowser start again.

Re: Turning a pile of documents into a searchable useable knowledge base

#36

Post by appstorelottery » Thu, Jul 09, 2026, 10:32 AM UTC

appstorelottery wrote: post #35

appstorelottery wrote: post #34

Anyone getting a bunch of permission errors when running (e.g. Traceback (most recent call last): File "/Users/tron/Applications/DocuBrowse/venv/lib/python3.9/site-packages/psutil/_psosx.py", line 347, in wrapper return fun(self, args, *kwargs) File "/Users/tron/Applications/DocuBrowse/venv/lib/python3.9/site-packages/psutil/_psosx.py", line 508, in net_connections rawlist = cext.proc_net_connections(self.pid, famili…

Living in bizarro world of AI. Install open source project, fails, feed into OpenCode w/DeepSeekFlash 4 -> feed error into it get fixed. The kill_port function only catches ImportError from the psutil block, so when psutil is installed but raises AccessDenied (common on macOS), it crashes instead of falling back to lsof. In platform_paths.py - add two lines after line 250: except psutil.Error: pass Fixed. Now when ps…

Disappointed that it wasn't returning a list of paragraphs from eBooks that semantically match; search only appears to list the publications - not the actual match within the document.

Re: Turning a pile of documents into a searchable useable knowledge base

#37

Post by Ozzie_osman » Thu, Jul 09, 2026, 11:13 AM UTC

This is really cool. Can it play nice with gdrive or Dropbox? For better or worse, that's just where my data lives now but I'd love this layer.

Re: Turning a pile of documents into a searchable useable knowledge base

#38

Post by fnordian » Thu, Jul 09, 2026, 1:31 PM UTC

linuxrebe1 wrote: post #2

I had an issue. A documents folder with over 12k objects in it. A hodgepodge of folders and sub-folders. That over time had created a mess that no amount of file movement was ever going to make it usable. I wanted: 1) To keep my data local 2) be able to filter out PII and other data 3) Be able to find and delete duplicates 4) Get short synopsis of what a document is 5) Semantic and keyword search 6) All of this kept…

It’s either restricted to personal use, or it’s GPL-3. How can you have both?

Re: Turning a pile of documents into a searchable useable knowledge base

#39

Post by hunmernop » Thu, Jul 09, 2026, 2:18 PM UTC

Can you make a dockerfile and docker compose file?

Re: Turning a pile of documents into a searchable useable knowledge base

#40

Post by linuxrebe1 » Thu, Jul 09, 2026, 4:47 PM UTC

appstorelottery wrote: post #36

appstorelottery wrote: post #35

Earlier quoted context omitted.

Living in bizarro world of AI. Install open source project, fails, feed into OpenCode w/DeepSeekFlash 4 -> feed error into it get fixed. The kill_port function only catches ImportError from the psutil block, so when psutil is installed but raises AccessDenied (common on macOS), it crashes instead of falling back to lsof. In platform_paths.py - add two lines after line 250: except psutil.Error: pass Fixed. Now when ps…

Disappointed that it wasn't returning a list of paragraphs from eBooks that semantically match; search only appears to list the publications - not the actual match within the document.

Noted the bug.