Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turning a pile of documents into a searchable useable knowledge base

github.com

Post reply on HN
Print view View original

41–50 of 58 posts

Re: Turning a pile of documents into a searchable useable knowledge base

#41

Post by linuxrebe1 »

appstorelottery wrote: post #34

Anyone getting a bunch of permission errors when running (e.g. Traceback (most recent call last): File "/Users/tron/Applications/DocuBrowse/venv/lib/python3.9/site-packages/psutil/_psosx.py", line 347, in wrapper return fun(self, args, *kwargs) File "/Users/tron/Applications/DocuBrowse/venv/lib/python3.9/site-packages/psutil/_psosx.py", line 508, in net_connections rawlist = cext.proc_net_connections(self.pid, famili…

I was building for mac on a VM install of OSX. Will take this as a bug, Thanks.

Re: Turning a pile of documents into a searchable useable knowledge base

#43

Post by linuxrebe1 »

subhobroto wrote: post #25
linuxrebe1 wrote: post #22

Earlier quoted context omitted.

I'm actually thinking of this for a commercial product feature. However, if you use a tool like Rclone on Windows, Linux or Mac. Mount the s3 bucket and you can then run DocuBrowse as if the s3 bucket were local.

I love your project on many fronts. One, you're using Claude. Two, you used Python - but most importantly, you personally care about it. I will be using this, and I will be making contributions to it as well. > I'm actually thinking of this for a commercial product feature Would you consider writing down which features you would like to make commercial product features and how you would like to price them?

Consider it yes, However having experience in this ... not really. For now there is a file called Decisions.md in the repo that is my "notes to self" if you will about where and what I need to do.

Re: Turning a pile of documents into a searchable useable knowledge base

#44

Post by linuxrebe1 »

linuxrebe1 wrote: post #21
drizzler wrote: post #10

I just installed this and, after a few hiccups, got it up and running on my Ubuntu system. Works great, looks great. Thank you for this. Half of my documents are OpenDocument format. Is there any chance you'll be supporting ODF in the future?

Yes, not supporting it is an oversight I will correct.

Will have version 0.9.1 out later today to support ODF formats.

Re: Turning a pile of documents into a searchable useable knowledge base

#46

Post by linuxrebe1 »

esperent wrote: post #32
linuxrebe1 wrote: post #9

Earlier quoted context omitted.

Let me look into this

I've been working on something related - extracting tons of data from various formats to allow searching them - and the solution I chose for xlxs and xls files was headless LibreOffice to convert them to CSV. There's also exceljs but I found it didn't work for many old xls files. I didn't find screenshotting of spreadsheets worked well, vision wasn't very accurate on them. I do use it for PDFs though. For docx it's p…

I went with the python libraries (pydoc and pyxls for example), because it's portable and doesn't require a big download to a users system if they don't already have it installed.

Re: Turning a pile of documents into a searchable useable knowledge base

#47

Post by linuxrebe1 »

fnordian wrote: post #38
linuxrebe1 wrote: post #2

I had an issue. A documents folder with over 12k objects in it. A hodgepodge of folders and sub-folders. That over time had created a mess that no amount of file movement was ever going to make it usable. I wanted: 1) To keep my data local 2) be able to filter out PII and other data 3) Be able to find and delete duplicates 4) Get short synopsis of what a document is 5) Semantic and keyword search 6) All of this kept…

It’s either restricted to personal use, or it’s GPL-3. How can you have both?

By restricted for personal use I mean it's not networked. It's running on your system only. It's not a networked commercial product able to do SSO etc. It's not an enterprise level product.

Re: Turning a pile of documents into a searchable useable knowledge base

#48

Post by linuxrebe1 »

linuxrebe1 wrote: post #44
linuxrebe1 wrote: post #21

Earlier quoted context omitted.

Yes, not supporting it is an oversight I will correct.

Will have version 0.9.1 out later today to support ODF formats.

v0.9.1 is in the repo and packages have been built. It now does all of the ODT formats.

Re: Turning a pile of documents into a searchable useable knowledge base

#49

Post by linuxrebe1 »

appstorelottery wrote: post #35
appstorelottery wrote: post #34

Anyone getting a bunch of permission errors when running (e.g. Traceback (most recent call last): File "/Users/tron/Applications/DocuBrowse/venv/lib/python3.9/site-packages/psutil/_psosx.py", line 347, in wrapper return fun(self, args, *kwargs) File "/Users/tron/Applications/DocuBrowse/venv/lib/python3.9/site-packages/psutil/_psosx.py", line 508, in net_connections rawlist = cext.proc_net_connections(self.pid, famili…

Living in bizarro world of AI. Install open source project, fails, feed into OpenCode w/DeepSeekFlash 4 -> feed error into it get fixed. The kill_port function only catches ImportError from the psutil block, so when psutil is installed but raises AccessDenied (common on macOS), it crashes instead of falling back to lsof. In platform_paths.py - add two lines after line 250: except psutil.Error: pass Fixed. Now when ps…

I fixed this in version 0.9.1 (just released) thanks for the bug (seriously)

Re: Turning a pile of documents into a searchable useable knowledge base

#50

Post by linuxrebe1 »

gatnoodle wrote: post #31
linuxrebe1 wrote: post #2

I had an issue. A documents folder with over 12k objects in it. A hodgepodge of folders and sub-folders. That over time had created a mess that no amount of file movement was ever going to make it usable. I wanted: 1) To keep my data local 2) be able to filter out PII and other data 3) Be able to find and delete duplicates 4) Get short synopsis of what a document is 5) Semantic and keyword search 6) All of this kept…

This looks really cool. Can you tell me the minimum specs required to run this? It would nice if you could add it to the readme as well.

I've run it on a VM with 4G ram and no GPU. It runs, But I really recommend 8G ram at least. If you have a GPU (like I do) with 4G vRAM that is ideal. Will get this in the readme. Thanks for the suggestion. I really tried to build this to minimal spec.
Post reply on HN