Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turning a pile of documents into a searchable useable knowledge base

github.com

Post reply on HN
Print view View original

11–20 of 58 posts

Re: Turning a pile of documents into a searchable useable knowledge base

#12

Post by password4321 »

linuxrebe1 wrote: post #2

I had an issue. A documents folder with over 12k objects in it. A hodgepodge of folders and sub-folders. That over time had created a mess that no amount of file movement was ever going to make it usable. I wanted: 1) To keep my data local 2) be able to filter out PII and other data 3) Be able to find and delete duplicates 4) Get short synopsis of what a document is 5) Semantic and keyword search 6) All of this kept…

Personal use? I need this at work, dragging useful info from tarpits like Teams and GitLab.

Also need to search git repos including all branches and history (TIL/xkcd#153'd GitLab's web search can basically only do one branch at a time).

Re: Turning a pile of documents into a searchable useable knowledge base

#13

Post by clif_mcIrvin »

linuxrebe1 wrote: post #9
bobim wrote: post #3

Earlier quoted context omitted.

Could it be extended so it also extracts pictures from pptx and xlsx and run vision to get a description to be added to the text content before indexing?

Let me look into this

How about jpegs or other scanner images files? We have hundreds of scanned documents that were never pdf wrapped.

Re: Turning a pile of documents into a searchable useable knowledge base

#16

Post by vsviridov »

seb1204 wrote: post #8
linuxrebe1 wrote: post #2

I had an issue. A documents folder with over 12k objects in it. A hodgepodge of folders and sub-folders. That over time had created a mess that no amount of file movement was ever going to make it usable. I wanted: 1) To keep my data local 2) be able to filter out PII and other data 3) Be able to find and delete duplicates 4) Get short synopsis of what a document is 5) Semantic and keyword search 6) All of this kept…

Sounds similar to https://docs.paperless-ngx.com/ Key difference I see is that you point it to a folder instead of uploading to a system.

I think paperless devs are working on AI integration, and there are 3rd party solutions. I'm holding out for an official one, so far.

It's pretty cool, I've set up a share where the scanner scans, and it automatically picks it up from there and ingests it into the system.

Re: Turning a pile of documents into a searchable useable knowledge base

#18

Post by hankbond »

asciimoo wrote: post #5

We need projects like this. Automatically classifying the files is smart. I'm working on a similar application called Hister ( https://github.com/asciimoo/hister ). I should borrow some of your ideas. =]

I have not set up Hister yet but it's on my list to try out. How would I do something like host it on my Unraid box but have it index/persist my local MacBook browsing history?

Re: Turning a pile of documents into a searchable useable knowledge base

#19

Post by linuxrebe1 »

password4321 wrote: post #12
linuxrebe1 wrote: post #2

I had an issue. A documents folder with over 12k objects in it. A hodgepodge of folders and sub-folders. That over time had created a mess that no amount of file movement was ever going to make it usable. I wanted: 1) To keep my data local 2) be able to filter out PII and other data 3) Be able to find and delete duplicates 4) Get short synopsis of what a document is 5) Semantic and keyword search 6) All of this kept…

Personal use? I need this at work, dragging useful info from tarpits like Teams and GitLab. Also need to search git repos including all branches and history (TIL/xkcd#153'd GitLab's web search can basically only do one branch at a time).

I creating DocuRepo as well. though not as fleshed out.

Re: Turning a pile of documents into a searchable useable knowledge base

#20

Post by linuxrebe1 »

jphorism wrote: post #11

Nice, what are you hoping to accomplish with this project?

- Filling a need I personally have. - Learning how to leverage AI for real world use not just to fill up a data center. - Personal knowledge -developing skills

Pretty much in that order

Post reply on HN