Earlier quoted context omitted.
Let me look into this
How about jpegs or other scanner images files? We have hundreds of scanned documents that were never pdf wrapped.
hmmmm :)
Turning a pile of documents into a searchable useable knowledge base
51–58 of 58 posts
Re: Turning a pile of documents into a searchable useable knowledge base
#52We need projects like this. Automatically classifying the files is smart. I'm working on a similar application called Hister ( https://github.com/asciimoo/hister ). I should borrow some of your ideas. =]
I just had a wild thought. Combine Hister with my RepoSearch app. Point it at a companies Internal github/gitlab and have a searchable knowledge base of your git repos.
Re: Turning a pile of documents into a searchable useable knowledge base
#53Earlier quoted context omitted.
Let me look into this
I've been working on something related - extracting tons of data from various formats to allow searching them - and the solution I chose for xlxs and xls files was headless LibreOffice to convert them to CSV. There's also exceljs but I found it didn't work for many old xls files. I didn't find screenshotting of spreadsheets worked well, vision wasn't very accurate on them. I do use it for PDFs though. For docx it's p…
My take was on pictures embedded into those documents, I'm not sure screenshotting would help as the text/numeric data is already there. Just saying.
Re: Turning a pile of documents into a searchable useable knowledge base
#54Earlier quoted context omitted.
Personal use? I need this at work, dragging useful info from tarpits like Teams and GitLab. Also need to search git repos including all branches and history (TIL/xkcd#153'd GitLab's web search can basically only do one branch at a time).
But how’d you access teams when it’s work teams and don’t have api access ?
Microsoft Graph API
Re: Turning a pile of documents into a searchable useable knowledge base
#55We need projects like this. Automatically classifying the files is smart. I'm working on a similar application called Hister ( https://github.com/asciimoo/hister ). I should borrow some of your ideas. =]
I just had a wild thought. Combine Hister with my RepoSearch app. Point it at a companies Internal github/gitlab and have a searchable knowledge base of your git repos.
I like the idea. Could you share your RepoSearch app? Also, we have Discord & IRC channels, please join and start brainstorming.
Re: Turning a pile of documents into a searchable useable knowledge base
#56Many such open source projects already (which is fantastic). I lose track of them.
Today it happened I needed a simple way to embed & query 1TB+ of documents, and I was looking at open source options. Can anyone tell me what their go-to solution is now? Could this be the one?
And what are the key differences vs. other open source RAG tools like kotaemon?