Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xapian: Open source search engine library

xapian.org

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Xapian: Open source search engine library

#31

Post by theolivenbaum »

rbanffy wrote: post #22

I remember having used, a very long time ago, a self-hosted search engine on my library of PDFs, and it was unbelievably useful. I dream about a similar thing that can do OCR on scanned docs and extract text from my also sprawling library of epub and mobi files. If someone builds something like this, with maybe a LOCAL LLM to extract text descriptions from photos and movies as well as indexing metadata for everything…

That's what our app does: curiosity.ai, local index, support for many files types and apps out of the box, and integrated local OCR, STT and even local LLM

Re: Xapian: Open source search engine library

#32

Post by Bluestein »

theolivenbaum wrote: post #31
rbanffy wrote: post #22

I remember having used, a very long time ago, a self-hosted search engine on my library of PDFs, and it was unbelievably useful. I dream about a similar thing that can do OCR on scanned docs and extract text from my also sprawling library of epub and mobi files. If someone builds something like this, with maybe a LOCAL LLM to extract text descriptions from photos and movies as well as indexing metadata for everything…

That's what our app does: curiosity.ai, local index, support for many files types and apps out of the box, and integrated local OCR, STT and even local LLM

That is the future :) Much success!

Re: Xapian: Open source search engine library

#33

Post by pdw »

rbanffy wrote: post #22

I remember having used, a very long time ago, a self-hosted search engine on my library of PDFs, and it was unbelievably useful. I dream about a similar thing that can do OCR on scanned docs and extract text from my also sprawling library of epub and mobi files. If someone builds something like this, with maybe a LOCAL LLM to extract text descriptions from photos and movies as well as indexing metadata for everything…

> similar thing that can do OCR on scanned docs

It's only part of what you want, but ocrmypdf will add a OCRed text layer to PDF files, making the text selectable and indexable

Re: Xapian: Open source search engine library

#35

Post by rmholt »

theolivenbaum wrote: post #31
rbanffy wrote: post #22

I remember having used, a very long time ago, a self-hosted search engine on my library of PDFs, and it was unbelievably useful. I dream about a similar thing that can do OCR on scanned docs and extract text from my also sprawling library of epub and mobi files. If someone builds something like this, with maybe a LOCAL LLM to extract text descriptions from photos and movies as well as indexing metadata for everything…

That's what our app does: curiosity.ai, local index, support for many files types and apps out of the box, and integrated local OCR, STT and even local LLM

I couldn't find any mention on your website about LOCAL LLMs and according to your FAQ, it requires an account with your website.

Is there a way how to run curiosity.ai fully offline, without an account on your servers?

Re: Xapian: Open source search engine library

#38

Post by donio »

infocollector wrote: post #3

This project has been around and maintained for more than a decade! Small footprint, good speed. One downside might be GPL v2 for commercial use.

A lot more than a decade. I've been using it for 15 years at it was a very mature project even then. Repo history goes back to 1999 and according to the history page the project's roots go back to the 80s. A bit like Postgres in this respect.

https://xapian.org/history https://sigir.org/files/forum/S2000/MUSCAT_note.pdf

Re: Xapian: Open source search engine library

#39

Post by nanna »

nickpsecurity wrote: post #24
nanna wrote: post #13

Earlier quoted context omitted.

I use recoll to index and search thousands of pdfs. Because I always have the author name in the filename I can filter queries like this: Cybernetics OR steering filename:Heidegger ext:pdf It's an absolute power tool.

I do it like this: Title Year Author Name.pdf Same benefits as you mentioned. You can also filter by time that way.

My way is to do

year__author1_author2_author-n~~title-of-book~subtitle##tag1#tag-2#tagn.pdf

This means that files are automatically organised by year of publication,that I can search by tag name, and that I dont have to escape chars in the terminal. One day I hope to get round to building an Emacs mode to filter by the different elements.

Re: Xapian: Open source search engine library

#40

Post by donio »

nanna wrote: post #13
openrisk wrote: post #6

used also by recoll, the desktop search app: https://www.recoll.org/

I use recoll to index and search thousands of pdfs. Because I always have the author name in the filename I can filter queries like this: Cybernetics OR steering filename:Heidegger ext:pdf It's an absolute power tool.

If your PDFs have Author properties then you might be able to do "author:Heidegger" too. The Recoll PDF filter extracts some of these fields and if I remember right it can be configured to extract additional custom properties too.
Post reply on HN