FastMail's Email Search Architecture
blog.fastmail.com
FastMail's Email Search Architecture
1–10 of 32 posts
Re: FastMail's Email Search Architecture
#2Does anyone know why they chose Xapian over elasticsearch?
Re: FastMail's Email Search Architecture
#3If you want Xapian search on a local maildir, I highly recommend notmuch[1]. Adding new mail and updating the index can take noticable time, but searching is super fast, it allows easy custom tagging, and search results are better than gmail in my experience.
I use it from the emacs notmuch mode.
Re: FastMail's Email Search Architecture
#4Does anyone know why they chose Xapian over elasticsearch?
Xapian is to Cyrus what Lucene is to Elasticsearch - the search engine embedded into the larger server.
Re: FastMail's Email Search Architecture
#5Does anyone know why they chose Xapian over elasticsearch?
For me it would be the use of Java. I've just had too much bad luck with it. Admittedly that's not really very objective reasoning.
Re: FastMail's Email Search Architecture
#6If you want Xapian search on a local maildir, I highly recommend notmuch[1]. Adding new mail and updating the index can take noticable time, but searching is super fast, it allows easy custom tagging, and search results are better than gmail in my experience. I use it from the emacs notmuch mode. [1] http://notmuchmail.org/
Can notmuch do regular expression searches?
Re: FastMail's Email Search Architecture
#7If you want Xapian search on a local maildir, I highly recommend notmuch[1]. Adding new mail and updating the index can take noticable time, but searching is super fast, it allows easy custom tagging, and search results are better than gmail in my experience. I use it from the emacs notmuch mode. [1] http://notmuchmail.org/
Can notmuch do regular expression searches?
It can do simple wildcarding with the * character, but I'm not aware of any general regexp capability, same as with any indexed database. Since you still have the underlying maildirs you can always just use egrep I guess.
Re: FastMail's Email Search Architecture
#8Although in the past I've implemented Xapian [1] over Sphinx [2], Sphinx today seems to be much better, but both Xapian and Sphinx are under-appreciated compared to Solr [3] and Elasticsearch [4].
Re: FastMail's Email Search Architecture
#9Does anyone know why they chose Xapian over elasticsearch?
For me it would be the use of Java. I've just had too much bad luck with it. Admittedly that's not really very objective reasoning.
From memory (sorry, it was a while back) we actually started with Elasticsearch, but it was way to heavy and looked for an alternative solution. Even with a single user it was consuming way too much memory.
Re: FastMail's Email Search Architecture
#10An obvious question that I didn't hit on in the blog is "what about host crashes"? The nice thing is, every index knows exactly which messages it covers - and we can quite quickly (within an hour or so for an entire server) scan all mailboxes and index the missing messages - it's more efficient than doing it in a real time, because you are often indexing multiple messages in the same mailbox.
Once the indexes are up to date, we can switch back to being masters again. We index on all the replicas independently so that they are always ready.