Bleve: Full-text search and indexing for Go

First, here are the top two requests that we do NOT plan to implement.

1. Make Bleve a distributed index, or make Bleve into something that is a more direct ES competitor.

We have no plans to do this because we think that is better built at a different layer. We have hooks we introduce in certain places where we need to plug-in code that would otherwise violate the boundaries. And that is an arrangement that has worked well so far. There are multiple projects built on top of bleve that allow you to index/search across nodes.

2. Make an adapter for the XYZ key/value store.

This request goes back to the original bleve index which is serialized into a key/value abstraction layer. When users run into size/speed issues with bleve, many assume that just plugging in a faster key/value store will help. (Hey we thought that too when we built it this way)

But, we've now replaced that index scheme with a new implementation called scorch. Scorch is considerably smaller and faster, and manages it's own index on disk, without using any key/value store.

As for things that we DO plan to implement:

1. Size of the index still comes up a lot. Couchbase is a very performnace sensitive user of Bleve, so I expect they'll lead the way on this front.

2. Better (pluggable) scoring. Today our search result scoring is broken for several types of queries, and the stuff that does score right is too tightly coupled to the searching logic.

3. Overhaul index mapping. Today bleve uses a mapping object to describe how source objects/documents are indexed. One of the best ways we can simplify the mapping is to make things more explicit. I think we tried to embrace the concept of reasonable defaults, but we ended up with inheritance hierarchies that are difficult to reason about.

There are lots of miscellaneous things like adding a data type that supports IPv6, or more advanced queries (lots of variations on span queries).

We were made aware of this in September: https://github.com/blevesearch/bleve/issues/783#issuecomment...

Who reads docs?

We recently added an ASCII folding filter, which may help: https://github.com/blevesearch/bleve/pull/1070

Really nice to see the Go ecosystem developing. The other day I was searching for a link/URL extraction library and the best one I could find was in Go: https://github.com/mvdan/xurls ("best" because it actually uses a list of TLDs, for example: https://github.com/mvdan/xurls/blob/master/tlds.go). Was an unusual experience not finding something as good for Java.