Bleve: full-text search and indexing for Go • Hacker News Board

Re: Bleve: full-text search and indexing for Go

#11

Post by mschoch » Sun, Jan 07, 2018, 4:06 AM UTC

wejick wrote: post #10

mschoch wrote: post #9

Earlier quoted context omitted.

Bleve has support for querying across multiple indexes (shards) but does not prescribe any mechanism to split the data. So, it's up the application to divide the data how it sees fit, but you can use Bleve functionality to execute the same query across multiple indexes and merge the results. Merging is required and is indeed somewhat resource intensive. Bleve's current indexing approach has no segments, instead all i…

Yes and if I'm not mistaken that's what alias on bleve for. Not yet looking into scorch. So scorch would replace other storage engine like rocksdb and leveldb?

Yes, when you create a new index, you can choose the index implementation. The previous one was called 'upsidedown', and this one used key/value stores for the actual storage. So, you would also specify BoltDB/RocksDB/LevelDB/Moss/etc. The new index implementation is called 'scorch' and it writes directly to disk instead of going through a key/value store.

Re: Bleve: full-text search and indexing for Go

#12

Post by billconan » Sun, Jan 07, 2018, 4:54 AM UTC

is there a c++ library like this?

Re: Bleve: full-text search and indexing for Go

#13

Post by bogomipz » Sun, Jan 07, 2018, 4:58 AM UTC

wejick wrote: post #6

I am using elasticsearch on production with golang as the both indexer and search service. Last holiday I played with blevesearch to make it works like Elasticsearch search, but the work is far far far from complete. https://github.com/wejick/balasticsearch

>"I am using elasticsearch on production with golang as the both indexer and search service."

Can you describe how this golang plus elasticsarch hybrid works? Is this is distinct from the link you provided?

Re: Bleve: full-text search and indexing for Go

#14

Post by latenightcoding » Sun, Jan 07, 2018, 5:48 AM UTC

billconan wrote: post #12

is there a c++ library like this?

There is a C library: https://lucy.apache.org/

Re: Bleve: full-text search and indexing for Go

#15

Post by wejick » Sun, Jan 07, 2018, 7:03 AM UTC

bogomipz wrote: post #13

wejick wrote: post #6

I am using elasticsearch on production with golang as the both indexer and search service. Last holiday I played with blevesearch to make it works like Elasticsearch search, but the work is far far far from complete. https://github.com/wejick/balasticsearch

>"I am using elasticsearch on production with golang as the both indexer and search service." Can you describe how this golang plus elasticsarch hybrid works? Is this is distinct from the link you provided?

Basically my applications are consuming elasticsearch REST API. One acts as indexer, the other one is using search API. I'm using this amazing package https://github.com/olivere/elastic

Re: Bleve: full-text search and indexing for Go

#16

Post by wejick » Sun, Jan 07, 2018, 7:04 AM UTC

bogomipz wrote: post #13

wejick wrote: post #6

I am using elasticsearch on production with golang as the both indexer and search service. Last holiday I played with blevesearch to make it works like Elasticsearch search, but the work is far far far from complete. https://github.com/wejick/balasticsearch

>"I am using elasticsearch on production with golang as the both indexer and search service." Can you describe how this golang plus elasticsarch hybrid works? Is this is distinct from the link you provided?

Yes it is different from balasticsearch

Re: Bleve: full-text search and indexing for Go

#17

Post by hroman » Sun, Jan 07, 2018, 8:27 AM UTC

billconan wrote: post #12

is there a c++ library like this?

Xapian https://xapian.org/

Re: Bleve: full-text search and indexing for Go

#18

Post by ddorian43 » Sun, Jan 07, 2018, 10:53 AM UTC

billconan wrote: post #12

is there a c++ library like this?

modern C++ https://github.com/phaistos-networks/Trinity

Re: Bleve: full-text search and indexing for Go

#19

Post by shabbyrobe » Sun, Jan 07, 2018, 11:28 AM UTC

I ran into some significant performance issues with Bleve during a weekend hackathon a few months ago. I was trying to index the stackoverflow data dump for fun and I couldn't get it to successfully complete. I'm guessing i was running into some boltdb related limitations but I didn't have the time to dig deeper before the party ended and I had to get back to the day job.

SQLite's FTS5 allowed me to load the entire data set without breaking a sweat, but query performance was unexpectedly poor for some combinations of terms with no apparent consistency, and it became unusable due to a temp table I couldn't work out how to avoid when attempting to search in descending primary key order.

It was many months ago now and I have forgotten some of the particulars but thought it might make an interesting jumping-off point for discussion about any ideas anyone might have for indexing 60gb of data into a flat file setup like Bleve or SQLite uses.

Would the new scorch experiment perform better with an index of that size than the boltdb backend?

Re: Bleve: full-text search and indexing for Go

#20

Post by oelmekki » Sun, Jan 07, 2018, 12:17 PM UTC

mschoch wrote: post #2

Well since someone submitted us here with no apparent reason or context, allow me to provide something of interest. (primary contributor of bleve here) Just recently we merged support a new experimental index scheme called 'scorch'. This new index scheme is designed from the ground up to reduce index size and improve performance. It features: - a segment based approach, much like Lucene - vellum FTS for the term dict…

Hi,

There's a question I don't find an answer for by looking at homepage and documentation: does it handle concurrent queries? (I wonder, since I see a store is a single file)

That is, is it something akin sqlite, meant to be used as an embedded engine for standalone applications, or is it fit to be used by a centralized api?