Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bleve: full-text search and indexing for Go

blevesearch.com

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Bleve: full-text search and indexing for Go

#21

Post by millisecond »

wejick wrote: post #8
mschoch wrote: post #2

Well since someone submitted us here with no apparent reason or context, allow me to provide something of interest. (primary contributor of bleve here) Just recently we merged support a new experimental index scheme called 'scorch'. This new index scheme is designed from the ground up to reduce index size and improve performance. It features: - a segment based approach, much like Lucene - vellum FTS for the term dict…

Are there any plan to have sharding mechanism implemented on the blevesearch? IMHO it should be on the application level, but having this on blevesearch would be good. Regarding segments based approach like lucene, means we will need to do segments merging which in my experience quite resource intensive. I don't actually know how blevesearch handles this prior segment based approach.

Not directly sharding but I've been working on a fork of Bleve that uses Cassandra as the backing store which allows for horizontal scaling of a single index: https://github.com/wrble/flock

Still very much a work in progress but the core is functional.

Re: Bleve: full-text search and indexing for Go

#25

Post by mschoch »

shabbyrobe wrote: post #19

I ran into some significant performance issues with Bleve during a weekend hackathon a few months ago. I was trying to index the stackoverflow data dump for fun and I couldn't get it to successfully complete. I'm guessing i was running into some boltdb related limitations but I didn't have the time to dig deeper before the party ended and I had to get back to the day job. SQLite's FTS5 allowed me to load the entire d…

Yeah, the old index format had many factors contributing to it taking up considerable space. As a single data point, we have a beer-search sample app, this includes a data directory with 29MB of JSON files. In the old index format, with mapping that did a realistic configuration of indexing many fields, and storing some of them, the bleve index size was over 200MB.

With scorch the index size is 22MB. Query performance is comparable (and we haven't even gotten to really tuning this yet).

Re: Bleve: full-text search and indexing for Go

#26

Post by mschoch »

st3fan wrote: post #24

What kind of index size does Bleve work with well? Megabytes? Gigabytes? I’d love to understand at what scale people are using this engine.

With the current index scheme we would regularly do 10s of gigabytes (which is embarrassingly small in our opinion). We haven't done large scale testing with scorch yet, but in some smaller configurations the new index is 1/10th the size of the previous one, so we hope to have moved the bar considerably on data sizes that work.

Re: Bleve: full-text search and indexing for Go

#27

Post by yalue »

lima wrote: post #23

I'm new to Go - but it looks like the example code completely ignores the error codes. Is that a common thing?

It isn't common, and the code example wouldn't even compile because "err" is assigned but never used. I imagine that they included the "err" variables to show that they're available, but didn't want to clutter the example with error checks.

You can ignore error codes in go by assigning them to the special variable "_", but, outside of very short toy examples, it is a huge warning sign for terrible code. It's certainly not common to ignore errors in Go.

Re: Bleve: full-text search and indexing for Go

#28

Post by mschoch »

lima wrote: post #23

I'm new to Go - but it looks like the example code completely ignores the error codes. Is that a common thing?

You are correct. I think when I originally designed the homepage I wanted to show how you index and search in just a few lines. I thought omitting the error handling boilerplate was acceptable, but properly handling these errors is also key for a good initial experience. So I'm persuaded to at least revisit the decision.

https://github.com/blevesearch/blevesearch.github.io-hugo/is...

Re: Bleve: full-text search and indexing for Go

#29

Post by mschoch »

oelmekki wrote: post #20
mschoch wrote: post #2

Well since someone submitted us here with no apparent reason or context, allow me to provide something of interest. (primary contributor of bleve here) Just recently we merged support a new experimental index scheme called 'scorch'. This new index scheme is designed from the ground up to reduce index size and improve performance. It features: - a segment based approach, much like Lucene - vellum FTS for the term dict…

Hi, There's a question I don't find an answer for by looking at homepage and documentation: does it handle concurrent queries? (I wonder, since I see a store is a single file) That is, is it something akin sqlite, meant to be used as an embedded engine for standalone applications, or is it fit to be used by a centralized api?

Concurrent queries are supported (not sure what you mean by store being a single file, it is a directory of many files).

Concurrent indexing is also possible, so long as you can arrange to not put duplicate document ids into batches executing concurrently.

As for usage, it is just a library, so it is typically embedded in a single process (though this can serve multiple clients concurrently).

Distributing the index across multiple nodes is done at the application level. At Couchbase we do this with bleve in a separate project called 'cbft'.

https://github.com/couchbase/cbft

Re: Bleve: full-text search and indexing for Go

#30

Post by oelmekki »

mschoch wrote: post #29
oelmekki wrote: post #20

Earlier quoted context omitted.

Hi, There's a question I don't find an answer for by looking at homepage and documentation: does it handle concurrent queries? (I wonder, since I see a store is a single file) That is, is it something akin sqlite, meant to be used as an embedded engine for standalone applications, or is it fit to be used by a centralized api?

Concurrent queries are supported (not sure what you mean by store being a single file, it is a directory of many files). Concurrent indexing is also possible, so long as you can arrange to not put duplicate document ids into batches executing concurrently. As for usage, it is just a library, so it is typically embedded in a single process (though this can serve multiple clients concurrently). Distributing the index a…

> (not sure what you mean by store being a single file, it is a directory of many files)

I see, my bad. I saw one binary file named "store" in the directory created by the example code, I thought it would be it.

Thanks for explanation!

Post reply on HN