Bleve: full-text search and indexing for Go • Hacker News Board

Re: Bleve: full-text search and indexing for Go

#31

Post by ddorian43 » Sun, Jan 07, 2018, 6:28 PM UTC

millisecond wrote: post #21

wejick wrote: post #8

Earlier quoted context omitted.

Are there any plan to have sharding mechanism implemented on the blevesearch? IMHO it should be on the application level, but having this on blevesearch would be good. Regarding segments based approach like lucene, means we will need to do segments merging which in my experience quite resource intensive. I don't actually know how blevesearch handles this prior segment based approach.

Not directly sharding but I've been working on a fork of Bleve that uses Cassandra as the backing store which allows for horizontal scaling of a single index: https://github.com/wrble/flock Still very much a work in progress but the core is functional.

see elassandra for something similar

Re: Bleve: full-text search and indexing for Go

#32

Post by shabbyrobe » Mon, Jan 08, 2018, 12:52 AM UTC

mschoch wrote: post #25

shabbyrobe wrote: post #19

I ran into some significant performance issues with Bleve during a weekend hackathon a few months ago. I was trying to index the stackoverflow data dump for fun and I couldn't get it to successfully complete. I'm guessing i was running into some boltdb related limitations but I didn't have the time to dig deeper before the party ended and I had to get back to the day job. SQLite's FTS5 allowed me to load the entire d…

Yeah, the old index format had many factors contributing to it taking up considerable space. As a single data point, we have a beer-search sample app, this includes a data directory with 29MB of JSON files. In the old index format, with mapping that did a realistic configuration of indexing many fields, and storing some of them, the bleve index size was over 200MB. With scorch the index size is 22MB. Query performanc…

Nice one! I'll keep my eyes peeled for a release and give it another crack. I think all those creaky old scripts still work.

Re: Bleve: full-text search and indexing for Go

#33

Post by leetcrew » Mon, Jan 08, 2018, 5:53 PM UTC

billconan wrote: post #12

is there a c++ library like this?

if all you care about are relatively simple queries in english, clucene or lucy could probably work for you.

AFAIK, there is no native C/C++ library that comes anywhere near modern lucene. for me the killer feature is that you get analyzers for many different languages out of the box with current lucene/elasticsearch.

Re: Bleve: full-text search and indexing for Go

#34

Post by roblaszczak » Mon, Jan 08, 2018, 10:39 PM UTC

lima wrote: post #23

I'm new to Go - but it looks like the example code completely ignores the error codes. Is that a common thing?

for examples like this it is much better to explicit ignore error:

_ = index.Index(identifier, your_data)

it will also pass https://github.com/kisielk/errcheck check

eventually we can just panic error

if err := index.Index(identifier, your_data); err != nil { panic(err) }