Lucene: The Good Parts • Hacker News Board

Re: Lucene: The Good Parts

#11

Post by proveanegative » Fri, Mar 13, 2015, 11:13 PM UTC

MichaelGG wrote: post #3

Lucene is quite fantastic and Elasticsearch makes it a joy to use. Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some parts…

Is there a self-contained alternative to ElasticSearch specifically? If there was one written in Go or otherwise statically linkable that would be great from a deployment standpoint. I could deal with somewhat worse performance in exchange for that.

Re: Lucene: The Good Parts

#12

Post by frik » Fri, Mar 13, 2015, 11:26 PM UTC

proveanegative wrote: post #11

MichaelGG wrote: post #3

Lucene is quite fantastic and Elasticsearch makes it a joy to use. Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some parts…

Is there a self-contained alternative to ElasticSearch specifically? If there was one written in Go or otherwise statically linkable that would be great from a deployment standpoint. I could deal with somewhat worse performance in exchange for that.

Search for "golang full text search database".

Lucene & Hadoop meant a big push for the Java eco-system, it's like a lock-in. Native C++ libraries and other free text search implementations have a smaller community and are usually less known. With Go, C++11 and Rust the future looks bright but it will take some time to catch up.

Re: Lucene: The Good Parts

#13

Post by proveanegative » Fri, Mar 13, 2015, 11:35 PM UTC

frik wrote: post #12

proveanegative wrote: post #11

Earlier quoted context omitted.

Is there a self-contained alternative to ElasticSearch specifically? If there was one written in Go or otherwise statically linkable that would be great from a deployment standpoint. I could deal with somewhat worse performance in exchange for that.

Search for "golang full text search database". Lucene & Hadoop meant a big push for the Java eco-system, it's like a lock-in. Native C++ libraries and other free text search implementations have a smaller community and are usually less known. With Go, C++11 and Rust the future looks bright but it will take some time to catch up.

>Search "golang full text search database".

I have. The problem is picking one that is mature enough and will be supported for years as you can expect ElasticSearch to be, which is what I meant by "an alternative".

I agree with you about the future looking bright but I meant something you could use right now.

Re: Lucene: The Good Parts

#14

Post by syen » Sat, Mar 14, 2015, 12:08 AM UTC

frik wrote: post #12

proveanegative wrote: post #11

Earlier quoted context omitted.

Is there a self-contained alternative to ElasticSearch specifically? If there was one written in Go or otherwise statically linkable that would be great from a deployment standpoint. I could deal with somewhat worse performance in exchange for that.

Search for "golang full text search database". Lucene & Hadoop meant a big push for the Java eco-system, it's like a lock-in. Native C++ libraries and other free text search implementations have a smaller community and are usually less known. With Go, C++11 and Rust the future looks bright but it will take some time to catch up.

Agree, it's early days for non-Java based alternatives.

One of my colleagues, Marty Schoch, has been working on a full text search engine in golang, called bleve [1]

1: http://www.blevesearch.com/

Re: Lucene: The Good Parts

#15

Post by bkanber » Sat, Mar 14, 2015, 4:59 AM UTC

Great article. I've rolled my own full-text search engines in the past and it's a category of problems that I love, but even I have to admit that I'm often astounded by Lucene's performance. The inverted index really lets you stretch commodity hardware into pretty huge use-cases.

If you've never used ElasticSearch, I should note that that's one of ES's many strengths -- it takes advantage of Lucene and makes deployments on commodity hardware work really well. An ES cluster on five small EC2 instances can handle a tremendous workload.

There is one thing about ES/Lucene that bugs me though... in the 3+ years I've been running it in production, I still haven't been able to solve the "every once in a while java utilizes 100% CPU until you restart the service" issue. I suspect it has to do with Lucene's index merge operation, but no amount of tinkering has solved the problem.

Re: Lucene: The Good Parts

#16

Post by frik » Sat, Mar 14, 2015, 6:37 AM UTC

proveanegative wrote: post #13

frik wrote: post #12

Earlier quoted context omitted.

Search for "golang full text search database". Lucene & Hadoop meant a big push for the Java eco-system, it's like a lock-in. Native C++ libraries and other free text search implementations have a smaller community and are usually less known. With Go, C++11 and Rust the future looks bright but it will take some time to catch up.

>Search "golang full text search database". I have. The problem is picking one that is mature enough and will be supported for years as you can expect ElasticSearch to be, which is what I meant by "an alternative". I agree with you about the future looking bright but I meant something you could use right now.

It's hard to say, for Go there is e.g. bleve FTS and there are ports of Java Lucene to Go (e.g. https://github.com/balzaczyy/golucene). Such ports are either semi-automatic or automatic, only automatic ports. It's hard for Lucene ports to keep up, as Lucene is moving fast and most ports stalled.

One could also use a service oriented architecture and use e.g. ElasticSearch Rest API or C++ based Sphinx Search, both need litttle configuration and no custom code.

Re: Lucene: The Good Parts

#17

Post by mirad » Sat, Mar 14, 2015, 6:56 AM UTC

MichaelGG wrote: post #3

Lucene is quite fantastic and Elasticsearch makes it a joy to use. Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some parts…

> Lucene is quite fantastic and Elasticsearch makes it a joy to use.

I have to admit that this is new to me. I'm gonna check it out!

Re: Lucene: The Good Parts

#18

Post by darklajid » Sat, Mar 14, 2015, 8:54 AM UTC

MichaelGG wrote: post #3

Lucene is quite fantastic and Elasticsearch makes it a joy to use. Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some parts…

I tried implementing the same paper's algorithm in the past and somewhat succeeded - but gave up in the end. Precumputing the automatons was slow as hell, I came to a similar conclusion as the authors (N > 2 isn't really feasible, but was something I was interested in) and my plumbing sucked.

I'm really not experienced reading papers and this was the only one I ever tried, so I cannot compare it to others. It certainly was quite hard to follow for me and took some month of nightly dabbling before I reached the point above.

Re: Lucene: The Good Parts

#19

Post by _wmd » Sat, Mar 14, 2015, 2:22 PM UTC

bkanber wrote: post #15

Great article. I've rolled my own full-text search engines in the past and it's a category of problems that I love, but even I have to admit that I'm often astounded by Lucene's performance. The inverted index really lets you stretch commodity hardware into pretty huge use-cases. If you've never used ElasticSearch, I should note that that's one of ES's many strengths -- it takes advantage of Lucene and makes deployme…

One of the boons of Java is its remote debugging support.. you can attach a profiler to a process when something like this happens, extract thread names & stacks, and so on.

AFAIK you can also use the Linux 'perf trace' command on a Java process, but probably there is some more setup involved.