Lucene: The Good Parts
blog.parsely.com
Lucene: The Good Parts
1–10 of 19 posts
Re: Lucene: The Good Parts
#2[deleted]
Re: Lucene: The Good Parts
#3Lucene is quite fantastic and Elasticsearch makes it a joy to use.
Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some parts with C++ and saw massive gains (but with a disclaimer that this was in no way indicative that Java wasn't fast).
I've considered trying to implement something like Lucene in, say, Rust, but then I see just how utterly massive Lucene is. Just the fuzzy search part alone required implementing code to generate code from a Russian PhD thesis they didn't fully understand.[1] So, no matter how many cycles the JVM is needlessly burning, Lucene just seems to advanced to write it without the overhead. (And maybe my intuition is just wrong and the overhead is only a few percent.)
1: http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is...
Re: Lucene: The Good Parts
#4Lucene is quite fantastic and Elasticsearch makes it a joy to use. Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some parts…
That fuzzy search story is more worrying than anything else, really. What they did seems just crazy to me.
First, the paper really doesn't seem so difficult. Second, they don't even think about reaching out to the author/s?
I suppose I shouldn't talk until I've tried their task. But I've implemented a lot of algorithms from papers, and their story had me shaking my head.
Re: Lucene: The Good Parts
#5Lucene is quite fantastic and Elasticsearch makes it a joy to use. Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some parts…
That fuzzy search story is more worrying than anything else, really. What they did seems just crazy to me. First, the paper really doesn't seem so difficult. Second, they don't even think about reaching out to the author/s? I suppose I shouldn't talk until I've tried their task. But I've implemented a lot of algorithms from papers, and their story had me shaking my head.
I've also implemented a lot of algorithms from papers, and their story has me nodding my head in agreement. Some algorithms papers are just downright impenetrable.
Re: Lucene: The Good Parts
#6Most devs don't know that there is CLucene, a C++ port of the Java based Lucene. It's lacking devs, so it's some versions behind. An alternative to CLucene and also native C++ is Sphinx search (similar to CLucene what Nginx is to Apache). Also SQLite has an official full text search addon named FTS4.
Lucene in Action 1st and 2nd edition are great books, I have them both. The first edition was like the missing manual and it covers the API of the Lucene 1.4 with its rather rough API objects. Lucene 2.x+ API improved a lot.
Re: Lucene: The Good Parts
#7Lucene is quite fantastic and Elasticsearch makes it a joy to use. Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some parts…
There is a port of Lucene to C++, CLucene[1], it's compatible with version 2.3 of Java Lucene, the project is stopped long time ago, but it's very much stable, and works perfectly.
An other port which is compatible with version 3 of java Lucene is LucenePlusPlus, but it use a lot of boost's smart pointers, the port seems like t was automated. This port was why CLucene development stopped, the maintainers wanted to make this new port faster by not using smart pointers whenever possible, but that didn't happen.
1: http://sourceforge.net/projects/clucene 2: https://github.com/luceneplusplus/LucenePlusPlus
Re: Lucene: The Good Parts
#8Most devs don't know that there is CLucene, a C++ port of the Java based Lucene. It's lacking devs, so it's some versions behind. An alternative to CLucene and also native C++ is Sphinx search (similar to CLucene what Nginx is to Apache). Also SQLite has an official full text search addon named FTS4. Lucene in Action 1st and 2nd edition are great books, I have them both. The first edition was like the missing manual…
there is also LucenePlusPlus:
https://github.com/luceneplusplus/LucenePlusPlus
Re: Lucene: The Good Parts
#9Lucene is quite fantastic and Elasticsearch makes it a joy to use. Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some parts…
There is a port of Lucene to C++, CLucene[1], it's compatible with version 2.3 of Java Lucene, the project is stopped long time ago, but it's very much stable, and works perfectly. An other port which is compatible with version 3 of java Lucene is LucenePlusPlus, but it use a lot of boost's smart pointers, the port seems like t was automated. This port was why CLucene development stopped, the maintainers wanted to ma…
Oddly enough, I don't see anyone talking about benchmarks for those projects. I found one offhand comment saying it was 2-3 faster than Java for indexing, but only 10% better for search. No real benchmarks or such. I suppose that's not the only reason to want a non-JVM version but it seems like a pretty major reason and something that'd warrant headline treatment in the readme...
Re: Lucene: The Good Parts
#10Lucene is quite fantastic and Elasticsearch makes it a joy to use. Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some parts…
Java overhead is not a huge issue unless you are embedding Lucene into some low spec devices. A consumer search system is usually relying heavily on cache (just like any databases), so even a 30-50% latency hit on cold queries is not that big a deal if > 90% of your queries are served from cache.
GC is a big problem when you don't know the expected query distribution which is the case for Elasticsearch's analytics. There is a lot more to a search engine than packing, decoding and merging posting lists. I've never seen anything that compares with Lucene text analysis and scoring API supports.