Optimizing Solr (Or How To 7x Your Search Speed) • Hacker News Board

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#31

Post by chenli » Mon, Mar 26, 2012, 3:00 AM UTC

nkurz wrote: post #28

chenli wrote: post #22

This is the founder of Bimaple. We provide hosted search and license our engine software with significantly better performance and capabilities than Lucene: (1) Supporting "Google-Instant" search experiences on your data; (2) Powerful error correction by doing fuzzy search; (3) Optimized for mobile users by doing instant fuzzy search with a speed 10x-100x higher than Lucene; (4) Optimized for geo-location apps; (5) D…

Hi Chen, It seems like an interesting product. Advertising here on HN is always tricky --- it's a balancing act between self-promotion and restraint. I voted your comment up in the hope that you'll stick around and tell us more about it. But perhaps more on the tech side, and less on the marketing. Have you published any papers describing your approach? Or white papers with more meaty details? I work on the Apache Lu…

Hi nkurz,

Thanks for voting my comment up. Based on your request, I provide the following information. Let me know if you want to move this discussion to a private email discussion.

We have two published white papers to compare our engine with Lucene/Sor:

(1) A comparison white paper in the case of traditional keyword search: http://www.bimaple.com/files/Bimaple-Keyword-Search-Comparis... . We have a demonstration site to support instant, fuzzy search on Stack Overflow data (600K question titles as of January 2011): http://demo.bimaple.com/stackoverflow

(2) A comparison white paper in the case of geo-location keyword search: http://www.bimaple.com/files/Bimaple-Map-Search-White-Paper-... . We have a demonstration site to support instant, fuzzy search on 17 million US business listings: http://www.omniplaces.com . It has an iphone app at: http://itunes.apple.com/us/app/omniplaces/id466162583?mt=8

If you have questions, please feel free to contact me at chenli AT bimaple DOT com. Thank you.

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#32

Post by astubbs » Tue, Mar 27, 2012, 1:47 PM UTC

gpapilion wrote: post #30

I'm curious what your queries look like, because these performance numbers are awful. I'm currently running an index that is 96 million documents(393GB) using a single shard with a response time of 18ms. If you're comfortable with it, I'd suggest profiling Solr. We found that we were spending more time garbage collecting than expected, and spent some time to speed up an minimize the impact of it. Most of this was rel…

Re GC - were you using standard or parallel GC?

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#33

Post by falcolas » Tue, Mar 27, 2012, 8:21 PM UTC

raccoonone wrote: post #13

falcolas wrote: post #10

Our company had to set up a Solr implementation with some pretty crazy requirements (hundreds of shards, tens of thousands of requests per second, etc), and we ended up with 4 machines - one for indexing, one as a backup indexer/searcher, and 2 just doing load balanced searches. Replication was interesting but easy to set up (since it's basically an rsync of the indexes between servers). The end result works very wel…

What was the reason for having hundreds of shards on each server? Were you still seeing performance benefits to sharding it that aggresively?

Multi-tennent data separation required by contract.