Live data from Hacker News

Optimizing Solr (Or How To 7x Your Search Speed)

carsabi.com

21–30 of 33 posts

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#21
post #20

I improved solr performance by switching to elasticsearch. =)

Same here. Just make sure to set your inital shards correctly so that you can grow for a while. Once you have to increase shards you have to re-index :(. But that's fine. Just go with 4x or more the shards you need now and you'll be able to scale out to that many boxes.

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#22
This is the founder of Bimaple. We provide hosted search and license our engine software with significantly better performance and capabilities than Lucene: (1) Supporting "Google-Instant" search experiences on your data; (2) Powerful error correction by doing fuzzy search; (3) Optimized for mobile users by doing instant fuzzy search with a speed 10x-100x higher than Lucene; (4) Optimized for geo-location apps; (5) Designed and developed ground up using C++. We have demonstrations on our homepage. If interested in using our service or software, please email contact AT bimaple.com.

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#26
post #24
post #23

anybody can recommend a good way to get startet with Solr ?

If you're using Django, Haystack makes it trivially easy to set up a Solr index against your existing Django models. http://haystacksearch.org/

in a similar vein, if using ruby/rails Sunspot(sunspot_rails)[1] is awesome. http://sunspot.github.com/

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#27
post #20

I improved solr performance by switching to elasticsearch. =)

it would be actually interesting and make the comment more useful if you'd share your own benchmark or real life anecdote about this, otherwise it's trollish.

(I'd expect performance between ES and Solr to be about the same and highly dependent on the underlying lucene engine)

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#28
post #22

This is the founder of Bimaple. We provide hosted search and license our engine software with significantly better performance and capabilities than Lucene: (1) Supporting "Google-Instant" search experiences on your data; (2) Powerful error correction by doing fuzzy search; (3) Optimized for mobile users by doing instant fuzzy search with a speed 10x-100x higher than Lucene; (4) Optimized for geo-location apps; (5) D…

Hi Chen,

It seems like an interesting product. Advertising here on HN is always tricky --- it's a balancing act between self-promotion and restraint. I voted your comment up in the hope that you'll stick around and tell us more about it. But perhaps more on the tech side, and less on the marketing.

Have you published any papers describing your approach? Or white papers with more meaty details? I work on the Apache Lucy project, and am very interested in things that work better than Lucene.

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#29
One thing I think would be valuable to know here is how many threads each shard is using, and what effect changing that number would have.

(rather: why is it useful to explicitly shard vs running one big instance with all of the memory and the same total number of threads? queuing theory would lead me to believe the latter would be better)

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#30
I'm curious what your queries look like, because these performance numbers are awful.

I'm currently running an index that is 96 million documents(393GB) using a single shard with a response time of 18ms.

If you're comfortable with it, I'd suggest profiling Solr. We found that we were spending more time garbage collecting than expected, and spent some time to speed up an minimize the impact of it. Most of this was related to our IO though.

Second, don't use the default settings. Adjust the cache sizes, rambuffer, and other settings so they are appropriate for you application.

I'd also start instrumenting you web application such that you can start testing removal of query options that may be creating your CPU usage issue. You get a lot of bang for your buck this way, and you may find the options you were using provide no meaningful improvement in search. A metric like mean reciprocal rank can go a long way to improve your performance.

Post reply on HN