Optimizing Solr (Or How To 7x Your Search Speed)
1–10 of 33 posts
Re: Optimizing Solr (Or How To 7x Your Search Speed)
#2I'd love to have them try out Searchify's hosted search and see how fast it is. The key to fast search is RAM, which is why we run our search indexes from RAM (not cheap), and most queries are served within 100ms. If you're the author of the blog post, please contact me, chris at searchify, if you'd like to do this comparison, and I'll set you up with a test acct.
Re: Optimizing Solr (Or How To 7x Your Search Speed)
#3I'd love to have them try out Searchify's hosted search and see how fast it is. The key to fast search is RAM, which is why we run our search indexes from RAM (not cheap), and most queries are served within 100ms. If you're the author of the blog post, please contact me, chris at searchify, if you'd like to do this comparison, and I'll set you up with a test acct.
I sure am the author. email sent :)
Re: Optimizing Solr (Or How To 7x Your Search Speed)
#4take a look at this presentation if you are interested in NRT Solr (although it was done before Solr added the latest NRT features):
Tuning Solr in Near Real Time Search environments: https://vimeo.com/17402451
Re: Optimizing Solr (Or How To 7x Your Search Speed)
#5My experience working with Solr is that a lot of the time people don't have a good working knowledge of how to optimize an index because it's so easy not to. At my last job, the initial implementation they we involved storing the full text of millions of documents, even though they never needed to be retrieved (just searched). If you're running Solr as a front-end search for another database, the best way I've seen to optimize performance is just to make sure you're not storing data unnecessarily.
Maybe everyone already should already know this, but I was working on a very smart team, and we totally missed this initially. Setting "stored" to false for most fields resulted in a 90% reduction of the index size, which means less to fit into RAM.
Re: Optimizing Solr (Or How To 7x Your Search Speed)
#6I'd love to have them try out Searchify's hosted search and see how fast it is. The key to fast search is RAM, which is why we run our search indexes from RAM (not cheap), and most queries are served within 100ms. If you're the author of the blog post, please contact me, chris at searchify, if you'd like to do this comparison, and I'll set you up with a test acct.
Hi Chris, what is Searchify's relationship with the old IndexTank (http://indextank.com) team?
Re: Optimizing Solr (Or How To 7x Your Search Speed)
#7My experience working with Solr is that a lot of the time people don't have a good working knowledge of how to optimize an index because it's so easy not to. At my last job, the initial implementation they we involved storing the full text of millions of documents, even though they never needed to be retrieved (just searched). If you're running Solr as a front-end search for another database, the best way I've seen t…
Yep, totally agree with this. Last month we spent an hour or two going through the schema and removing any fields that didn't need to be stored, and making sure that only fields which we actually do queries on had index=true. I didn't test before and after results, but qualitatively it seemed to be faster afterwards.
Re: Optimizing Solr (Or How To 7x Your Search Speed)
#8I'd love to have them try out Searchify's hosted search and see how fast it is. The key to fast search is RAM, which is why we run our search indexes from RAM (not cheap), and most queries are served within 100ms. If you're the author of the blog post, please contact me, chris at searchify, if you'd like to do this comparison, and I'll set you up with a test acct.
Hi Chris, what is Searchify's relationship with the old IndexTank ( http://indextank.com ) team?
There's no formal relationship, although I have met several of the IndexTank guys and they're a cool group. And Searchify is based on the IndexTank open-source project.
Re: Optimizing Solr (Or How To 7x Your Search Speed)
#9Hey, Websolr founder here.
Websolr's indexes return in under 50ms for queries of average complexity.
The more expensive queries usually involve "faceting" or sorting a large number of results. For an example, say you search Github for "while." Github used to do language facets, where it would tell you that out of a million results, 200103 files were in javascript, 500358 files were in C, etc.
The problem with this is that you have to count over a million records, on every search! Unlike most search operations which are IO bound, the counting can be CPU-bound, so sharding on one box will let you take advantage of multiple cores.
Racoonone is "sorting on two dimensions, a geo bounding box, four numeric range filters, one datetime range filter, and a categorical range filter." This should put him in a cpu-bound range (in particular because of the sort).
Websolr has customers on sharded plans, but they are usually used in custom sales cases where we're serving many, many millions of documents. We'll look at adding sharding as an option to our default plans, so that they'll be more accessible for people like raccoonone. In the meantime, if you send an email to info@onemorecloud.com, we'll try to accomodate use cases like this.
Edit: Also, other possible optimizations include (1) indexing in the same order you will sort on, if you know ahead of time, and (2) using the TimeLimitedCollector.
Re: Optimizing Solr (Or How To 7x Your Search Speed)
#10Our company had to set up a Solr implementation with some pretty crazy requirements (hundreds of shards, tens of thousands of requests per second, etc), and we ended up with 4 machines - one for indexing, one as a backup indexer/searcher, and 2 just doing load balanced searches. Replication was interesting but easy to set up (since it's basically an rsync of the indexes between servers).
The end result works very well, though it's a real memory hog when you get into the "hundreds" of shards on an individual server.