Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Optimizing Solr (Or How To 7x Your Search Speed)

carsabi.com

Post reply on HN
Print view View original

11–20 of 33 posts

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#11

Post by raccoonone »

fizx wrote: post #9

Hey, Websolr founder here. Websolr's indexes return in under 50ms for queries of average complexity. The more expensive queries usually involve "faceting" or sorting a large number of results. For an example, say you search Github for "while." Github used to do language facets, where it would tell you that out of a million results, 200103 files were in javascript, 500358 files were in C, etc. The problem with this is…

Awesome, lemme know when you guys have sharding available! Would love to not have to worry about running our own index, again.

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#13

Post by raccoonone »

falcolas wrote: post #10

Our company had to set up a Solr implementation with some pretty crazy requirements (hundreds of shards, tens of thousands of requests per second, etc), and we ended up with 4 machines - one for indexing, one as a backup indexer/searcher, and 2 just doing load balanced searches. Replication was interesting but easy to set up (since it's basically an rsync of the indexes between servers). The end result works very wel…

What was the reason for having hundreds of shards on each server? Were you still seeing performance benefits to sharding it that aggresively?

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#15

Post by fizx »

raccoonone wrote: post #11
fizx wrote: post #9

Hey, Websolr founder here. Websolr's indexes return in under 50ms for queries of average complexity. The more expensive queries usually involve "faceting" or sorting a large number of results. For an example, say you search Github for "while." Github used to do language facets, where it would tell you that out of a million results, 200103 files were in javascript, 500358 files were in C, etc. The problem with this is…

Awesome, lemme know when you guys have sharding available! Would love to not have to worry about running our own index, again.

It's available now, just not as part of a standard plan. I'll send you an email, just to make sure we're getting you the best setup for your needs.

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#16

Post by simonw »

sudoman69 wrote: post #12

does any one have experience with adding shards on the fly??we have a requirement weher we get millions of docs every day and we need to have an environment that can handle real-time as well previous days' data...any thoughts on this will be appreciated...

The shards that are used in a Solr query are specified at runtime (you pass a list of shard URLs as part of the search query string) so adding new shards on the fly should Just Work.

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#17

Post by matan_a »

There are quite a few other performance related points to think about for Solr speed for queries and indexing.

Here are some that come to mind right now that are very useful:

- Be smart about your commit strategy if you're indexing a lot of documents (commitWithin is great). Use batches too.

- Many times, i've seen Solr index documents faster than the database could create them (considering joins, denormalizing, etc). Cache these somewhere so you don't have to recreate the ones that haven't changed.

- Set up and use the Solr caches properly. Think about what you want to warm and when. Take advantage of the Filter Queries and their cache! It will improve performance quite a bit.

- Don't store what you don't need for search. I personally only use Solr to return IDs of the data. I can usually pull that up easily in batch from the DB / KV store. Beats having to reindex data that was just for show anyway...

- Solr (Lucene really) is memory greedy and picky about the GC type. Make sure that you're sorted out in that respect and you'll enjoy good stability and consistent speed.

- Shards are useful for large datasets, but test first. Some query features aren't available in a sharded environment (YMMV).

- Solr is improving quickly and v4 should include some nice cloud functionality (zookeeper ftw).

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#18

Post by mthreat »

matan_a wrote: post #17

There are quite a few other performance related points to think about for Solr speed for queries and indexing. Here are some that come to mind right now that are very useful: - Be smart about your commit strategy if you're indexing a lot of documents (commitWithin is great). Use batches too. - Many times, i've seen Solr index documents faster than the database could create them (considering joins, denormalizing, etc)…

These are good points. Solr/Lucene tuning is an art, so much so that some search consulting companies charge tens (or hundreds) of thousands of dollars for these services. That's the value proposition of Searchify's hosted search - if you just want search, you shouldn't have to worry about shards, commit strategies, batching, GC, etc. You just want to add your documents, search them, and get great, fast results, without having to become a Lucene expert in the process.

If this sounds interesting, check us out at http://www.searchify.com - We offer true real-time, fast hosted search, without requiring you to learn the innards of Solr or Lucene.

Re: Optimizing Solr (Or How To 7x Your Search Speed)

#19

Post by matan_a »

mthreat wrote: post #18
matan_a wrote: post #17

There are quite a few other performance related points to think about for Solr speed for queries and indexing. Here are some that come to mind right now that are very useful: - Be smart about your commit strategy if you're indexing a lot of documents (commitWithin is great). Use batches too. - Many times, i've seen Solr index documents faster than the database could create them (considering joins, denormalizing, etc)…

These are good points. Solr/Lucene tuning is an art, so much so that some search consulting companies charge tens (or hundreds) of thousands of dollars for these services. That's the value proposition of Searchify's hosted search - if you just want search, you shouldn't have to worry about shards, commit strategies, batching, GC, etc. You just want to add your documents, search them, and get great, fast results, with…

Good stuff. I see you're on Heroku as well which is always a win.

Now if someone could put SenseiDB on the cloud, i'd pay for it...

Post reply on HN