11–20 of 30 posts
Is this sort of parallelism also doable with Solr as well?
Just batch loading the data one time shouldn't create much of a problem for Solr either.
I use bulk api with .net Nest client.
I can easily put 1000 documents per second (which also include 3-10 nested documents) with 4core i5 machine. Serialization is the cheapest operation in my case. I would blame ruby in your workflow.
Elasticsearch is awesome. It may be a good idea to use the bulk API that is built into elasticsearch, use some joins in your SQL query, and index more than just one record at a time. In the implementation I have, I batched my query to 50,000 records at a time that then index into elasticsearch. For the 2.7 million records I indexed this week, it took a total of 54 queries to the database (50,000 records returned at a…
I have no idea why the phrase "high availability" is in this post.
Agree with their use of the term or not, they give you their reasoning at the end of the article.
Earlier quoted context omitted.
Do you mind sharing how long it took for the indexing portion?
It took me about 20 minutes to index 2.7M records. It was pulling from 3 tables using joins, and using the .Net NEST client.
I have no idea why the phrase "high availability" is in this post.
>We call these extensions "high availability" because this approach means that re-indexing a production system can happen much faster, reducing downtime for our users. Agree with their use of the term or not, they give you their reasoning at the end of the article.
But stale data isn't "downtime." This is tech marketing at like, MongoDB level.
Elasticsearch is awesome. It may be a good idea to use the bulk API that is built into elasticsearch, use some joins in your SQL query, and index more than just one record at a time. In the implementation I have, I batched my query to 50,000 records at a time that then index into elasticsearch. For the 2.7 million records I indexed this week, it took a total of 54 queries to the database (50,000 records returned at a…
I started using elasticsearch recently and I was wondering, does the indexing happen in real time during the index request? How do you know how long the indexing process takes?
http://hostname:9200/_cat/indices
The 6th column gives you the number of records indexed. Send your request and refresh the browser.
Elasticsearch is awesome. It may be a good idea to use the bulk API that is built into elasticsearch, use some joins in your SQL query, and index more than just one record at a time. In the implementation I have, I batched my query to 50,000 records at a time that then index into elasticsearch. For the 2.7 million records I indexed this week, it took a total of 54 queries to the database (50,000 records returned at a…
I started using elasticsearch recently and I was wondering, does the indexing happen in real time during the index request? How do you know how long the indexing process takes?
"27 reports per second" what? I use bulk api with .net Nest client. I can easily put 1000 documents per second (which also include 3-10 nested documents) with 4core i5 machine. Serialization is the cheapest operation in my case. I would blame ruby in your workflow.
Earlier quoted context omitted.
>We call these extensions "high availability" because this approach means that re-indexing a production system can happen much faster, reducing downtime for our users. Agree with their use of the term or not, they give you their reasoning at the end of the article.
That's crazy misleading. This is just a post saying, "hey, this is a way to sync data faster." Awesome! Much kudos. But stale data isn't "downtime." This is tech marketing at like, MongoDB level.