Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postmortem: Outage due to Elasticsearch’s flexibility and our carelessness

lambda.grofers.com

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Postmortem: Outage due to Elasticsearch’s flexibility and our carelessness

#11

Post by vaidik »

castell wrote: post #5

It's not the first time I read about service (near) outages and post mortem that involve ElasticSearch. It's marketed as NoSQL solution, but some devs don't read the details in the documentation like "Elasticsearch is not schema-less". Knowing Lucene and its index structure for years, I wouldn't advise to use a Lucene database as primary storage. Always keep the original data at least in a log-queue or in a separate…

Agreed. We don't use it as a primary database as well. In fact, all of our data persists in PostgreSQL. However, a smart thing for us to do would have been using PostgreSQL as a fallback (postgis for spatial queries, something that we heavily do with Elasticsearch).

Obviously incidents like such have made us cautious and we will be working on fixing these architectural issues quickly.

Re: Postmortem: Outage due to Elasticsearch’s flexibility and our carelessness

#14

Post by vaidik »

shubhamjain wrote: post #10

> The values for field price in products mapping and the values for field price in promotions mapping (in the same index) will essentially be added to the same list at Lucene Segment level. And it will not fail or throw an exception. Probably a stupid question but why can't ElasticSearch translate both mappings into something like, "products_price", and "promotions_price" before adding to the 'list'?

Something I wondered too. But I think in the 2.x versions, they are already planning something around it. Honestly, I have not looked at it much yet but even creating another mapping with the same field name but different type should perhaps throw a warning or not be allowed unless overridden as they lead to things like these.

Re: Postmortem: Outage due to Elasticsearch’s flexibility and our carelessness

#15

Post by orkj »

Very interesting article, and a few lessons learned. And very well written. I feel like I can feel the tenseness in the team when you are debugging and searching for a solution under stress :)

Just wanted to tip you off to what I assume is a typo:

  But lack of automated regression testing would have caught this issue.
I understand what you mean, but if you mean that literally then that is indeed a new and interesting paradigm in software testing :)

Re: Postmortem: Outage due to Elasticsearch’s flexibility and our carelessness

#16

Post by vaidik »

joepvd wrote: post #6

Really awesome to read such a nice writeup of an error. It got me tangentially thinking about why we do not read stuff like: When the patient was brought in, he was barely breathing. Due to the spots on his face, we assumed it was X. Then we made an incision in his pelvis to fix Y. Suddenly, he died. Gosh, we really should have kept an eye on the meter in the corner! I understand things like litigation, but it makes…

Not completely sure about this. But may be we are just not in the same circles of other fields where things like these are shared. I can only speak for myself obviously. But it would be hard to believe that people in the field of medical sciences don't practice this.

Re: Postmortem: Outage due to Elasticsearch’s flexibility and our carelessness

#17

Post by vaidik »

orkj wrote: post #15

Very interesting article, and a few lessons learned. And very well written. I feel like I can feel the tenseness in the team when you are debugging and searching for a solution under stress :) Just wanted to tip you off to what I assume is a typo: But lack of automated regression testing would have caught this issue. I understand what you mean, but if you mean that literally then that is indeed a new and interesting…

I might be missing something. But to my understanding I think it is correct. I didn't quite get what you exactly mean by that? Educate me please.

To explain myself, I meant that tests that were succeeding previously would have failed due to this issue if we had tests written for it. Does my statement not convey this?

Re: Postmortem: Outage due to Elasticsearch’s flexibility and our carelessness

#18

Post by anarazel »

vaidik wrote: post #17
orkj wrote: post #15

Very interesting article, and a few lessons learned. And very well written. I feel like I can feel the tenseness in the team when you are debugging and searching for a solution under stress :) Just wanted to tip you off to what I assume is a typo: But lack of automated regression testing would have caught this issue. I understand what you mean, but if you mean that literally then that is indeed a new and interesting…

I might be missing something. But to my understanding I think it is correct. I didn't quite get what you exactly mean by that? Educate me please. To explain myself, I meant that tests that were succeeding previously would have failed due to this issue if we had tests written for it. Does my statement not convey this?

Original statement:

> But lack of automated regression testing would have caught this issue.

Your question:

> To explain myself, I meant that tests that were succeeding previously would have failed due to this issue if we had tests written for it. Does my statement not convey this?

No. The sentence says that the lack (i.e. not having) tests would have caught the issue. If you remove 'But lack of', the sentence makes sense ;)

Re: Postmortem: Outage due to Elasticsearch’s flexibility and our carelessness

#19

Post by vaidik »

shubhamjain wrote: post #10

> The values for field price in products mapping and the values for field price in promotions mapping (in the same index) will essentially be added to the same list at Lucene Segment level. And it will not fail or throw an exception. Probably a stupid question but why can't ElasticSearch translate both mappings into something like, "products_price", and "promotions_price" before adding to the 'list'?

And BTW, that is something that Elasticsearch recommends you to do that manually in the current versions as well. Read this: https://www.elastic.co/guide/en/elasticsearch/guide/current/...

Re: Postmortem: Outage due to Elasticsearch’s flexibility and our carelessness

#20

Post by dap »

joepvd wrote: post #6

Really awesome to read such a nice writeup of an error. It got me tangentially thinking about why we do not read stuff like: When the patient was brought in, he was barely breathing. Due to the spots on his face, we assumed it was X. Then we made an incision in his pelvis to fix Y. Suddenly, he died. Gosh, we really should have kept an eye on the meter in the corner! I understand things like litigation, but it makes…

Doctors absolutely do this:

https://en.m.wikipedia.org/wiki/Morbidity_and_mortality_conf...

Post reply on HN