Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-mortem of Christmas Eve Amazon ELB outage

aws.amazon.com

Post reply on HN
Print view View original

11–20 of 40 posts

Re: Post-mortem of Christmas Eve Amazon ELB outage

#11

Post by cperciva »

mmastrac wrote: post #2

"The data was deleted by a maintenance process that was inadvertently run against the production ELB state data. This process was run by one of a very small number of developers who have access to this production environment."

This is why I love AWS post-mortems: They don't hide things.

Most companies would have said "the outage was caused by a system being accidentally misconfigured" or even less; Amazon, in contrast, admitted (a) that a specific person was identified who made the mistake, (b) that he had access to the systems in question because a process was being run manually which should have been automated, and (c) that if there hadn't been an error in the access control rules, he wouldn't have been able to make that mistake.

I often wish that Amazon was more open about their internal systems; post-mortems are the one time when I'm never wondering what they're not telling me.

Re: Post-mortem of Christmas Eve Amazon ELB outage

#12

Post by michaelfairley »

briandear wrote: post #4
mmastrac wrote: post #2

"The data was deleted by a maintenance process that was inadvertently run against the production ELB state data. This process was run by one of a very small number of developers who have access to this production environment."

And the developer ought to be fired. He cost me a considerable amount of money since my heroku site with SSL endpoint was down over 12 hours during one if our biggest days of the year. Of course, heroku still (over)bills me those hours for Dynos and Workers. I course, I am small potatoes compared to Netflix being down (while Amazon instant video wasn't.) Amazon is populated by jackasses apparently and Heroku isn't mu…

http://www.whoownsmyavailability.com/

Re: Post-mortem of Christmas Eve Amazon ELB outage

#13

Post by niggler »

msbarnett wrote: post #10
briandear wrote: post #4

Earlier quoted context omitted.

And the developer ought to be fired. He cost me a considerable amount of money since my heroku site with SSL endpoint was down over 12 hours during one if our biggest days of the year. Of course, heroku still (over)bills me those hours for Dynos and Workers. I course, I am small potatoes compared to Netflix being down (while Amazon instant video wasn't.) Amazon is populated by jackasses apparently and Heroku isn't mu…

You weren't down because an Amazon engineer fat-fingered a maintenance command, you were down because you chose to put all of your eggs in one basket. The basket you chose failed, and you had no redundancy basket to fail over to. Choosing a new basket to put all your eggs in isn't going to fix anything just because the new basket hasn't failed on you yet. Make smarter decisions. If being up on December 24 was really…

Interestingly enough a similar article on the front page scapegoated Windows Azure.

People can't own up to the fact that customers don't really care if it's Amazon AWS or Heroku or some other platform that failed -- if your service is down, its your fault!

Re: Post-mortem of Christmas Eve Amazon ELB outage

#14

Post by msbarnett »

niggler wrote: post #13
msbarnett wrote: post #10

Earlier quoted context omitted.

You weren't down because an Amazon engineer fat-fingered a maintenance command, you were down because you chose to put all of your eggs in one basket. The basket you chose failed, and you had no redundancy basket to fail over to. Choosing a new basket to put all your eggs in isn't going to fix anything just because the new basket hasn't failed on you yet. Make smarter decisions. If being up on December 24 was really…

Interestingly enough a similar article on the front page scapegoated Windows Azure. People can't own up to the fact that customers don't really care if it's Amazon AWS or Heroku or some other platform that failed -- if your service is down, its your fault!

> People can't own up to the fact that customers don't really care if it's Amazon AWS or Heroku or some other platform that failed -- if your service is down, its your fault!

I don't, necessarily, have a problem with saying, without animosity, "we're down because our hosting provider is down". For many small businesses, being down when your hosting provider is down is an acceptable tradeoff -- your business simply isn't critical enough for it to be worth the costs of maintaining fail-over redundancy in case of primary hosting loss.

That's totally fine. Most people probably don't want to pay the higher cost of having every service they use having the ability to survive their main host going down. A couple 9s of uptime is more than enough for the vast majority of businesses.

What I have a problem is with people who were behaving as though they were fine with that tradeoff screaming about how they needed to be up when it comes time to pay in a little downtime for those cheaper hosting costs. Grow up and accept the tradeoffs you choose to make.

Re: Post-mortem of Christmas Eve Amazon ELB outage

#15

Post by jacquesm »

briandear wrote: post #4
mmastrac wrote: post #2

"The data was deleted by a maintenance process that was inadvertently run against the production ELB state data. This process was run by one of a very small number of developers who have access to this production environment."

And the developer ought to be fired. He cost me a considerable amount of money since my heroku site with SSL endpoint was down over 12 hours during one if our biggest days of the year. Of course, heroku still (over)bills me those hours for Dynos and Workers. I course, I am small potatoes compared to Netflix being down (while Amazon instant video wasn't.) Amazon is populated by jackasses apparently and Heroku isn't mu…

Judging by your two comments in this thread you should probably study statistics for a short while, even a minimal amount of knowledge in that field would have saved you from two mistakes!

Re: Post-mortem of Christmas Eve Amazon ELB outage

#16

Post by cmelbye »

cperciva wrote: post #7
______ wrote: post #5

It's amazing that the team worked through what is likely the least fun night of the year to be working to fix this issue.

I wonder who gets stuck working those shifts -- would it just be following the normal schedule, in the name of fairness? Do more senior people get to claim the day of vacation? Do parents get priority over single people?

They might've had employees that don't celebrate Christmas.

Re: Post-mortem of Christmas Eve Amazon ELB outage

#17

Post by saosebastiao »

briandear wrote: post #4
mmastrac wrote: post #2

"The data was deleted by a maintenance process that was inadvertently run against the production ELB state data. This process was run by one of a very small number of developers who have access to this production environment."

And the developer ought to be fired. He cost me a considerable amount of money since my heroku site with SSL endpoint was down over 12 hours during one if our biggest days of the year. Of course, heroku still (over)bills me those hours for Dynos and Workers. I course, I am small potatoes compared to Netflix being down (while Amazon instant video wasn't.) Amazon is populated by jackasses apparently and Heroku isn't mu…

If you can't engineer your way around downtime, the fault is yours.

Re: Post-mortem of Christmas Eve Amazon ELB outage

#18

Post by niggler »

msbarnett wrote: post #14
niggler wrote: post #13

Earlier quoted context omitted.

Interestingly enough a similar article on the front page scapegoated Windows Azure. People can't own up to the fact that customers don't really care if it's Amazon AWS or Heroku or some other platform that failed -- if your service is down, its your fault!

> People can't own up to the fact that customers don't really care if it's Amazon AWS or Heroku or some other platform that failed -- if your service is down, its your fault! I don't, necessarily, have a problem with saying, without animosity, "we're down because our hosting provider is down". For many small businesses, being down when your hosting provider is down is an acceptable tradeoff -- your business simply is…

"people who were behaving as though they were fine with that tradeoff"

I'm not sure who is to blame at this point -- is it the fault of the individual for not understanding uptime? is it the fault of the service for not articulating clearly what it means? Is it the fault of the industry for inculcating an unjustified sense of entitlement?

Re: Post-mortem of Christmas Eve Amazon ELB outage

#19

Post by blantonl »

Want to learn great infrastructure management tips? Read and digest these post-mortems.... Regardless of who the provider is.

There is always one takeaway from these that I can use to better my own infrastructure management activities - even when most of my infrastructure runs on AWS. :)

Re: Post-mortem of Christmas Eve Amazon ELB outage

#20

Post by bigdubs »

ryanpetrich wrote: post #9
briandear wrote: post #3

12:24 on 12/24. Interesting coincidence. What are the odds?

Just as likely as 12:25 on 12/24.

Though technically correct, this (seemingly) common response to highlighted dates belies the fact that no, 12:24 12/24 is in fact more rare than an arbitrary other minute because of the pattern in it's configuration.

Sometimes pedantry gets in the way of appreciating something as small as this.

Post reply on HN