Get thousands of webcamshow for free. Sign up there http://lynk5r.skhit.in/
Full technical details on Asana's worst outage
41–50 of 70 posts
Re: Full technical details on Asana's worst outage
#42Earlier quoted context omitted.
I'm not sure whats better. 1. Describing the root cause and what you failed at. 2. Blame the stuff you are using / other people (clouds you use) 3. just says nothing and try to forget what happened. What do you think is best?
The post like this is definitely confusing. I think you do need to at least acknowledge the problem. With a clear non-technical explanation of the problem in the first paragraph. The rest should go into real technical details of the result of the investigation, not an investigation itself.
Why is it confusing? I did not find it confusing. I like such excruciatingly detailed postmortem analyses, they make for great reading and my respect for the company that does this is increased by reading these.
Re: Full technical details on Asana's worst outage
#43This is "not that different" from getting a very high load spike do you guys not have some autoscaling setup?
We do, but it didn't help given the cause of the high cpu was our logging infrastructure (Amazon Kinesis) being overloaded by the webservers.
Re: Full technical details on Asana's worst outage
#44These sort of deeply apologetic and hyper-transparent post-mortems have become commonplace, but sometimes I wonder how beneficial they are. Customers appreciate transparency, but perhaps delving into the fine details of the investigation (various hypotheses, overlooked warning signs, yada yada) might actually end up leaving the customer more unsettled than they would have been otherwise. Today I learned that Asana ha…
Every response from our users so far has been thanking is for the transparency. It also represents our internal transparency, and that has a real impact on recruiting.
Re: Full technical details on Asana's worst outage
#45Earlier quoted context omitted.
I'm not sure whats better. 1. Describing the root cause and what you failed at. 2. Blame the stuff you are using / other people (clouds you use) 3. just says nothing and try to forget what happened. What do you think is best?
It's maybe a level of details thing. "A bad deploy went unnoticed, causing a cascading failure. We identified how that happened and have new checks in place to prevent it in the future." Two lines, with the same information someone not very technically literate would understand from the OP. I agree with being transparent, but I also believe in not unnecessarily scaring and/or confusing customers, either. (Pretty soon…
We will never out individuals. The person who committed the code was innocent. We got him a fun gift as a sort of joke.
Re: Full technical details on Asana's worst outage
#46Earlier quoted context omitted.
This was the first time we had this class of outage. Many things were in a very bad state, and many of these symptoms were more familiar to us. So we spent time ruling them out before realising webserver CPU was closer to the root cause than the other symptoms. We roll back by reverting to a previous release on the load balancers, which is usually pretty instant. The previous releases were bad and themselves rolled b…
What I don't get is why you didn't see the relatively low cpu usage on the database server and the super high ones on the webserver immediately in a nagios (or similar) dashboard.
It's because they don't have a simple rollup dashboard that you can see that at a glance, like most places. Can you imagine if your car just showed you an event log for a door open, oil, turn singles on etc. that's what most monitoring systems are like these days.
Re: Full technical details on Asana's worst outage
#47Not a bad reaction. With all the reverts is there a QA issue? Or too many releases?
When you do daily deployments, you can't QA every one much. You rely on automated tests and Internal users using the new code for a couple hours before the deployment. We were unlucky in this case with the number of bad releases. Each was relatively minor, and ironically one was to fix a bug with the code that caused this outage. We run a 5 whys for most of them.
Re: Full technical details on Asana's worst outage
#48Reading through this, it sounds like some basic monitoring would've quickly allowed them to pinpoint the cause instead of wasting time with database servers. All it would take is pulling up the charts in Munin or Datadog or whatever and seeing "Oh, there's a big spike correlated with our deploy and the server is redlining now, better roll that back". A bug or issue in the recent deploy would logically be one of the first suspects in such a circumstance. Don't know why they wasted 30-60 minutes on a red herring. The correlation would be even more obvious if they took advantage of Datadog's event stream and marked each deployment.
Additionally, CPU alarms on the web servers should've informed them that the app was inaccessible because the web servers did not have sufficient resources to serve requests. This can be alleviated prior to pinpointing the cause by a) spinning up more web servers and adding them to the load balancer; or b) redirecting portions of the traffic to a static "try again later" page hosted on a CDN or static-only server. This can be done at the DNS level.
Let this be a lesson to all of us. Have basic dashboards and alarming.
Re: Full technical details on Asana's worst outage
#49These sort of deeply apologetic and hyper-transparent post-mortems have become commonplace, but sometimes I wonder how beneficial they are. Customers appreciate transparency, but perhaps delving into the fine details of the investigation (various hypotheses, overlooked warning signs, yada yada) might actually end up leaving the customer more unsettled than they would have been otherwise. Today I learned that Asana ha…
Every response from our users so far has been thanking is for the transparency. It also represents our internal transparency, and that has a real impact on recruiting.
Not an Asana user but if I were this kind of response is exactly what I like to see as a both a user and a developer so well done.
Re: Full technical details on Asana's worst outage
#50The real support for a frequent deployment system is in the immune system!
I've had good luck with a deployment immune system that rolls back if CPU or other load jumps, even if it doesn't immediately cause user failure. (I e, monitor crucial internals, not just user availability)