61–70 of 70 posts
>And to make things even more confusing, our engineers were all using the dogfooding version of Asana, which runs on different AWS-EC2 instances than the production version ... That kind of defeats the purpose of "dogfooding". Sure, you have to use the same code (hopefully) but it doesn't give you the same experience.
Not a bad reaction. With all the reverts is there a QA issue? Or too many releases?
When you do daily deployments, you can't QA every one much. You rely on automated tests and Internal users using the new code for a couple hours before the deployment. We were unlucky in this case with the number of bad releases. Each was relatively minor, and ironically one was to fix a bug with the code that caused this outage. We run a 5 whys for most of them.
In that case, should you be doing daily deployments to production?
Earlier quoted context omitted.
It's maybe a level of details thing. "A bad deploy went unnoticed, causing a cascading failure. We identified how that happened and have new checks in place to prevent it in the future." Two lines, with the same information someone not very technically literate would understand from the OP. I agree with being transparent, but I also believe in not unnecessarily scaring and/or confusing customers, either. (Pretty soon…
We will never out individuals. The person who committed the code was innocent. We got him a fun gift as a sort of joke.
- train the people more - help them to get over (some ppl could be really mad and infconfident after they did bad)
These sort of deeply apologetic and hyper-transparent post-mortems have become commonplace, but sometimes I wonder how beneficial they are. Customers appreciate transparency, but perhaps delving into the fine details of the investigation (various hypotheses, overlooked warning signs, yada yada) might actually end up leaving the customer more unsettled than they would have been otherwise. Today I learned that Asana ha…
The latter is useful for example when my boss asks me to evaluate whether to continue using a service after an incident. If I can't get enough information to make a recommendation I might propose a switch out of distrust. Especially when to problem was related to security or privacy.
Not a bad reaction. With all the reverts is there a QA issue? Or too many releases?
When you do daily deployments, you can't QA every one much. You rely on automated tests and Internal users using the new code for a couple hours before the deployment. We were unlucky in this case with the number of bad releases. Each was relatively minor, and ironically one was to fix a bug with the code that caused this outage. We run a 5 whys for most of them.
Are the daily drops predominately bug fixes or also a regular drip of new functionality?
I think the old world of quarterly releases was also bad for other reasons. I'm curious about the right middle point.
Every time a company like Asana comes clean about outages and software quality issues, the canon of knowledge improves. Thank you for sharing!
Reading through this, it sounds like some basic monitoring would've quickly allowed them to pinpoint the cause instead of wasting time with database servers. All it would take is pulling up the charts in Munin or Datadog or whatever and seeing "Oh, there's a big spike correlated with our deploy and the server is redlining now, better roll that back". A bug or issue in the recent deploy would logically be one of the f…
We have very comprehensive dashboards. Getting the perfect ones that help in all cases, while not being information overload (the problem here) and being discoverable is a hard, iterative process.
I'm not sure what you're using for dashboards but Datadog makes it pretty easy to find this stuff. I'm not a Datadog shill and I actually am not a huge fan of the product, but it's what we use and it's been a big help over our previous Munin installation.
Other process changes that could prevent this are good load testing in a stage environment and getting your company using the real prod code on the real prod infrastructure as its main/default install. A lot of the benefits of "dogfooding" are lost if you're using alpha code on dev-only boxes (as you state that you are in another comment).
As another commenter said, I'm not sure that postmortems like this are valuable unless the problem was particularly complex/interesting. I'm sure that a lot of people at Asana know how to fix this and that it's just a matter of getting management to allow them to do so. I'm sure you owe your customers an explanation of some sort, but I don't know if you need to get into details that say "Yeah, it was just a pretty typical organizational failure, we really should've known better". Everyone has those, but it's best not to publicize them too much.
I'm not going to hold it against Asana because I've worked at a lot of companies and I know how this goes, but when people come here and analyze the cause, as a postmortem invites the readers to do, you seem a little defensive. Perhaps it's best to keep the explanation more brief/vague when it's not a complex failure.
Earlier quoted context omitted.
I think the OP didn't mean that these post mortems are not beneficial internally, what he said that disclosing all these details to the public can be confusing and maybe counter productive.
I'm not sure whats better. 1. Describing the root cause and what you failed at. 2. Blame the stuff you are using / other people (clouds you use) 3. just says nothing and try to forget what happened. What do you think is best?
"Asana had an outage for 45 minutes yesterday. This was due to an issue with a deploy that was pushed the night prior. We apologize for the inconvenience and are undertaking a thorough review of processes to ensure that similar events don't occur in the future. Please be assured everything is back in working order now. Thank you for your patience and continued patronage."
Big detailed postmortems like this should remain internal documents unless they describe a complex or rare technical failure, news and/or discussion of which will actually benefit the larger community.
Earlier quoted context omitted.
I think the OP didn't mean that these post mortems are not beneficial internally, what he said that disclosing all these details to the public can be confusing and maybe counter productive.
I'm not sure whats better. 1. Describing the root cause and what you failed at. 2. Blame the stuff you are using / other people (clouds you use) 3. just says nothing and try to forget what happened. What do you think is best?
You can simply make a statement that goes something like this:
"We've completed our investigation of the outage and we found that it was caused due to both technical and procedural errors in the manner in which we deploy our code and monitor the environment. We gathered all the information we require and have made improvements based on it that would help to prevent these issues and other issues with similar causality from occurring. While we do apologize for any inconvenience that the outage may have caused we do want to stress that because of the lessons learned from it our service would grow to be more robust and reliable in the future."
That's it, simple even if generic, having to read 3 pages of technical details isn't really helpful to anyone, if anything the more "suspicious" people might see that as an attempt to mask the real cause of the issues.
But overall when you go into specific what you also give is for people the ability to focus their frustrations and disapproval on a specific subject which is never good. After reading this what I "feel" at first glance is that the the fault lies in the engineers that monitored the environment, so the engineers are incapable of performing their duties, now i feel like the hiring and management processes in that company are not working well if they let "unqualified" engineers in. This is how how a minor outage now blows into a specific complaint or negative bias towards a company and you can easily avoid it by giving enough "reassuring" information but not enough for anyone to actually sink their teeth at.
Overall a generic positive statements is more likely to be accepted as well it sucks but shit melts down sometimes and sometimes people make mistake. A a more technical statement might be accepted as "well why did you hire bob in the first place?" or "why fuck are you using this_framework_i_dont_like?".
Earlier quoted context omitted.
It's maybe a level of details thing. "A bad deploy went unnoticed, causing a cascading failure. We identified how that happened and have new checks in place to prevent it in the future." Two lines, with the same information someone not very technically literate would understand from the OP. I agree with being transparent, but I also believe in not unnecessarily scaring and/or confusing customers, either. (Pretty soon…
We will never out individuals. The person who committed the code was innocent. We got him a fun gift as a sort of joke.
Earlier quoted context omitted.
When you do daily deployments, you can't QA every one much. You rely on automated tests and Internal users using the new code for a couple hours before the deployment. We were unlucky in this case with the number of bad releases. Each was relatively minor, and ironically one was to fix a bug with the code that caused this outage. We run a 5 whys for most of them.
> When you do daily deployments, you can't QA every one much. In that case, should you be doing daily deployments to production?