Full technical details on Asana's worst outage • Hacker News Board

Re: Full technical details on Asana's worst outage

#51

Post by marcog1 » Sat, Sep 10, 2016, 3:41 PM UTC

tonfa wrote: post #21

marcog1 wrote: post #8

Earlier quoted context omitted.

Correct. We don't roll out during peak load either.

Considered at least starting your release canary during peak load?

We have talked about it. It is unlikely to helped with an event like this, and I don't recall an event where it would have. It also has the downside of extending our deployment cycle by a lot. Notably, we do run a canary internally, and that had no issues, which actually through us off for a while because while the app was partially down for users it was working for us and that hasn't happened to us in a while.

Re: Full technical details on Asana's worst outage

#52

Post by jwatte » Sat, Sep 10, 2016, 3:47 PM UTC

themartorana wrote: post #39

merb wrote: post #32

Earlier quoted context omitted.

I'm not sure whats better. 1. Describing the root cause and what you failed at. 2. Blame the stuff you are using / other people (clouds you use) 3. just says nothing and try to forget what happened. What do you think is best?

It's maybe a level of details thing. "A bad deploy went unnoticed, causing a cascading failure. We identified how that happened and have new checks in place to prevent it in the future." Two lines, with the same information someone not very technically literate would understand from the OP. I agree with being transparent, but I also believe in not unnecessarily scaring and/or confusing customers, either. (Pretty soon…

The fact that you think an engineer can be "outed" is a culture problem.

The process failed the engineer. Testing, deployment, and monitoring infrastructure was not up to the task of supporting human beings. That it happened to be triggered by engineer X instead of engineer Y is entirely coincidental.

The audience of the post mortem matters. When I see the two line summary, I have no idea whether that's a CYA whitewash, or a sincere part of a process of improvement. When I see the full PM, it builds more trust.

If you're not an engineer capable of understanding the details, it may have a different effect. And if you're part of a corporate culture of politics, shaming, and status chasing, it must feel totally alien.

Three cheers for transparency!

Re: Full technical details on Asana's worst outage

#53

Post by marcog1 » Sat, Sep 10, 2016, 3:47 PM UTC

cookiecaper wrote: post #48

Reading through this, it sounds like some basic monitoring would've quickly allowed them to pinpoint the cause instead of wasting time with database servers. All it would take is pulling up the charts in Munin or Datadog or whatever and seeing "Oh, there's a big spike correlated with our deploy and the server is redlining now, better roll that back". A bug or issue in the recent deploy would logically be one of the f…

We have very comprehensive dashboards. Getting the perfect ones that help in all cases, while not being information overload (the problem here) and being discoverable is a hard, iterative process.

Re: Full technical details on Asana's worst outage

#54

Post by rossjudson » Sat, Sep 10, 2016, 3:49 PM UTC

katzgrau wrote: post #24

These sort of deeply apologetic and hyper-transparent post-mortems have become commonplace, but sometimes I wonder how beneficial they are. Customers appreciate transparency, but perhaps delving into the fine details of the investigation (various hypotheses, overlooked warning signs, yada yada) might actually end up leaving the customer more unsettled than they would have been otherwise. Today I learned that Asana ha…

Your approach works for the incident, but not for the relationship. Transparency about the technical nature of the outage is a commitment to the client that this type of outage won't recur, and steps are being taken to ensure that. It pierces the veil of arrogance by assuming client competence. That client is actually someone who reports to someone else, and they're going to have to explain their outage to the boss. For cloud providers, this kind of transparent post-mortem is the root of a fan-out of incident analysis.

Re: Full technical details on Asana's worst outage

#55

Post by jwatte » Sat, Sep 10, 2016, 3:50 PM UTC

zzzcpan wrote: post #34

Strangely, there are no actual technical details in the report and the blame is on the process. Although most of the times there is some way to prevent bugs from causing problems with better architecture.

The detail was right there: debugging something in security caused massive logging which caused CPU bottlenecking.

Performance is the hardest thing to integration test for. Keeping careful track of CPU/memory/network/disk load with automated alerts can help.

(Fancy systems like running a traffic replica can help, too, but at a much higher cost.)

Re: Full technical details on Asana's worst outage

#56

Post by madelinecameron » Sat, Sep 10, 2016, 3:51 PM UTC

>And to make things even more confusing, our engineers were all using the dogfooding version of Asana, which runs on different AWS-EC2 instances than the production version

... That kind of defeats the purpose of "dogfooding". Sure, you have to use the same code (hopefully) but it doesn't give you the same experience.

Re: Full technical details on Asana's worst outage

#57

Post by matt_wulfeck » Sat, Sep 10, 2016, 3:52 PM UTC

marcog1 wrote: post #43

qaq wrote: post #23

This is "not that different" from getting a very high load spike do you guys not have some autoscaling setup?

We do, but it didn't help given the cause of the high cpu was our logging infrastructure (Amazon Kinesis) being overloaded by the webservers.

Does kinesis not support UDP sylog style logging, some of these old technologies had the right idea: if your sending too much data, drop the packets on the floor instead of falling over!

Re: Full technical details on Asana's worst outage

#58

Post by jwatte » Sat, Sep 10, 2016, 3:53 PM UTC

gjtorikian wrote: post #13

bArray wrote: post #9

Was this incident really recorded minute by minute or is that made up? I've noticed a lot of companies that give this kind of detail like to give a minute by minute report, I just don't understand how they get that accuracy?

Oh, man. Most definitely that's real. If you're working in Slack or chat, you've got a minimum of half a dozen people typing and putting out suggestions and offering to investigate something. That's all time stamped. And even if you're not doing that real-time, you may be using something like a GitHub issue to discuss the problem via comments, which are also time-stamped. No one at the moment of the incident is proba…

Yes, slack and irc time stamps is common. Ideally your shell and auditing gives you that for commands, too!

Re: Full technical details on Asana's worst outage

#59

Post by marcog1 » Sat, Sep 10, 2016, 3:56 PM UTC

jwatte wrote: post #55

zzzcpan wrote: post #34

Strangely, there are no actual technical details in the report and the blame is on the process. Although most of the times there is some way to prevent bugs from causing problems with better architecture.

The detail was right there: debugging something in security caused massive logging which caused CPU bottlenecking. Performance is the hardest thing to integration test for. Keeping careful track of CPU/memory/network/disk load with automated alerts can help. (Fancy systems like running a traffic replica can help, too, but at a much higher cost.)

We actually have a traffic replica (dark client) setup for the new webserver architecture we are gradually migrating to. It likely would have caught this before deploying to users.

Re: Full technical details on Asana's worst outage

#60

Post by jwatte » Sat, Sep 10, 2016, 3:57 PM UTC

marcog1 wrote: post #3

kctess5 wrote: post #2

I find it interesting that they didn't notice the overloading for so long. Also that it took so long to roll back. Given that they reportedly roll out twice a day, it seems like identifying a rollback target would be fairly quick.

This was the first time we had this class of outage. Many things were in a very bad state, and many of these symptoms were more familiar to us. So we spent time ruling them out before realising webserver CPU was closer to the root cause than the other symptoms. We roll back by reverting to a previous release on the load balancers, which is usually pretty instant. The previous releases were bad and themselves rolled b…

Roll backs are in chat logs? I'd assume your scripts would record what they do when they do it, including roll backs.

Also, when only deploying two times a day, it's harder to tell which of the included changes have the problem. That's an argument for more frequent deploys!