Deno’s July 13th incident update • Hacker News Board

Re: Deno’s July 13th incident update

#21

Post by lucacasonato » Sat, Jul 16, 2022, 2:39 PM UTC

alluro2 wrote: post #17

I don't mean anything bad to Deno's team (I'm very partial to what they're building), but I'm rather surprised whenever a widely-publicized service has an outage that lasts hours or more than 24h. I'm genuinely curious to understand whether it's typically due to complexity of infrastructure and how hard it is to find route causes, how long it takes to redirect traffic / patch temporarily when the cause is found, or i…

Our issue here was very much in finding the root cause. Because the failed traffic was “black holed” (TCP connections were being dropped), we had very little information other than “it isn’t working” from the users that reported the issue. This caused us significant headaches in trying to figure out what the commonality between the incident reports of our users was (the geo region). Up until the point this was clear, we were also checking database clusters, DNS configurations, TLS certificates etc to try to isolate the issue.

After we managed to successfully isolate the issue we were able to disable the region within 30 minutes, because we had an established protocol for how to do that.

Here is a more typical incident update for us: https://deno.com/blog/2022-05-30-outage-post-mortem

Part of the issue was also that we did not realize the scope of the issue right at the start of the incident, because our automated monitoring did not catch the dropped traffic.

All that is to say: the outage is obviously unacceptable, and sincerely apologize for it. We are working very hard to make sure nothing similar can occur again in the future.

Re: Deno’s July 13th incident update

#22

Post by remram » Sat, Jul 16, 2022, 3:24 PM UTC

So three failures:

- The load balancer lost its connection to etcd and did not reconnect

- The load balancer had no healthy backend and did not un-advertise itself

- The load balancer did not report either of those issues to monitoring

Honestly this is a little concerning. Are they using their own load-balancing software? If yes, why?

Re: Deno’s July 13th incident update

#23

Post by alluro2 » Sat, Jul 16, 2022, 3:49 PM UTC

lucacasonato wrote: post #21

alluro2 wrote: post #17

I don't mean anything bad to Deno's team (I'm very partial to what they're building), but I'm rather surprised whenever a widely-publicized service has an outage that lasts hours or more than 24h. I'm genuinely curious to understand whether it's typically due to complexity of infrastructure and how hard it is to find route causes, how long it takes to redirect traffic / patch temporarily when the cause is found, or i…

Our issue here was very much in finding the root cause. Because the failed traffic was “black holed” (TCP connections were being dropped), we had very little information other than “it isn’t working” from the users that reported the issue. This caused us significant headaches in trying to figure out what the commonality between the incident reports of our users was (the geo region). Up until the point this was clear,…

Thanks for the insight - I definitely wasn't trying to dump on the team or handling of the issue - really just understand better so I have more awareness and can hopefully help my team (as a young CTO) be more prepared for different types of challenges.

As mentioned, I'm looking forward to continuing to follow Deno's progress and all the best in hardening your devops!

Re: Deno’s July 13th incident update

#24

Post by alluro2 » Sat, Jul 16, 2022, 3:57 PM UTC

viraptor wrote: post #18

alluro2 wrote: post #17

I don't mean anything bad to Deno's team (I'm very partial to what they're building), but I'm rather surprised whenever a widely-publicized service has an outage that lasts hours or more than 24h. I'm genuinely curious to understand whether it's typically due to complexity of infrastructure and how hard it is to find route causes, how long it takes to redirect traffic / patch temporarily when the cause is found, or i…

> is spontaneously met by my team as absolute emergency and typically fixed in Unless they had the "route the whole region over another one" in their prepared and practiced DR procedure, it would take any team a significant time to get that planned, approved, implemented and tested. If you're running something at tens of services scale and recovered in 10min, you're extremely lucky. I'd suggest that if you don't have…

That's a fair point and a good suggestion to consider.

One alleviating circumstance is that, running on AWS, a big portion of such issues (ones that would take a lot of time to resolve) would come from wider AWS outages - when there's significant leeway - the old adage that customers / big part of web would have bigger issues than us being down if an entire AWS region (or multiple) is down.

In Deno's case, most of "those" parts are self-managed and surely much more difficult to keep running reliably.

Re: Deno’s July 13th incident update

#25

Post by turtlebits » Sat, Jul 16, 2022, 4:12 PM UTC

Seem like a huge gap in observability - Low/zero healthy targets for a load balancer should be a P0/critical alert, especially when traffic is getting black holed.

LBs should also be alerting on health checks failures/no data for targets as well.

Re: Deno’s July 13th incident update

#26

Post by AtNightWeCode » Sat, Jul 16, 2022, 4:23 PM UTC

"... (a TCP load balancer). It does not record any diagnostics about dropped connections, nor does it have a return channel to return diagnostic information to the user (unlike HTTP loadbalancers, which can return a response header)."

And there is no API monitoring apparently.

Re: Deno’s July 13th incident update

#27

Post by steve_adams_86 » Sat, Jul 16, 2022, 4:42 PM UTC

ctvo wrote: post #16

> On July 13th, at around 18:45 UTC we started to receive reports of an outage from a small number of users. We investigated the status of our services, but were unable to confirm any of the reports. All of our status monitoring and tests reported that everything was operating normally. > Over the course of the outage, we continued to monitor our service status, and worked with some of the affected users to narrow do…

In this case it’s disappointing especially because we know they have exact numbers.

Re: Deno’s July 13th incident update

#28

Post by lowwave » Sat, Jul 16, 2022, 5:16 PM UTC

remram wrote: post #22

So three failures: - The load balancer lost its connection to etcd and did not reconnect - The load balancer had no healthy backend and did not un-advertise itself - The load balancer did not report either of those issues to monitoring Honestly this is a little concerning. Are they using their own load-balancing software? If yes, why?

instead of the AWS ones?

Re: Deno’s July 13th incident update

#29

Post by TechBro8615 » Sat, Jul 16, 2022, 5:34 PM UTC

ctvo wrote: post #16

> On July 13th, at around 18:45 UTC we started to receive reports of an outage from a small number of users. We investigated the status of our services, but were unable to confirm any of the reports. All of our status monitoring and tests reported that everything was operating normally. > Over the course of the outage, we continued to monitor our service status, and worked with some of the affected users to narrow do…

This is a brand new platform and I assume this blog post is one of their first post-mortems. I’m inclined to give them a break, and respect them for establishing a process of post-mortems.

That is all to say, regardless of whether the post itself is “a little concerning,” it would be more concerning if the post didn’t even exist. And if you weren’t one of their affected customers, you likely wouldn’t even know this happened. So they did the right thing by publishing it and opening themselves to your criticism, which is a positive sign for the future of the platform IMO.

Re: Deno’s July 13th incident update

#30

Post by remram » Sat, Jul 16, 2022, 6:45 PM UTC

lowwave wrote: post #28

remram wrote: post #22

So three failures: - The load balancer lost its connection to etcd and did not reconnect - The load balancer had no healthy backend and did not un-advertise itself - The load balancer did not report either of those issues to monitoring Honestly this is a little concerning. Are they using their own load-balancing software? If yes, why?

instead of the AWS ones?

Yes or even a more turn-key software package. It sounds like they had very custom software, I would expect that established load-balancing software doesn't fail to reconnect.