Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deno’s July 13th incident update

deno.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Deno’s July 13th incident update

#31

Post by jdlshore »

alluro2 wrote: post #17

I don't mean anything bad to Deno's team (I'm very partial to what they're building), but I'm rather surprised whenever a widely-publicized service has an outage that lasts hours or more than 24h. I'm genuinely curious to understand whether it's typically due to complexity of infrastructure and how hard it is to find route causes, how long it takes to redirect traffic / patch temporarily when the cause is found, or i…

I’m always curious to learn about why people create complex architectures. It’s off-topic, but why so much complexity for such a low volume?

Re: Deno’s July 13th incident update

#32

Post by lucacasonato »

remram wrote: post #22

So three failures: - The load balancer lost its connection to etcd and did not reconnect - The load balancer had no healthy backend and did not un-advertise itself - The load balancer did not report either of those issues to monitoring Honestly this is a little concerning. Are they using their own load-balancing software? If yes, why?

The system does have mitigation against the first two failures in isolation (as described in the post). The mitigations did not work correctly in this scenario with the combined failures unfortunately. This is obviously unexcusable, and we need to do better in the future.

To your final question: yes, we using our own load balancing software. We are building a global hosting platform that needs to be able to run on bare metal servers, not an end user application where load balancing is an afterthought. As such we can not use much of the software that a "regular" SaaS application may be able to. Some constraints our system needs to be able to solve:

- Our load balancers handle routing to 100s of thousands of unique deployments (services), all of which need to be accessible and routeable within milliseconds of a request coming in.

- We need to terminate TLS connections for thousands of unique domains.

- We need to be able to carefully control TLS handshakes, to be able to prewarm downstream services for an imminent request for a given deployment based on a TLS client hello SNI, before even having received an HTTP request yet.

- The system needs to handle hundreds of millions of hourly requests.

- The system needs to be able to run on bare metal.

- We currently handle 34 regions globally (up from 28 at the start of the year), which means that all of the data needed to fulfill the above requirements needs to be accessible from all of our PoPs in a matter of milliseconds.

For many companies global load balancing is something they can outsource to AWS, GCP, or Cloudflare. For us, this is core "business logic" that we need to have full control over. It's difficult for us to outsource, and it's questionable if it would be wise for us to do so. Building new systems is obviously always a complex undertaking, and there will be some stumbling stones in the way, but they can be overcome. We are still bullish that our path is the right one, even if we still have a lot of work ahead.

(if this seems interesting, and you want to work with us on building load balancers, among other things: https://deno.com/jobs)

Re: Deno’s July 13th incident update

#33

Post by lucacasonato »

AtNightWeCode wrote: post #26

"... (a TCP load balancer). It does not record any diagnostics about dropped connections, nor does it have a return channel to return diagnostic information to the user (unlike HTTP loadbalancers, which can return a response header)." And there is no API monitoring apparently.

A bit of a blunt statement on my part. There is monitoring on a multitude of other connection related issues (eg TLS handshake failures, missing SNI, etc). We should have had monitoring for this specific failure where the load balancer did not have any healthy backends, but as mentioned in the post, the load balancer was programmed in way that this should never have been able to happen in the first place (as the LB should have un-advertised itself if there are no unhealty backends).

We are capable of learning from past mistakes though, and as such we'll make sure to add more monitoring for these kinds of scenarios so we can be alerted to a root cause earlier. We will do better.

Re: Deno’s July 13th incident update

#34

Post by lucacasonato »

FBISurveillance wrote: post #2

Hugops to the team. A quick question: is it intentional that there's nothing on https://denostatus.com/ ?

No, it wasn't intentional. Because the incident was not triggered by automation, the incident on the status page was not automatically created. The team did not remember to update the status page while we were investigating the issue. Sorry for that!

I have retroactively added the outage to the status page now: https://denostatus.com/cl5ob2i5s943266vk890ushwov.

Re: Deno’s July 13th incident update

#35

Post by geysersam »

steve_adams_86 wrote: post #27
ctvo wrote: post #16

> On July 13th, at around 18:45 UTC we started to receive reports of an outage from a small number of users. We investigated the status of our services, but were unable to confirm any of the reports. All of our status monitoring and tests reported that everything was operating normally. > Over the course of the outage, we continued to monitor our service status, and worked with some of the affected users to narrow do…

In this case it’s disappointing especially because we know they have exact numbers.

Arguably the exact numbers aren't interesting either. 456 982 requests from 5674 projects by 3091 user accounts were dropped during the 23:53:42.104 hours the outage lasted. I think it makes sense to make an interpretation and present that instead of raw numbers.

Re: Deno’s July 13th incident update

#36

Post by FBISurveillance »

lucacasonato wrote: post #34
FBISurveillance wrote: post #2

Hugops to the team. A quick question: is it intentional that there's nothing on https://denostatus.com/ ?

No, it wasn't intentional. Because the incident was not triggered by automation, the incident on the status page was not automatically created. The team did not remember to update the status page while we were investigating the issue. Sorry for that! I have retroactively added the outage to the status page now: https://denostatus.com/cl5ob2i5s943266vk890ushwov .

Happens to the best of us :hug: You folks are working on a great product!

Re: Deno’s July 13th incident update

#37

Post by steve_adams_86 »

geysersam wrote: post #35
steve_adams_86 wrote: post #27

Earlier quoted context omitted.

In this case it’s disappointing especially because we know they have exact numbers.

Arguably the exact numbers aren't interesting either. 456 982 requests from 5674 projects by 3091 user accounts were dropped during the 23:53:42.104 hours the outage lasted. I think it makes sense to make an interpretation and present that instead of raw numbers.

Good point. I disagree for my own preference but I don’t doubt that you’re right that many people would rather get the gist of the issue. I’m a numbers person but I know many aren’t (which is perfectly fine — neither way is inherently better).

Re: Deno’s July 13th incident update

#38

Post by donavanm »

jdlshore wrote: post #31
alluro2 wrote: post #17

I don't mean anything bad to Deno's team (I'm very partial to what they're building), but I'm rather surprised whenever a widely-publicized service has an outage that lasts hours or more than 24h. I'm genuinely curious to understand whether it's typically due to complexity of infrastructure and how hard it is to find route causes, how long it takes to redirect traffic / patch temporarily when the cause is found, or i…

I’m always curious to learn about why people create complex architectures. It’s off-topic, but why so much complexity for such a low volume?

A lot of this can be from (un)intentionally trying to maintain separation of responsibilities between different teams or developers. Decoupling, interfaces, etc all add up and pretty soon you start building based on what's already done vs where you originally intended to go. And I don't think that's a poor choice; nine women can't have a baby in one month, but they can have nine in nine months (to butcher an old saying).

Re: Deno’s July 13th incident update

#39

Post by donavanm »

remram wrote: post #22

So three failures: - The load balancer lost its connection to etcd and did not reconnect - The load balancer had no healthy backend and did not un-advertise itself - The load balancer did not report either of those issues to monitoring Honestly this is a little concerning. Are they using their own load-balancing software? If yes, why?

I've helped build and run similar distributed systems with deep load balancing and network interactions. There are some packages out there that do bits and pieces of the problem. I don't know of anything COTS that is a suitable like for like replacement of the core components, much less a suitable system . On top of that, as Luca mentioned, you almost always get in to deep interactions between L3/4/5/7 and end up building bespoke logic that's tailored to the business or application needs. A trivial example would be the coupling from IP address assignment/announcement, to TLS cert, to SNI headers, to active customers, to application instance routing.

Re: Deno’s July 13th incident update

#40

Post by ctvo »

geysersam wrote: post #35
steve_adams_86 wrote: post #27

Earlier quoted context omitted.

In this case it’s disappointing especially because we know they have exact numbers.

Arguably the exact numbers aren't interesting either. 456 982 requests from 5674 projects by 3091 user accounts were dropped during the 23:53:42.104 hours the outage lasted. I think it makes sense to make an interpretation and present that instead of raw numbers.

No one needs to know the raw numbers. That may be a little more helpful than no numbers but that could also be internal data they’re not comfortable sharing, and that’s fine. There are numerous other ways to write this that would have communicated the scale of the impact more rigorously, but not deal in absolute, raw numbers.

For example, by using percentages: This outage impacted 1.5% of customer requests vs. a “small” amount.

Post reply on HN