Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s Outage on January 4th 2021

slack.engineering

Post reply on HN
Print view View original

51–60 of 110 posts

Re: Slack’s Outage on January 4th 2021

#51

Post by SilasX »

fipar wrote: post #27
cett wrote: post #13

Earlier quoted context omitted.

Though the nuance is Slack did know how to handle it, AWS didn't.

I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. This usually works well, under the rationale that "upstream provider does this for a living, so they must be better than us at this", but if you have too unique needs (or are just a bit "unlucky"), it can fail too. All this to say that the cloud isn't magic. From a risk/e…

>This usually works well, under the rationale that "upstream provider does this for a living, so they must be better than us at this", but if you have too unique needs (or are just a bit "unlucky"), it can fail too.

Heh, a while ago I joked that one way to scale is to "make it somebody else's problem", with the proviso that you need to make sure that the someone else can handle the load. And then (due to the context) a commenter balked at the idea that a big player like YouTube would be unable handle the scaling of their core business.

https://news.ycombinator.com/item?id=23170685

(If they're really blaming it on AWS, it really takes guts to do it so publicly, I think.)

Re: Slack’s Outage on January 4th 2021

#52

Post by ignoramous »

cett wrote: post #13
dplgk wrote: post #9

> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…

Though the nuance is Slack did know how to handle it, AWS didn't.

True, but with any managed service, hidden limits and a cloud provider's own engineering (or lack thereof) may come back to bite the top 0.1% (the whales).

One approach to solve problems of scale is to trim down scale and bound it across multiple disparate silos that do not absolutely interact with each other at all, under any circumstances, except for making quick, constant-time, scale-independent decisions, may be.

In short, do things that don't need scale.

Re: Slack’s Outage on January 4th 2021

#53

Post by dandigangi »

conradfr wrote: post #18
nickthemagicman wrote: post #3

Wow. The trail leads back to AWS. Wasn't there a number of other companies that were down around that same time or was that a different time?

Does AWS compensate you in cases like this?

They do. I can't remember where the documentation is but there's a clause where they payout as long as they aren't meeting their SLAs.

Who knows the different ways they may be able to get out of that. I assume this wasn't one of those times.

Re: Slack’s Outage on January 4th 2021

#55

Post by buildawesome »

tempest_ wrote: post #14
cett wrote: post #13

Earlier quoted context omitted.

Though the nuance is Slack did know how to handle it, AWS didn't.

Well Slack depended on the Cloud(tm). It is a interesting though because a lot of the blog posts like "How we handled a 3000% traffic increase overnight!" boil down to "We turned up the AWS knob". What happens when the AWS knob doesn't work?

You do what Slack did and call the maker of the AWS Knob:tm:.

Re: Slack’s Outage on January 4th 2021

#56

Post by thu2111 »

solidasparagus wrote: post #44
tw04 wrote: post #38

Earlier quoted context omitted.

>I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. Isn't that literally supposed to be the sales pitch for the cloud? Get away from the infrastructure as a whole so you can focus on code, and let the cloud providers wave their magic wand to enable scaling? If you're saying now the story is: well rely on them to auto scal…

No, the cloud provides scalable infrastructure, but once you are in the 0.01% and you have very unique usage patterns, you still need to know how to set up your infrastructure for your needs. The difference is that instead of writing and managing a scalable cache, you just need to build the layer that knows to pre-provision for that scale/talk with AWS to make sure the system has sufficient capacity. The cloud isn't…

This feels like AWS apologism.

Slack knew how to set up their infrastructure. Nothing in the postmortem implies AWS was misconfigured. AWS spotted the problem and fixed it entirely on their side.

Nothing in this report suggests that Slack has unique usage patterns. Users returning to work after Christmas is not a phenomenon unique to Slack.

Their problems were:

1. The AWS infrastructure broke due to an event as predictable as the start of the year. That's on Amazon.

2. Their infrastructure is too complicated. Their auto-scaling created chaos by shutting down machines whilst engineers were logged into them due to bad heuristics, although it's not like this was a good way to save money, and their separation of Slack into many different AWS accounts created weird bottlenecks they had no way to understand or fix.

3. They were unable to diagnose the root cause and the outage ended when AWS noticed the problem and fixed their gateway system themselves.

The cloud isn't some magic thing that solves all scaling problems

In this case it actually created scaling problems where none needed to exist. AWS is expensive compared to dedicated machines in a colo. Part of the justification for that high cost is seamless scalability and ability to 'flex'.

But Slack doesn't need the ability to flex here. Scaling down over the holidays and then back up once people returned to work just isn't that important for them - it's unlikely there were a large number of jobs queued up waiting to run on their spare hardware for a few days anyway. It just wasn't a good way to save money: a massive outage certainly cost them far more than they'll ever save.

Re: Slack’s Outage on January 4th 2021

#57

Post by hanikesn »

miyuru wrote: post #21
sargun wrote: post #5

I wonder why Slack uses TGW instead of VPC peering.

this article from 4 months ago explains it. https://slack.engineering/building-the-next-evolution-of-clo...

Interesting, I was wondering whether using a shared VPC would have been the better solution, but it turns out they use a shared VPC per region and peer them via TGW. IMHO it'd be worth peering those regions individually, to get rid of that potential bottleneck. Of course you loose quite a few interesting features.

Re: Slack’s Outage on January 4th 2021

#58

Post by jrockway »

gscho wrote: post #15

> our dashboarding and alerting service became unavailable. Sounds like the monitoring system needs a monitoring system.

It is quite awkward that the output of "working" and "completely broken" alerting systems have the same visible effect -- no alerts.

For Prometheus users, I wrote alertmanager-status to let a third-party "website up?" monitoring server check your alertmanager: https://github.com/jrockway/alertmanager-status

(I also wrote one of the main Google Fiber monitoring systems back when I was at Google. We spent quite a bit of time on monitoring monitoring, because whenever there was an actual incident people would ask us "is this real, or just the monitoring system being down?" Previous monitoring systems were flaky so people were kind of conditioned to ignore the improved system -- so we had to have a lot of dashboards to show them that there was really an ongoing issue.)

Re: Slack’s Outage on January 4th 2021

#59

Post by JMTQp8lwXL »

tw04 wrote: post #38
fipar wrote: post #27

Earlier quoted context omitted.

I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. This usually works well, under the rationale that "upstream provider does this for a living, so they must be better than us at this", but if you have too unique needs (or are just a bit "unlucky"), it can fail too. All this to say that the cloud isn't magic. From a risk/e…

>I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. Isn't that literally supposed to be the sales pitch for the cloud? Get away from the infrastructure as a whole so you can focus on code, and let the cloud providers wave their magic wand to enable scaling? If you're saying now the story is: well rely on them to auto scal…

You have to know how to write code that fits into the cloud. You can't arbitrarily read/write to the file system, acting as if there's only one instance of the server running (if you plan to run hundreds or thousands). So even by waving the cloud 'magic wand', you still need to understand writing code in a cloud-friendly way. So in some sense, it's a shared responsibility between the vendor and engineering. You need to understand how to apply the tools being given to you.

Re: Slack’s Outage on January 4th 2021

#60

Post by hnlmorg »

dplgk wrote: post #9

> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…

HN comments suggested a broad plethora of things. I’m not surprised some happened on the right cause.
Post reply on HN