Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s Outage on January 4th 2021

slack.engineering

Post reply on HN
Print view View original

101–110 of 110 posts

Re: Slack’s Outage on January 4th 2021

#101

Post by gowld »

thu2111 wrote: post #56
solidasparagus wrote: post #44

Earlier quoted context omitted.

No, the cloud provides scalable infrastructure, but once you are in the 0.01% and you have very unique usage patterns, you still need to know how to set up your infrastructure for your needs. The difference is that instead of writing and managing a scalable cache, you just need to build the layer that knows to pre-provision for that scale/talk with AWS to make sure the system has sufficient capacity. The cloud isn't…

This feels like AWS apologism. Slack knew how to set up their infrastructure. Nothing in the postmortem implies AWS was misconfigured. AWS spotted the problem and fixed it entirely on their side. Nothing in this report suggests that Slack has unique usage patterns. Users returning to work after Christmas is not a phenomenon unique to Slack. Their problems were: 1. The AWS infrastructure broke due to an event as predi…

It wasn't scaling "back up". It was a huge spike as evertone refilled cache at the same time.

It's similar to Black Friday spikes Amazon handles themselves.

Re: Slack’s Outage on January 4th 2021

#102

Post by gowld »

delfaras wrote: post #37
thunderbong wrote: post #33

Earlier quoted context omitted.

But Slack has been around for longer than a year, right? Shouldn't they have noticed this happening earlier? I mean, considering Slack is mostly used as a workplace chat mechanism, they should have faced this kind of a scenario previously and had a solution for this by now.

Yeah but this year the number of people working from home that would connect to slack directly at the beginning of their work day must be much much larger than the other years

But also smaller because much fewer people went on a long traveling vacation away from laptop.

Re: Slack’s Outage on January 4th 2021

#103

Post by floatingatoll »

encoderer wrote: post #94
floatingatoll wrote: post #84

Earlier quoted context omitted.

Non-randomized wallclock integers. For example: “sleep 60 seconds”, “cron 0 * * * * command”, “X-Retry-After: 300” Found in: recurring jobs, backoff algorithms, oauth tokens. Found in: ops-created tasks, dev-released software.

I'm building something at Cronitor to help detect those hot-spots! If you want to learn more, email me: shane at cronitor.io

Tell us more here!

Re: Slack’s Outage on January 4th 2021

#105

Post by mandis »

Lammy wrote: post #29
saurik wrote: post #7

Earlier quoted context omitted.

I miss EC2 Classic :/. It always feels like the entire world of VPCs must have come from the armies of network engineers who felt like if the world didn't support all of the complexity they had designed to fix a problem EC2 no longer had--the tyranny of cables and hubs and devices acting as routers--that maybe they would be out of a job or something, and so rather than design hierarchical security groups Amazon just…

My assumption is it's an IPv4 address exhaustion thing too.

Amazon and IPv4 exhaustion?

https://www.techradar.com/in/news/amazon-has-hoarded-billion...

Re: Slack’s Outage on January 4th 2021

#106

Post by grumple »

Automated scaling has been a persistent problem for me, especially if I try to scale on simple metrics, or even worse (in Slack's case) on metrics that could potentially compete. The situations in which multiple metrics could compete are sometimes difficult to conceive, but it will always happen if you aren't performing something more sophisticated than "up if metric > value, down if I think you really have to look at metric-based autoscaling and say: is it worth the X% savings per month? Or would I rather avoid the occasional severe headaches caused by autoscaling messing up my day? Obviously this depends on company scale and how much your load varies. I'd rather have an excess of capacity than any impact on users.

Re: Slack’s Outage on January 4th 2021

#107

Post by paranoidrobot »

solidasparagus wrote: post #44
tw04 wrote: post #38

Earlier quoted context omitted.

>I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. Isn't that literally supposed to be the sales pitch for the cloud? Get away from the infrastructure as a whole so you can focus on code, and let the cloud providers wave their magic wand to enable scaling? If you're saying now the story is: well rely on them to auto scal…

No, the cloud provides scalable infrastructure, but once you are in the 0.01% and you have very unique usage patterns, you still need to know how to set up your infrastructure for your needs. The difference is that instead of writing and managing a scalable cache, you just need to build the layer that knows to pre-provision for that scale/talk with AWS to make sure the system has sufficient capacity. The cloud isn't…

> The cloud isn't some magic thing that solves all scaling problems, it's a tool that gives you strong primitives (and once you're a large enough customer, an active partner) to help you solve your scaling problems.

I don't think anyone who's got any reasonable level of experience is expecting that it's a magic wand.

There are, though some things in AWS (and for sure other cloud providers) where you get no useful signals or controls. It's entirely managed by the cloud provider, based on their own internal metrics and scaling behaviors.

Behind the scenes, their load balancer services don't give you indications of how heavily loaded they are - nor do you get to directly control how many/big those load balancers are.

In some parts you can hack around this by pre-warming infrastructure by generating fake traffic - but that assumes that you have those metrics and knowledge that you even need to do this.

This applies to all sorts of things - there's hidden caps and other capacity limits all over AWSs platform that you don't know about until you hit them. There's even capacity limits that you can know about, because they're publicly documented, but AWS lies and won't tell you the actual limit being applied to your account - the console and documentation says one thing, but in reality it's a lot lower.

If that capacity limit resulted in an outage, well, tough luck.

Re: Slack’s Outage on January 4th 2021

#108

Post by throwdbaaway »

tobobo wrote: post #35

The big takeaway for me here is that this “provisioning service” had enough internal dependencies that they couldn’t bring up new nodes. Seems like the worst thing possible during a big traffic spike.

> We run a service, aptly named ‘provision-service’, which does exactly what it says on the tin. It is responsible for configuring and testing new instances, and performing various infrastructural housekeeping tasks. Provision-service needs to talk to other internal Slack systems and to some AWS APIs.

The "configuring and testing new instances" part also sounds very fishy to me. Configuration should be done when creating the image and launch template, while testing should be the job of the load balancing layer. Why do we need a separate "provision-service" to piece everything together?

Re: Slack’s Outage on January 4th 2021

#109

Post by allannienhuis »

plaidfuji wrote: post #104

Maybe I’m being naive, but what I’m curious about is: how did their whole team communicate through this triage? I assume not Slack?

our emergency backup for slack is zoom. Horrible UX for group chats, but everyone already has it installed and it's quick and simple to set up a new room for each team. For temporary use you can put up with a fair bit of annoying behaviour or lack of features.

Re: Slack’s Outage on January 4th 2021

#110

Post by keyle »

keyle wrote: post #99

Didn't we just read a story about the exact same issue? Traffic picked up heavily on some website or app, AWS didn't auto-scale fast enough or at all and the very systems that are designed to be elastic just tumbled down to a grinding halt?

Update: it was Advent Of Code 2020, where he reported the exact same issue. The AWS auto scaling framework rocked a pooper when the site exploded on release day.
Post reply on HN