Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s Outage on January 4th 2021

slack.engineering

Post reply on HN
Print view View original

91–100 of 110 posts

Re: Slack’s Outage on January 4th 2021

#91

Post by sokoloff »

0dmethz wrote: post #88
sokoloff wrote: post #79

Earlier quoted context omitted.

"Hey Boss, this system that our team selected and configured, behaved as documented but not in a way that protected our customers' experience. It's Amazon's fault, not ours..." If someone came to me with that, I'd educate them on how I saw it quite differently, politely but firmly.

Unless I'm misunderstanding something the system did not perform as documented. It should have scaled, it didn't. When a critical piece of infrastructure fails under massive load I'm not sure it it'll help much when you politely tell your engineers they fucked up for not anticipating it. You learn lessons. Both Slack and AWS seem to have learnt lessons here.

I agree with much of what you say, but if you change it to "It's Amazon's fault, not ours", that's where I diverge.

Slack did fuck up here, as evidenced by the outage and you seem to at least partially agree by the fact that Slack learned a lesson. Further, I think that "understanding how your system scales up from a low baseline to a high level of utilization (such as Black Friday/Cyber Monday for e-commerce, or special event launches, or a SuperBowl ad landing page)" is a standard, "par for the course" cloud engineering topic to be on top of nowadays.

Re: Slack’s Outage on January 4th 2021

#92

Post by t0mas88 »

rrrrrrrrrrrryan wrote: post #74
cett wrote: post #13

Earlier quoted context omitted.

Though the nuance is Slack did know how to handle it, AWS didn't.

Some of my co-workers came from active.com (a website that lets people register for marathons and events). The infrastructure had to handle massive spikes because registrations for big races would open all at once, so scalability was everything. They explained to me that they'd intentionally slam the production website with external traffic a couple of times per year, at a scheduled time in the middle of the night. L…

Very good test. The guys at iracing.com should have done this before organising the e-sports Daytona 24 hours race last week, it was by far their largest event (boosted by Covid lockdown). It crashed their central scheduling service with a database deadlock. Classic case of a bug you only find under heavy load.

Re: Slack’s Outage on January 4th 2021

#93

Post by ViViDboarder »

gen220 wrote: post #86
jrockway wrote: post #62

The thing that always worries me about cloud systems are the hidden dependencies in your cloud provider that work until they don't. They typically don't output logs and metrics, so you have no choice to pray that someone looks at your support ticket and clicks their internal system's "fix it for this customer" button. I'll also say that I'm interested in ubiquitous mTLS so that you don't have to isolate teams with VP…

Some customers have a hard requirement that their slack instances be behind a unique VPC. Other customers are easier to sell to if you sprinkle some “you’ll get your own closed network” on top of the offer, if security is something they’ve been burned by in the past. I agree with you the mTLS is the future. It exists within many companies internally (as a VPC alternative!) and works great. There’s some problems aroun…

mTLS appears to work great between servers, but I’ve been unable to get my iPhone to authenticate with a web server via Safari using mTLS. Even after installing the cert, it never presents it.

I wish it were better supported though.

Re: Slack’s Outage on January 4th 2021

#94

Post by encoderer »

floatingatoll wrote: post #84
nicoburns wrote: post #73

Earlier quoted context omitted.

> The mistake of ignoring the maxage of the request queue is perhaps the second most common blind spot in every Ops team I've ever worked with. No insult to my fellow Ops folks, but we've got to stop overlooking this. What's the first?

Non-randomized wallclock integers. For example: “sleep 60 seconds”, “cron 0 * * * * command”, “X-Retry-After: 300” Found in: recurring jobs, backoff algorithms, oauth tokens. Found in: ops-created tasks, dev-released software.

I'm building something at Cronitor to help detect those hot-spots! If you want to learn more, email me: shane at cronitor.io

Re: Slack’s Outage on January 4th 2021

#95

Post by nickthemagicman »

jjtheblunt wrote: post #4
nickthemagicman wrote: post #3

Wow. The trail leads back to AWS. Wasn't there a number of other companies that were down around that same time or was that a different time?

Is this the recent event you refer to? https://aws.amazon.com/message/11201/

Ah yes, that was the one I was referring to. Looks like a different event.

Re: Slack’s Outage on January 4th 2021

#97

Post by whatisthiseven »

gen220 wrote: post #87
whatisthiseven wrote: post #19

Earlier quoted context omitted.

Agreed, I always thought VPC and all that complexity was a big step backwards. My org is moving from a largely managed network into AWS, and now we have to configure the whole network and external gateways ourselves? What engineer wants to do this? VPCs are virtual, but I don't need VPCs, I need the entire network layer virtualized and abstracted. As you suggested,just grouping devices in a single network and saying…

People use it as a (imo lazy) form of enforcing access control. If two services aren’t in the same VPC, they can’t talk to each other. It theoretically limits the damage of a rogue node. Of course, it also creates a ton of overhead and complexity, because you still have to wire all your VPCs together to implement things like monitoring and log aggregation, for example. As other people have suggested, the better solut…

Sure, but we also do the things you suggested.

However, if you are relying on defense in depth for security, then having them be network separate helps prevent internal DDoS attacks, whether malicious or not.

Enforcing security across the entire network layer has many positives. But I don't want to be messing with the lower levels, and those lower levels all have the same security models and solutions as one another, at least if you view them at a high level.

VPCs have value as a security and availability solution, I just don't want to have to configure it to get what could be an automatic benefit.

Re: Slack’s Outage on January 4th 2021

#98

Post by kortilla »

cett wrote: post #13
dplgk wrote: post #9

> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…

Though the nuance is Slack did know how to handle it, AWS didn't.

No, using AWS does not absolve you of that responsibility. The game of paying a vendor to be an engineer is that you have to have strategies to test this kind of stuff.

Slack didn’t know how to handle it, they paid AWS hoping the product did what it said on the tin. They didn’t test for this case and got bit.

They have millions of clients they could have coordinated to load test this stuff by picking some time to disable the cache and fallback to cache if it failed.

Re: Slack’s Outage on January 4th 2021

#99

Post by keyle »

Didn't we just read a story about the exact same issue?

Traffic picked up heavily on some website or app, AWS didn't auto-scale fast enough or at all and the very systems that are designed to be elastic just tumbled down to a grinding halt?

Re: Slack’s Outage on January 4th 2021

#100

Post by bobthebuilders »

jeffrallen wrote: post #50

tldr: "we added complexity into our system to make it safer and that complexity blew us up. Also: we cannot scale up our system without everything working well, so we couldn't fix our stuff. Also: we were flying blind, probably because of the failing complexity that was supposed to protect us." I am really not impressed... with the state of IT. I could not have done better, but isn't it too bad that we've built these…

The thing is anyone can build a system that can scale out to Slack's level given enough machines and money. What's harder is scaling out to that level and not burning gobs of cash.

It's similar to the whole buildings a long time ago last much longer than those today. Its true in the literal sense, but it ignores the fact that we've gotten at reducing the cost of stuff like skyscrapers and bridges.

In our pursuit of efficiency, we do things like JIT delivery, dropshipping, scaling, building to the minimum spec. Sometimes, we get it wrong and it comes tumbling down (covid, HN hug of death, earthquakes).

Post reply on HN