Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

41–50 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#41

Post by WJW »

throwaway0a5e wrote: post #37
rkagerer wrote: post #25

Earlier quoted context omitted.

There's a reason the military does drills. Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. It's especially critical for disaster recovery (a DR plan that's never tested isn't worth the paper it's written on). Consider going one step further and randomly injecting artificial failures into production shards, so handling them becomes a regular affair…

>Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. All those man hours cost money. Is it worth it? Depends on how much downtime costs you.

Most companies don't know how much downtime costs them and from their behaviour seem to treat it either as $0/hr or $infinite/hr. To be fair, it's difficult to estimate correctly. Apart from the direct loss of income there is also reputation loss, possibly reduced signups and increased churn, etc. It's not at all constant either, one outage per 10 years will give you a radically different reputation than one outage per week.

Re: A terrible, horrible, no-good, very bad day at Slack

#42

Post by friseurtermin »

Stratoscope wrote: post #8
kd5bjo wrote: post #2

The title is an allusion to a popular children’s book[1]. I’m assuming that an automated algorithm pulled the “very;” hopefully the mods will consider restoring it. [1] https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...

That was the book that made me want to move to Australia! Or Timbuktu.

Well, did you end up moving to Australia?

Re: A terrible, horrible, no-good, very bad day at Slack

#43

Post by waheoo »

hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

This is what end to end and integration testing is for.

You would set up an integration test to trigger the state that results in this alert.

Re: A terrible, horrible, no-good, very bad day at Slack

#45

Post by user5994461 »

ublaze wrote: post #14
vecio wrote: post #5

What's the differences of using HAProxy or Envoy between using the cloud load balancers of AWS or Google Cloud?

Cloud load balancers can be sneakily expensive. Few months ago, we spent a few weeks replacing an ELB with naive client side load balancing via round robin, which saves us > 200k/year. ELBs charge per byte transmitted, which seems reasonable, but can end up really expensive.

They charge per byte and per request if I remember well, which can be really expensive for serving both small API call and large files.

Another limitation is that the ELB only works in AWS to AWS instances in the same location. Gotta use something else for geographic load balancing and for other datacenters.

Re: A terrible, horrible, no-good, very bad day at Slack

#46

Post by waheoo »

realtalk_sp wrote: post #16

Pretty fantastic case study in the perils of complex systems. Is there any place where these types of post-mortems are collected? Could be a very valuable resource for systems engineers.

I dunno, it sounds more like they just got caught with their pants down not finishing a migration to some new shiny.

Also looks like they do blue green but dont confirm the replacements are live before considering the greens new blue.

Re: A terrible, horrible, no-good, very bad day at Slack

#47

Post by rorykoehler »

WJW wrote: post #39
jwr wrote: post #36

Earlier quoted context omitted.

> because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours This is based on faith — there might, or might not be a specialized 24/7 devops team who runs these things better than you. My rational mind has trouble accepting things based on faith, which is also…

The specialized 24/7 devops team (if it's there) also has a few thousand other customers instead of being there just for you. They might have other priorities at any given moment. It's not like AWS or GCP are renowned for the quality of their customer service.

AWS is pretty good whenever we've needed them. Google? Probably not.

Re: A terrible, horrible, no-good, very bad day at Slack

#49

Post by user5994461 »

nik_0_0 wrote: post #7

Super interesting post. Following blog links, the timeline in https://slack.engineering/all-hands-on-deck-91d6986c3ee also offers a look at the play by play. However, as far as I can read it, they have somewhat different views on the root cause? "Soon, it became clear we had stale HAProxy configuration files, as a result of linting errors preventing re-rendering of the configuration." vs. "The program which synced th…

The way they are doing things. HAProxy is configured with a fixed amount of slots. This effectively acts as a maximum limit, so should be enough for the running instances + newer instances coming up anytime due to auto scaling.

They have a tool listening to applications starting and shutting down. It's adjusting the configuration live while running to remove shut down instances (free a slot) and put in newer instances (find a free slot and reconfigure).

From the explanation on that day, there were more instances than usual due to high load. It seems the tool was looking for a free slot at some point and found none and crashed.

I'd say, it's an issue with capacity planning because they didn't plan enough slots for their infra on high load and an issue with the tool because it shouldn't fail silently when out of slots.

Re: A terrible, horrible, no-good, very bad day at Slack

#50

Post by Ntrails »

WJW wrote: post #41
throwaway0a5e wrote: post #37

Earlier quoted context omitted.

>Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. All those man hours cost money. Is it worth it? Depends on how much downtime costs you.

Most companies don't know how much downtime costs them and from their behaviour seem to treat it either as $0/hr or $infinite/hr. To be fair, it's difficult to estimate correctly. Apart from the direct loss of income there is also reputation loss, possibly reduced signups and increased churn, etc. It's not at all constant either, one outage per 10 years will give you a radically different reputation than one outage p…

It's also working out which outages matter the most. A 10 day outage is not equivalent to 10 1 day outages, the latter has a multitude of important reputational risks - but the former is quite possibly enough to end the business completely.
Post reply on HN