Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

11–20 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#11

Post by ketzo »

nik_0_0 wrote: post #7

Super interesting post. Following blog links, the timeline in https://slack.engineering/all-hands-on-deck-91d6986c3ee also offers a look at the play by play. However, as far as I can read it, they have somewhat different views on the root cause? "Soon, it became clear we had stale HAProxy configuration files, as a result of linting errors preventing re-rendering of the configuration." vs. "The program which synced th…

Honestly it’s a bit tough for me to parse, but the way I’m reading it,

1. Stale configs led to an overabundance of web apps, and then

2. Old instances of the web app couldn’t be removed because of the consul-template bug.

so, yes, a combination (in sequence) of the two.

Hard for me to be sure because I’m by no means knowledgeable on this stuff.

Re: A terrible, horrible, no-good, very bad day at Slack

#12

Post by jacquesm »

hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

This same sort of thing tends to creep into backup systems that work for a long time without if the backups are never restored. The more reliable the systems they backup the bigger the chance that a restore will fail when you need it most. So test your backups.

Re: A terrible, horrible, no-good, very bad day at Slack

#14

Post by ublaze »

vecio wrote: post #5

What's the differences of using HAProxy or Envoy between using the cloud load balancers of AWS or Google Cloud?

Cloud load balancers can be sneakily expensive. Few months ago, we spent a few weeks replacing an ELB with naive client side load balancing via round robin, which saves us > 200k/year. ELBs charge per byte transmitted, which seems reasonable, but can end up really expensive.

Re: A terrible, horrible, no-good, very bad day at Slack

#15

Post by harpratap »

hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

Chaos Engineering comes into play here. Deliberately break your platform to see if everything works as expected.

Re: A terrible, horrible, no-good, very bad day at Slack

#17

Post by EdwardDiego »

nickromano wrote: post #10
hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

If you use something like CloudWatch, you can set an alarm for “insufficient data” for when a monitor stops reporting.

Yep, and Grafana allows alerting on No Data states.

Re: A terrible, horrible, no-good, very bad day at Slack

#18

Post by jsmeaton »

saagarjha wrote: post #9

> One of the incident’s effects was a significant scale-up of our main webapp tier. Sorry, I’m not very familiar with the terminology here; what is the “main webapp tier”?

Apps these days might have several groups of services. For a simple case you might have a web tier serving http requests from customers, and you might have a Worker/background task tier. They usually scale independently.

Re: A terrible, horrible, no-good, very bad day at Slack

#19

Post by input_sh »

harpratap wrote: post #15
hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

Chaos Engineering comes into play here. Deliberately break your platform to see if everything works as expected.

To add to that, Netflix has open sourced a tool called Chaos Monkey which randomly terminates production instances so that you can be sure that your overall system is resilient to instance failures.

Re: A terrible, horrible, no-good, very bad day at Slack

#20

Post by brazzy »

input_sh wrote: post #19
harpratap wrote: post #15

Earlier quoted context omitted.

Chaos Engineering comes into play here. Deliberately break your platform to see if everything works as expected.

To add to that, Netflix has open sourced a tool called Chaos Monkey which randomly terminates production instances so that you can be sure that your overall system is resilient to instance failures.

I don't think that would have ever provoked this error mode, though.
Post reply on HN