A terrible, horrible, no-good, very bad day at Slack
slack.engineering
A terrible, horrible, no-good, very bad day at Slack
1–10 of 282 posts
Re: A terrible, horrible, no-good, very bad day at Slack
#2The title is an allusion to a popular children’s book[1]. I’m assuming that an automated algorithm pulled the “very;” hopefully the mods will consider restoring it.
[1] https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...
Re: A terrible, horrible, no-good, very bad day at Slack
#3Interesting bit:
> The program which synced the host list generated by consul template with the HAProxy server state had a bug. It always attempted to find a slot for new webapp instances before it freed slots taken up by old webapp instances that were no longer running. This program began to fail and exit early because it was unable to find any empty slots, meaning that the running HAProxy instances weren’t getting their state updated
Re: A terrible, horrible, no-good, very bad day at Slack
#4I've just been bitten by this too:
The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change.
Re: A terrible, horrible, no-good, very bad day at Slack
#5What's the differences of using HAProxy or Envoy between using the cloud load balancers of AWS or Google Cloud?
Re: A terrible, horrible, no-good, very bad day at Slack
#6The title is an allusion to a popular children’s book[1]. I’m assuming that an automated algorithm pulled the “very;” hopefully the mods will consider restoring it. [1] https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...
HN automatically mangles titles in various ways I’m not fond of, dropping words that might or might not be significant, fiddling with capitalisation, &c., but the submitter can go back and edit the title back to what it was supposed to be, and the one time I’ve done that the system didn’t mangulate it again.
Re: A terrible, horrible, no-good, very bad day at Slack
#7Super interesting post. Following blog links, the timeline in https://slack.engineering/all-hands-on-deck-91d6986c3ee also offers a look at the play by play.
However, as far as I can read it, they have somewhat different views on the root cause?
"Soon, it became clear we had stale HAProxy configuration files, as a result of linting errors preventing re-rendering of the configuration."
vs.
"The program which synced the host list generated by consul template with the HAProxy server state had a bug. It always attempted to find a slot for new webapp instances before it freed slots taken up by old webapp instances that were no longer running. This program began to fail and exit early because it was unable to find any empty slots, meaning that the running HAProxy instances weren’t getting their state updated. As the day passed and the webapp autoscaling group scaled up and down, the list of backends in the HAProxy state became more and more stale."
Maybe a combination of the two?
Re: A terrible, horrible, no-good, very bad day at Slack
#8The title is an allusion to a popular children’s book[1]. I’m assuming that an automated algorithm pulled the “very;” hopefully the mods will consider restoring it. [1] https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...
That was the book that made me want to move to Australia!
Or Timbuktu.
Re: A terrible, horrible, no-good, very bad day at Slack
#9> One of the incident’s effects was a significant scale-up of our main webapp tier.
Sorry, I’m not very familiar with the terminology here; what is the “main webapp tier”?
Re: A terrible, horrible, no-good, very bad day at Slack
#10I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?
If you use something like CloudWatch, you can set an alarm for “insufficient data” for when a monitor stops reporting.