Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

51–60 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#51

Post by donavanm »

harpratap wrote: post #24
nickromano wrote: post #10

Earlier quoted context omitted.

If you use something like CloudWatch, you can set an alarm for “insufficient data” for when a monitor stops reporting.

I don't think it would have been enough for this exact situation. The solution you are providing is for cases where x==nil but in their case there was never an x, so you cannot do x==nil checks unless you deliberately set up x.

Not quite. Cloudwatch provides for both behaviors. You can treat periods with no datapoints as “breaching” the threshold criteria. You can also independently alert for “insufficient data” where the source is unavailable, doesnt contain enough data to evaluate, etc.

In the past I did both. Always emit a 0 datapoint every period + treat missing datapoints as breaching to discover if an application wasnt consistently emitting metrics. In addition a lower severity Insufficient Data alert was used to discover/validate when a meteic stream literally didnt exist (normally through misconfiguration of metric & alarm dimensions).

Docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

Re: A terrible, horrible, no-good, very bad day at Slack

#52

Post by battery423 »

hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

We use Google Cloud Alerting to watch that our alerting works.

I would have used some Heartbeatservice but thats not feasible in our environment.

Re: A terrible, horrible, no-good, very bad day at Slack

#53

Post by dikei »

hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

> Who watches the watchers?

You would need multiple watchers who watch one another. Of course, if all watchers die at the same time, you're out of luck.

This is only part of the solution though, because in order to write and test effective alerting rules, you need to repeatedly and frequently test them with either real or simulated failures, and it can get expensive.

Re: A terrible, horrible, no-good, very bad day at Slack

#54

Post by foota »

jwr wrote: post #36
sudhirj wrote: post #31

This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and b…

> because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours This is based on faith — there might, or might not be a specialized 24/7 devops team who runs these things better than you. My rational mind has trouble accepting things based on faith, which is also…

What? You can totally run disturbed SQL without data loss. Did you mean to imply something in particular?

Re: A terrible, horrible, no-good, very bad day at Slack

#57

Post by jjoonathan »

rorykoehler wrote: post #47
WJW wrote: post #39

Earlier quoted context omitted.

The specialized 24/7 devops team (if it's there) also has a few thousand other customers instead of being there just for you. They might have other priorities at any given moment. It's not like AWS or GCP are renowned for the quality of their customer service.

AWS is pretty good whenever we've needed them. Google? Probably not.

Google swears gcloud support is good, of course, but I've never actually used it -- but then again, I've never actually needed it.

Meanwhile, I need AWS support constantly because their entire platform is a gigantic social experiment in minimum viable products. How crusty are people willing to tolerate? Evidently: very, very crusty.

Re: A terrible, horrible, no-good, very bad day at Slack

#58

Post by sudhirj »

WJW wrote: post #39
jwr wrote: post #36

Earlier quoted context omitted.

> because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours This is based on faith — there might, or might not be a specialized 24/7 devops team who runs these things better than you. My rational mind has trouble accepting things based on faith, which is also…

The specialized 24/7 devops team (if it's there) also has a few thousand other customers instead of being there just for you. They might have other priorities at any given moment. It's not like AWS or GCP are renowned for the quality of their customer service.

This is actually a plus point if you ask me. These few thousand customers are all operating on the same racks as me - in an environment like AWS or GCP there's no special part of the datacenter reserved for fancy customers. The billion-dollar customers all run their VMs on the same rack as me. So whatever work the ops teams do to keep things reliable benefits me as much as the biggest customers.

Re: A terrible, horrible, no-good, very bad day at Slack

#59

Post by jjoonathan »

gchamonlive wrote: post #33
sudhirj wrote: post #31

This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and b…

You can't assume that, these costs, specifically the server less stack at AWS, scale up quite opaquely. You can put billing alerts in place, but once you tie your infrastructure to one specific serverless vendor, even if you identify a harmful cost scale, you can't easily mitigate it. I am not saying serverless is expensive, all I am advocating is extensive planning and preparing before adopting any particular server…

Not just costs! AWS services are opaque performance minefields, too.

If you need reliable low latency -- which is probably core to the slack experience, or any GUI in a competitive space, really -- lambda is not a good option. They just don't share your priorities. AWS support will happily waste a lot of your time chasing Just One More Trick to mitigate the problem, though.

Re: A terrible, horrible, no-good, very bad day at Slack

#60

Post by mcbain »

jwr wrote: post #36
sudhirj wrote: post #31

This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and b…

> because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours This is based on faith — there might, or might not be a specialized 24/7 devops team who runs these things better than you. My rational mind has trouble accepting things based on faith, which is also…

Could you describe what you mean by a distributed SQL database?

I don’t think RDS would generally fit that concept as I understand it. Aurora’s data store possibly but you choose to use that specifically.

Post reply on HN