Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS connectivity issues, but health dashboard says everything fine

news.ycombinator.com

Post reply on HN
Print view View original

61–67 of 67 posts

Re: Tell HN: AWS connectivity issues, but health dashboard says everything fine

#61

Post by hedgehog_irl »

autotune wrote: post #57
hedgehog_irl wrote: post #55

Earlier quoted context omitted.

So your logic is to accept poor quality service to keep your service online rather than trying to do better and improve service. So you are saying that rather than rewarding a company trying to do better just accept poor service from Aws.How is this better than "hosting on some dodgy start-up" This is nothing to do with my personal beliefs or opinion I'm trying to understand why it's accepted from Aws but not others…

My logic is to build highly resilient infrastructure given the constraints available. Your definition of "poor service" is not what I have experienced in my 10 year career as a SRE, because I build around your definition of what makes it poor and make it work as it should. It's called chaos engineering, and companies like Netflix have been doing it for years with their Chaos Monkey tool and SRE practices. Doesn't mat…

Ok simply question. Would you accept any other infra service provider having such poor customer service and not provided updated status of an outage/disruption for 45 min.

Re: Tell HN: AWS connectivity issues, but health dashboard says everything fine

#62

Post by autotune »

hedgehog_irl wrote: post #61
autotune wrote: post #57

Earlier quoted context omitted.

My logic is to build highly resilient infrastructure given the constraints available. Your definition of "poor service" is not what I have experienced in my 10 year career as a SRE, because I build around your definition of what makes it poor and make it work as it should. It's called chaos engineering, and companies like Netflix have been doing it for years with their Chaos Monkey tool and SRE practices. Doesn't mat…

Ok simply question. Would you accept any other infra service provider having such poor customer service and not provided updated status of an outage/disruption for 45 min.

You are deliberately avoiding the counter-points I already specifically addressed in response to that question to the point we are stuck in a loop, so I am going to leave this thread now. If you feel you can do a better job at SRE with your current mindset and believe you are better at choosing which cloud providers are worth using for an org, I welcome you to try.

Re: Tell HN: AWS connectivity issues, but health dashboard says everything fine

#63

Post by lamontcg »

leesalminen wrote: post #58
remus wrote: post #54

Earlier quoted context omitted.

I understand the frustration, but Im not convinced monitoring at large scale is that straightforward. The core question is: what constitutes degraded service? Would you say a service is experiencing downtime every time a 500 response is served? If you're serving millions to billions of requests/sec it seems a bit disproportionate to marka service down after a single 500 error, so then you need to work out some kind o…

We were pretty liberal with posting to our status. page for years and thought it was The Right Thing to do. I still do, to a point. But, what ended up happening was a competitor who didn't have a status page at all would use our status page against us in the sales process. They just never mentioned their lack of a status page to compare to. This was the same competitor who went 100% down for ~4 days during the busies…

Before 2003-ish Amazon used to have a static "gonefishing" page on www.amazon.com that was manually triggered during outages. Due to newspaper reporters writing scripts that would detect the GF pages they were removed and the site was allowed to just spew 500s for whatever segment of critical pages was busted.

Re: Tell HN: AWS connectivity issues, but health dashboard says everything fine

#64

Post by salil999 »

hedgehog_irl wrote: post #26

Everyone seems to overlook the point here. That yet again Amazon were slow as hell to be honest with their customers. I get it up down reports help but why do you keep using a service which lies to you about availability. I've read on HN in the past how the dashboard can only be updated to reflect an issue with approval. (Comments section on a similar posting, believe it if you wish). So why not move to a hosting com…

Good luck convincing your company "Hey because they were 45 minutes late in informing us we need to move all our cloud to a different provider."

Updating a dashboard can easily be an automated process but for business reasons it is not. AWS did not "lie" about the incident - they are extremely transparent for all outages and disruptions (btw this was a disruption - not an outage). They stated on the issue the exact time frame for when the issue started and when it ended.

Is it bad they were late? Definitely. AWS has a history of being late due to the sheer scale it works at. I've caused an outage myself when I used to work there and updating the dashboard requires several higher ups to understand what exactly the issue is and what is considered to be worthy of "informing of an incident." These processes take time. Is it perfect? Absolutely not. But there are legitimate reasons for it.

I'm not sure why you think Jeff is involved here. This kind of disruption isn't enough to warrant someone as high as Jeff to be involved.

As for SLA violations, AWS public SLAs for every service and they credit your account if it ever dips below those defined thresholds. And as for caring I don't know a single cloud provider with the level of great customer support AWS has. This is extremely opinionated but this is what I've observed in the industry.

I would recommend people to use AWS monitoring. But having some of your own basic internal dashboards / metrics is also worth having.

Re: Tell HN: AWS connectivity issues, but health dashboard says everything fine

#65

Post by everfrustrated »

A reminder that the public and personal health dashboards are not the the only port of call.

If you pay for the top tier of AWS support, if you have a suspected outage you'd be paging in AWS who will pick up the phone and start debugging your problem.

If your business depends on AWS you don't sit around clicking refresh on a status page hoping it might be updated.

Re: Tell HN: AWS connectivity issues, but health dashboard says everything fine

#66

Post by coredog64 »

everfrustrated wrote: post #65

A reminder that the public and personal health dashboards are not the the only port of call. If you pay for the top tier of AWS support, if you have a suspected outage you'd be paging in AWS who will pick up the phone and start debugging your problem. If your business depends on AWS you don't sit around clicking refresh on a status page hoping it might be updated.

At some level of spend, your account team will know what services you use and know when those services are having higher than normal error rates.

Re: Tell HN: AWS connectivity issues, but health dashboard says everything fine

#67

Post by merek »

salil999 wrote: post #64
hedgehog_irl wrote: post #26

Everyone seems to overlook the point here. That yet again Amazon were slow as hell to be honest with their customers. I get it up down reports help but why do you keep using a service which lies to you about availability. I've read on HN in the past how the dashboard can only be updated to reflect an issue with approval. (Comments section on a similar posting, believe it if you wish). So why not move to a hosting com…

Good luck convincing your company "Hey because they were 45 minutes late in informing us we need to move all our cloud to a different provider." Updating a dashboard can easily be an automated process but for business reasons it is not. AWS did not "lie" about the incident - they are extremely transparent for all outages and disruptions (btw this was a disruption - not an outage). They stated on the issue the exact t…

> I would recommend people to use AWS monitoring

Use a monitoring service to monitor the provider of the monitoring service? Wouldn't it be better to use a monitoring service hosted on a totally different provider?

I'm not even sure running your own monitoring is sufficient in this case. Sure it's useful to have, but when something goes wrong, the first thing I want to know is if it's us or them. If it's us, I/the team scramble to fix it in a panicked frenzy. If it's them (the cloud provider), and they acknowledge it early, even a simple "we're investigating an issue with X", we can at least take some comfort from the fact that it's out of our hands.

If we just don't know the cause, we assume it's us and jump into panicked frenzy mode. Panicked frenzy days are the worst days of my life, especially if it's discovered that it was all in vain.

Post reply on HN