Live data from Hacker News

Microsoft Azure Outage

azure.microsoft.com

131–140 of 178 posts

Re: Microsoft Azure Outage

#131

come on! give them some slack.. they probably aren't very experienced at managing their linux servers! ;)

Clearly neither did the developers of these apps which are now down who thought of spending as few pennies as possible and save a few other pennies with load balancing failover, and then expecting magic!

Re: Microsoft Azure Outage

#132

The idea of cloud storage being down is less of an issue - I don't like it, but I understand it. What bothers me about this is: 1. I was never notified of the outage. I noticed it myself when attempting to log into one of my VMs and then started looking for status updates. Sadly, the best status updates I got were here on Hacker News. 2. When my servers did come back up, at least one of my IP addresses had changed, w…

Do you having monitoring setup on your cloud vm, because you should have alerts triggering emails to notify you when this is happening.

Secondly, you are using an IP address and expecting that to be static? The recommended approach is to use a CNAME so you don't hit that issue, alternatively, you can have up to 5 Reserved-IPs per subscription and attach that Reserved-IP to your VM : New-AzureReservedIP from powershell

Edit : see http://azure.microsoft.com/blog/2014/05/14/reserved-ip-addre...

Re: Microsoft Azure Outage

#133

Reality call: ANY and ALL Cloud services, be it Google, Azure, AWS etc, will be down for hours at some point every few years.

Reality call: ANY and ALL services, be it local or remote, will be down for hours at some point every few years.

If it's my own fault, I can at least curse my own lack of knowledge and expertise, and I can strive to do better in the future.

When the cloud is down, all we can do is fiddle our thumbs and hope it doesn't happen again. Or maybe we could send an angry letter to Microsoft, and hope somebody reads it.

Re: Microsoft Azure Outage

#134

The idea of cloud storage being down is less of an issue - I don't like it, but I understand it. What bothers me about this is: 1. I was never notified of the outage. I noticed it myself when attempting to log into one of my VMs and then started looking for status updates. Sadly, the best status updates I got were here on Hacker News. 2. When my servers did come back up, at least one of my IP addresses had changed, w…

2. Azure does not guarantee that you keep your ip address by default. You should configure a cname if you use Azure Websites or get a reserved ip address, available with Cloud Services

Also as I found out, set the TTL on your CNAME very low.

Also this is a PITA if you use the @ entry in your DNS.

Re: Microsoft Azure Outage

#135
post #133

Earlier quoted context omitted.

Reality call: ANY and ALL services, be it local or remote, will be down for hours at some point every few years.

If it's my own fault, I can at least curse my own lack of knowledge and expertise, and I can strive to do better in the future. When the cloud is down, all we can do is fiddle our thumbs and hope it doesn't happen again. Or maybe we could send an angry letter to Microsoft, and hope somebody reads it.

It's about abstracting away the cost of it being your own fault. Realistically the cost of employing enough people, and buying enough hardware, to provide anything close to 99.X% uptime is much more than punting that over to a Cloud Provider.

Re: Microsoft Azure Outage

#136

Azure support is probably the worst one I had ever deal with. When my account (and service itself) stopped working, I haven't received any email. When I tried to sign in, all I got was some generic error saying "There's something wrong with your account". My services of course were down and I couldn't do ANYTHING. I've contacted the support to learn that my account has been blocked (!) because there was some suspicio…

I had a similar situation with DigitalOcean. Droplets were taken offline with no notification whatsoever, only answered after I submitted a ticket. This has happened to me twice.

Re: Microsoft Azure Outage

#137

The idea of cloud storage being down is less of an issue - I don't like it, but I understand it. What bothers me about this is: 1. I was never notified of the outage. I noticed it myself when attempting to log into one of my VMs and then started looking for status updates. Sadly, the best status updates I got were here on Hacker News. 2. When my servers did come back up, at least one of my IP addresses had changed, w…

Sadly this is a common situation. They appear to hold off making any public notice (including often on their own "service dashboards") until support forums are screaming with upset users.

Google App Engine has had numerous outages like this, the only one I can find any public documentation for being a 6 hour outage in 2012: http://googleappengine.blogspot.co.uk/2012/10/about-todays-a... (and let's not forget the old-style Datastore corruption incident, where every App Engine user got to manually merge split-brain database tables after a messed up failover)

Re: Microsoft Azure Outage

#138
post #133

Earlier quoted context omitted.

If it's my own fault, I can at least curse my own lack of knowledge and expertise, and I can strive to do better in the future. When the cloud is down, all we can do is fiddle our thumbs and hope it doesn't happen again. Or maybe we could send an angry letter to Microsoft, and hope somebody reads it.

It's about abstracting away the cost of it being your own fault. Realistically the cost of employing enough people, and buying enough hardware, to provide anything close to 99.X% uptime is much more than punting that over to a Cloud Provider.

I've found very few cases where "punting that over to a cloud provider" has been remotely cost effective for base load. It's gotten closer over the years, but the gap is still massive for all but some very specific types of workloads.

It's great for convenience, and it's great for managing without certain skillsets that may be hard to obtain, and it's great for temporary capacity, but it's not cheap.

Re: Microsoft Azure Outage

#139
Judging by how cloud services "frequently" go down when everything is normal, it makes me wonder what would happen in case of a real problem (volcano eruption, social unrest, nuclear disaster, alien invasion ...). I still don't get the cloud infatuation, and no you don't have to get off my lawn, I'm "only" 36 (yeah I know, in IT I'm already a dinosaur).

Re: Microsoft Azure Outage

#140
post #138

Earlier quoted context omitted.

It's about abstracting away the cost of it being your own fault. Realistically the cost of employing enough people, and buying enough hardware, to provide anything close to 99.X% uptime is much more than punting that over to a Cloud Provider.

I've found very few cases where "punting that over to a cloud provider" has been remotely cost effective for base load. It's gotten closer over the years, but the gap is still massive for all but some very specific types of workloads. It's great for convenience, and it's great for managing without certain skillsets that may be hard to obtain, and it's great for temporary capacity, but it's not cheap.

It's not cheap to have any confidence of any uptime realistically at all. The thing is that most people either live without that guarantee, or just get lucky enough not to care. It becomes problematic if you've made promises to others about uptime that are built on a house of sand.

Unless your base load cloud costs are more than the cost of full time, ready at a moments notice, experienced ops people you don't get close to any guarantee of uptime by non-managed hosting. The salary cost alone of that is substantial, let alone hardware spread across multiple locations. My firm pays at least 7 figures a year on IT ops and don't come close to 99.9% uptime across everything.

Post reply on HN