Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft Azure Outage

azure.microsoft.com

Post reply on HN
Print view View original

41–50 of 178 posts

Re: Microsoft Azure Outage

#41

Post by toomuchtodo »

photorized wrote: post #36
Encosia wrote: post #32

Earlier quoted context omitted.

For what it's worth, I've been watching several Azure-hosted sites that I control and they've been coming back online sequentially (and are all back online now). Whatever they're fixing, it seems to be taking some time, but is progressing steadily at a good pace in the last hour.

Mine have been gradually coming back, too. Timing couldn't have been any better for me. Some Alanis material there: https://twitter.com/bizspark/status/534858596748906496 Now I'll have to distribute between AWS and Azure, too.

Why not stick with AWS? (Just curious!)

Re: Microsoft Azure Outage

#42

Post by toomuchtodo »

inglor wrote: post #33

Our sites have been down for more than 3 hours now. EDIT2: Now the databases are down, this is costing us a lot of money. EDIT: Just went up again. It would be great if anyone knows how to mitigate these in the future - what can I do to protect myself against this in the future? (Except leave Azure)

As others have mentioned, multiple cloud providers, service checks, and withdrawing bad providers at DNS.

Re: Microsoft Azure Outage

#43

Post by joshuak »

inglor wrote: post #33

Our sites have been down for more than 3 hours now. EDIT2: Now the databases are down, this is costing us a lot of money. EDIT: Just went up again. It would be great if anyone knows how to mitigate these in the future - what can I do to protect myself against this in the future? (Except leave Azure)

Major outages should absolutely weigh into your decisions as to what platform to use. That being said you can mitigate the effect of instability by engineering your app to failover to other availability zones or even to another cloud platform (depending on your app) if the entire platform goes down.

Obviously there is a segnifigant cost associated with engineering this level of cross platform redundancy which is why reliability is an important factor in making your platform choices. If you can tolerate some downtime, you can be more flexible, otherwise it will costs one way or the other.

In any case you should consider having a user notification site setup on a completely different service (or two) so that when things go wrong you can redirect everyone to that site to keep your customers informed. This is especially important when you have partial outages that could create inconstancies in your database or application state if you where to continue to allow users to interact with it in a degraded state.

Re: Microsoft Azure Outage

#44

Post by keithwarren »

BeeDunc wrote: post #38

This outage exposes the clowns that actually chose Azure as their cloud provider. If you use AMZN and it goes down, at least you're in good company, with the likes of Netflix, Twitter, Instagram, and so on. It's like yeah, I'm big like they are. So what, it went down, so is Netflix. What does your client/customer think of you being on Azure? That you chose the crappy solution because your low-tech infrastructure stil…

Over 80% of the Fortune 500 run on Azure.

20% of Azure VMs are Linux.

You are not well informed.

Re: Microsoft Azure Outage

#45

Post by photorized »

photorized wrote: post #36
Encosia wrote: post #32

Earlier quoted context omitted.

For what it's worth, I've been watching several Azure-hosted sites that I control and they've been coming back online sequentially (and are all back online now). Whatever they're fixing, it seems to be taking some time, but is progressing steadily at a good pace in the last hour.

Mine have been gradually coming back, too. Timing couldn't have been any better for me. Some Alanis material there: https://twitter.com/bizspark/status/534858596748906496 Now I'll have to distribute between AWS and Azure, too.

They all have outages periodically, in my experience.

Re: Microsoft Azure Outage

#46

Post by dangrossman »

ZoF wrote: post #34
craigvn wrote: post #25

Earlier quoted context omitted.

It is totally frustrating, but at the end of the day similar outages happen with all cloud providers.

Can you point to an example of this happening on AWS in multiple regions simultaneously?

Yes. The day half the internet seemed to be down. It took Amazon more than 24 hours to recover, and having your services in multiple availability zones did not shield you from the failure.

http://aws.amazon.com/message/65648/

Re: Microsoft Azure Outage

#47

Post by ZoF »

dangrossman wrote: post #46
ZoF wrote: post #34

Earlier quoted context omitted.

Can you point to an example of this happening on AWS in multiple regions simultaneously?

Yes. The day half the internet seemed to be down. It took Amazon more than 24 hours to recover, and having your services in multiple availability zones did not shield you from the failure. http://aws.amazon.com/message/65648/

Ah, cool, thanks for the info.

Interesting to think about the potentially compounding failure modes these services are dealing with.

I'll have to look that incident up to check out their postmortem vs. what Microsoft ends up putting out.

Thanks again ~

Re: Microsoft Azure Outage

#48

Post by cddotdotslash »

photorized wrote: post #26

I feel like an idiot. MS featured my Azure startup today, quoting me about overall stability etc (which has been the case for us, until today). They then proceeded to go down, taking all our production systems with them. (yes we do have AWS, too) Sigh.

Just curious - if you have AWS too, then why did it take everything down? Can't you just swap the DNS?

Re: Microsoft Azure Outage

#49

Post by inglor »

joshuak wrote: post #43
inglor wrote: post #33

Our sites have been down for more than 3 hours now. EDIT2: Now the databases are down, this is costing us a lot of money. EDIT: Just went up again. It would be great if anyone knows how to mitigate these in the future - what can I do to protect myself against this in the future? (Except leave Azure)

Major outages should absolutely weigh into your decisions as to what platform to use. That being said you can mitigate the effect of instability by engineering your app to failover to other availability zones or even to another cloud platform (depending on your app) if the entire platform goes down. Obviously there is a segnifigant cost associated with engineering this level of cross platform redundancy which is why…

Thanks! This is very helpful.

Our big hosted site is hosted in Europe is actually working but our blogs and a news website are both down. We offer a paid service at 600$ a year and if the main site was down it would be very bad for our reputation.

Our DNS points to Azure on all these domains and things are hosted as "Azure Web Site" - how would notifications work if Azure itself is failing? Would I need to proxy the traffic through elsewhere?

Are there any services that solve this problem for me? I really don't mind paying a few dollars every month and not worry about this.

Re: Microsoft Azure Outage

#50

Post by inglor »

barkingllama wrote: post #39
inglor wrote: post #33

Our sites have been down for more than 3 hours now. EDIT2: Now the databases are down, this is costing us a lot of money. EDIT: Just went up again. It would be great if anyone knows how to mitigate these in the future - what can I do to protect myself against this in the future? (Except leave Azure)

On site disaster recovery? Off site disaster recovery? Split your hosting between multiple providers? It really depends on how much risk you're willing to accept, and how much that is worth to you. It can be quantified via revenue lost, but reputation is much harder to put a number on.

If I had to quantify this - 3 hours * 3 people who can't work and publish posts + about a week of marketing costs for damaged rep (apologies, PR, ads for exposure). I'd say that for the very least this cost us at least 1000$ and probably north of 3000$.

This is not the first time this has happened in the last two months (after a relatively reliable year). The problem is I'm not sure any other hosting provider would do any better.

Post reply on HN