Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft Azure suffers outage after cooling issue

datacenterdynamics.com

Post reply on HN
Print view View original

51–60 of 113 posts

Re: Microsoft Azure suffers outage after cooling issue

#51

Post by i_have_to_speak »

Theodores wrote: post #20
ZainRiz wrote: post #16

Earlier quoted context omitted.

That's why T-Mobile's on-call engineers carry around AT&T phones. (source: friend who's an engineer at T-Mobile)

That is a top 'did you know' factoid that I am sure I will tell others. But do AT+T engineers carry T-Mobile phones? If yes then they should put themselves together a deal so that none of the on-call engineers have to worry about running up big bills using their phones. When there are freak weather events they are all in it together.

"factoid": I don't think it means what you think it means.

https://www.merriam-webster.com/dictionary/factoid

Re: Microsoft Azure suffers outage after cooling issue

#52

Post by dmarlow »

mrep wrote: post #48
dmarlow wrote: post #40

The worst part has been the poor communication. If they were to give clearer insight from the get go, that'd give me more confidence and patience. Saying "check back in 2 hours" isn't useful.

> Saying "check back in 2 hours" isn't useful. Having worked for a cloud provider, the reason they are saying that is because they are actively working to understand and fix the problem but haven't come to a well resound solution and thus they cannot give you a decent time estimate because you will probably get even more mad if they under/over estimate the time it took to fix it.

If they said this, "because they are actively working to understand and fix the problem but haven't come to a well resound solution and thus they cannot give you a decent time estimate because you will probably get even more mad if they under/over estimate the time it took to fix it." I would thank and applaud them. Tell me what it is you're doing at least. Why don't you understand the problem? What are you investigating? Some transparency goes a long way for me.

Re: Microsoft Azure suffers outage after cooling issue

#53

Post by dmarlow »

partiallypro wrote: post #50
mrep wrote: post #48

Earlier quoted context omitted.

> Saying "check back in 2 hours" isn't useful. Having worked for a cloud provider, the reason they are saying that is because they are actively working to understand and fix the problem but haven't come to a well resound solution and thus they cannot give you a decent time estimate because you will probably get even more mad if they under/over estimate the time it took to fix it.

Exactly, this happens even in just a normal production failure. I don't know what else they could have said/communicated. Not to mention this is the 7th largest data center in the world, resolving the problem likely took/is taking a long long time just because there are so many machines. I was lucky that the only outage effect I've suffered from is that my storage is locked, which means I can't add new file/edit code…

Were you able to perform operations given ARM services were down? Were you using your secondary DB for read-only or did you fail-over some other way?

Re: Microsoft Azure suffers outage after cooling issue

#54

Post by fphhotchips »

i_have_to_speak wrote: post #51
Theodores wrote: post #20

Earlier quoted context omitted.

That is a top 'did you know' factoid that I am sure I will tell others. But do AT+T engineers carry T-Mobile phones? If yes then they should put themselves together a deal so that none of the on-call engineers have to worry about running up big bills using their phones. When there are freak weather events they are all in it together.

"factoid": I don't think it means what you think it means. https://www.merriam-webster.com/dictionary/factoid

The second definition there fits perfectly.

Re: Microsoft Azure suffers outage after cooling issue

#55

Post by AaronFriel »

bitmapbrother wrote: post #33
Someone1234 wrote: post #12

Earlier quoted context omitted.

AWS and Azure have had "big" outages people because actually use them. Rackspace and IBM are almost neck and neck with Google's best efforts (3% markshare Vs. 30%/40% for Azure/AWS)[0]. [0] https://www.skyhighnetworks.com/cloud-security-blog/microsof...

The biggest and most popular services run on Google Cloud. https://cloud.google.com/customers/ You know....services consumers actually use.

I'm sure services I use run on all of the major cloud providers, but if that page was supposed to be enlightening, I only recognized one brand from the first page of customers.

I refreshed a couple times, and sure, I saw more (on average 1 or 2) that I recognized on each page. But I don't think your response is particularly persuasive. Are you suggesting that the services that I use that run on AWS are in fact, not services I actually use?

Or am I not a consumer? I'm confused.

Edit: Do you hold any Alphabet/Google stock? I've noticed your comment history trends toward dismissing criticism of Google, praising their products, and taking opportunities to speak about the flaws of their top competitors.

Re: Microsoft Azure suffers outage after cooling issue

#57

Post by toomuchtodo »

okcwarrior wrote: post #21
Someone1234 wrote: post #9

Visual Studio Online has been offline all day. They say it is due to the same Azure outage. This has had a productivity impact. If Microsoft didn't own GitHub, this may have prompted a move, but since they do it seems a little redundant given that Github will likely be on Azure too before long. https://blogs.msdn.microsoft.com/vsoservice/?p=17405

Having VSTS down all day meant I got exactly 0 done today. Completely crazy to me.

Not intended to be snarky: why is this crazy to you? All cloud providers have had downtime incidents, major hosted VCS providers, SaaS products. Downtime is a fact of life in tech.

Re: Microsoft Azure suffers outage after cooling issue

#58

Post by asdfasgasdgasdg »

toomuchtodo wrote: post #57
okcwarrior wrote: post #21

Earlier quoted context omitted.

Having VSTS down all day meant I got exactly 0 done today. Completely crazy to me.

Not intended to be snarky: why is this crazy to you? All cloud providers have had downtime incidents, major hosted VCS providers, SaaS products. Downtime is a fact of life in tech.

Hrm. I've worked at a large tech firm for more than a decade and there has never been a full day where VCS or the build farm were down all day. It's notable when it's down for more than twenty minutes.

Re: Microsoft Azure suffers outage after cooling issue

#59

Post by asdfasgasdgasdg »

ohthehugemanate wrote: post #23
king_magic wrote: post #17

Edit: out my rant. It's been a long day because of this. Just going to leave it at that.

They have some services that are "global", ie not tied to a given region. Those services' requests are actually processed all over the place, but south central is a big datacenter. The 9th biggest in the world, apparently. When it lost cooling and shut down, everything routed around it as planned... But it caused so much extra traffic that it overwhelmed the connections to other datacenters. The backlog of requests i…

> Honestly it's hard to imagine a good mitigation for this.

Besides the one that AWS and GCP have implemented? That is, to have at least N+1 datacenters? Actually, I think N+1 is the old Google prod regime. I suspect that GCP is at least N+1 per continental region, and I'd be surprised if AWS isn't as well.

Re: Microsoft Azure suffers outage after cooling issue

#60

Post by msandford »

cloakandswagger wrote: post #27
eric_b wrote: post #25

Earlier quoted context omitted.

> "Keep enough spare capacity around to handle losing one of the biggest datacenters in the world" is pretty unreasonable. Err what? It's entirely reasonable to expect Azure to handle the loss of a single DC and not have a 14+ hour global outage. I don't care how big the DC is, losing one should not take out the world, especially not for the length of time this one has been going on.

Indeed. This article by AWS VP James Hamilton gives a unique insight into how Amazon approaches the problem of sizing data centers for redundancy: https://perspectives.mvdirona.com/2017/04/how-many-data-cent...

Here I was hoping this was a reference to James Mickens: https://blogs.microsoft.com/ai/james-mickens-the-funniest-ma...
Post reply on HN