Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft Azure suffers outage after cooling issue

datacenterdynamics.com

Post reply on HN
Print view View original

81–90 of 113 posts

Re: Microsoft Azure suffers outage after cooling issue

#81

Post by dustinmoris »

toomuchtodo wrote: post #63
asdfasgasdgasdg wrote: post #58

Earlier quoted context omitted.

Hrm. I've worked at a large tech firm for more than a decade and there has never been a full day where VCS or the build farm were down all day. It's notable when it's down for more than twenty minutes.

As counterpoint, I’ve seen banks down for extended periods of time, hours occasionally stretching to a day or two (TSB, LLoyds, Bank Of America, BankSimple [BBVA]). Downtime is a fact of life. Google, Amazon, IBM, and Microsoft have had major cloud outages. GitLab nuked their production DB. Slack and Reddit are frequently down. Unless it’s life critical (911, air traffic control), if it’s down its only going to hampe…

If a single bank is down or Reddit (lol) then the impact is fairly limited, but if one of the 3 major cloud providers, which powers large parts of the internet is down for an entire day, then the impact is a little bit more critical I would say ;).

There's a reason why Azure has a SLA and Reddit doesn't ;)

Also if you start comparing the big companies with GitLab then we don't have to continue talking anymore. It's not ok to nuke your production db and that's why everyone in the tech scene laughs about GitLab and comparing them to Azure is like comparing a lego house to brick and mortar.

Re: Microsoft Azure suffers outage after cooling issue

#82

Post by amaccuish »

TeMPOraL wrote: post #75
whoisjuan wrote: post #24

Earlier quoted context omitted.

Just today I was having issues with the Prettier extension in VS Code, and I uninstalled it to see if that would fix it (I read that usually fixes the issues I was having). Then I realized that I couldn't install it again because VS Marketplace was down. This was like 8 hours ago and still no signs of recovery. Of course, all my builds are failing because of some stupid formatting issue that Prettier usually would so…

> Of course, all my builds are failing because of some stupid formatting issue that Prettier usually would solve, so yeah..thanks MSFT. Is failing builds due to formatting issues really a sound setup?

I assume their build system checks for formatting and will raise an error if it doesn't conform. And this person would use the VS Code extension to auto-conform their code.

Re: Microsoft Azure suffers outage after cooling issue

#83

Post by cm2187 »

ohthehugemanate wrote: post #23
king_magic wrote: post #17

Edit: out my rant. It's been a long day because of this. Just going to leave it at that.

They have some services that are "global", ie not tied to a given region. Those services' requests are actually processed all over the place, but south central is a big datacenter. The 9th biggest in the world, apparently. When it lost cooling and shut down, everything routed around it as planned... But it caused so much extra traffic that it overwhelmed the connections to other datacenters. The backlog of requests i…

But you are basically saying that azure cannot deliver the redundancy they charge their customers for?

Re: Microsoft Azure suffers outage after cooling issue

#84

Post by ti_ranger »

mrep wrote: post #28
lozaning wrote: post #13

Earlier quoted context omitted.

And Amazon. I recall that during their most recent s3 outage Amazon's status page was green across the board, because somehow all the assets that were supposed to be displayed when things went wrong were themselves hosted on the thing that was down.

Yes, changing the status page required updating a S3 bucket that was located in us-east-1 which is where it went down so they couldn't change it.

And now, S3 has cross-region replication ...

And so does DynamoDB, and Aurora ...

Re: Microsoft Azure suffers outage after cooling issue

#85

Post by filomeno »

We all ditched the Unix model of a central server with dumb terminals around because Microsoft told us the future was everybody having a full OS in their workstation. Now they tell us the future is going back 30 years, and having all of our data and programs in somebody else's machines (theirs).

Sometimes I can't understand people.

Re: Microsoft Azure suffers outage after cooling issue

#86

Post by ti_ranger »

karmicthreat wrote: post #5

So AWS has had some big outages, as has Azure. Has GCP had any big outages yet?

The biggest AWS outage in the past 3 years was S3 being down in one region for 2 hours in Feb 2018 ...

No other regions were affected, except for global APIs (e.g. create S3 bucket), which one shouldn't rely on on your critical path.

Many new customer features have been delivered to allow mitigation of this kind of failure (e.g. cross-region S3 replication).

AWS had a power failure at one data-center in us-east-1 earlier this year, which had very little impact (basically only customers who didn't have sufficient redundancy in other AZs were affected).

Re: Microsoft Azure suffers outage after cooling issue

#87

Post by filomeno »

okcwarrior wrote: post #21
Someone1234 wrote: post #9

Visual Studio Online has been offline all day. They say it is due to the same Azure outage. This has had a productivity impact. If Microsoft didn't own GitHub, this may have prompted a move, but since they do it seems a little redundant given that Github will likely be on Azure too before long. https://blogs.msdn.microsoft.com/vsoservice/?p=17405

Having VSTS down all day meant I got exactly 0 done today. Completely crazy to me.

For me, it's even crazier to rely on a third party provider to be able to do my work. I wouldn't be comfortable at all knowing that my development infrastructure is beyond my control.

Re: Microsoft Azure suffers outage after cooling issue

#88

Post by TeMPOraL »

amaccuish wrote: post #82
TeMPOraL wrote: post #75

Earlier quoted context omitted.

> Of course, all my builds are failing because of some stupid formatting issue that Prettier usually would solve, so yeah..thanks MSFT. Is failing builds due to formatting issues really a sound setup?

I assume their build system checks for formatting and will raise an error if it doesn't conform. And this person would use the VS Code extension to auto-conform their code.

I'm questioning the soundness of such setup in general, and especially if it means that losing connection to a third-party prettifier makes you unable to work on your own codebase.

Re: Microsoft Azure suffers outage after cooling issue

#89

Post by robkop »

partiallypro wrote: post #67
asdfasgasdgasdg wrote: post #61

Earlier quoted context omitted.

Google is fairly near MS (Azure did $1.9B in Q1, and GCP+Google Apps did $1.7B). But AWS is ~2.5x the other two.

Google Apps is not GCS. If you wanted to compare that number you'd have to throw in Office 365/Azure into the same number and it would dwarf Google. Google has ~3% market share compared to Microsoft's ~28% and Amazon's ~40%. Not even in the same league at the moment. Google is more on par with IBM and Rackspace, for now. Google will undoubtedly make strides in the space, but they haven't been tested.

Depends who's figures you go by. As per [1] they're at 6%, double your source.

[1] https://www.google.com.au/amp/s/www.cnbc.com/amp/2018/04/27/...

Re: Microsoft Azure suffers outage after cooling issue

#90

Post by ti_ranger »

karmicthreat wrote: post #5

So AWS has had some big outages, as has Azure. Has GCP had any big outages yet?

GCP has had multiple many-hour (6+) GLOBAL outages in the past year. I think it's at about 3 so far this year.

But, it doesn't make the headlines like a 2-hour S3 outage in a single region, which must mean something ...

Post reply on HN