Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft's Azure cloud down and out for 8 hours

theregister.co.uk

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Microsoft's Azure cloud down and out for 8 hours

#51

Post by krobertson »

andypants wrote: post #49
latchkey wrote: post #34

Heroku Dyno's were down for a lot of clients for several hours on saturday. One site that I use a lot, Intercom.io, was completely off the grid because of this. I was wondering why there was no news about it here. https://status.heroku.com/incident/308

I'm sure I read about the heroku downtime on hacker news. Maybe it got pushed off the front page pretty quickly though.

It was on the front page for a while, however it was the weekend, so likely didn't garner as much attention. And was like 2 hours... not 8-9+ like Azure.

Re: Microsoft's Azure cloud down and out for 8 hours

#52

Post by sriramk »

chubot wrote: post #36

So just "service management" is down, but the apps themselves are up? If so that's better than Heroku's recent downtime.

Disclaimer: ex-Windows Azure person, no insider knowledge on this particular outage

That's typically because the 'service management service' typically kicks in when it needs to do things - allocate capacity, restart things when stuff goes down, etc. By default, it isn't touching the running apps inside VMs. There is no Windows Azure equivalent to Heroku's routing mesh to be taken down; the requests go to the VMs directly via the various networking layers.

Re: Microsoft's Azure cloud down and out for 8 hours

#53

Post by jstclair »

krmmalik wrote: post #30

I wonder if this explains why i have been having problems with Siri? I know that sounds far fetched but isnt Apple relying on Azure now?

Don't know why you got downvoted - AFAIK Siri isn't tied to Azure (although it could be using it for storage), but iCloud runs across both Azure and Amazon.

Re: Microsoft's Azure cloud down and out for 8 hours

#54

Post by DrJokepu »

powertower wrote: post #46
ww520 wrote: post #38

Earlier quoted context omitted.

Can you elaborate what is the cert issue that causing the service down? It's always good to learn from temporary failure.

Today, February 29th, is a "leap day" which happens every 4 years in a "leap year"... Otherwise, the last day of February is the 28th, and today would normally be March 1st. I'm assuming that when the day rolled over, it caused an issue with an internal system at Microsoft, that had something to do with certificate dates and/or timestamps. It's not just Microsoft that it's happening to, today I could not automaticall…

How the hell is that possible? Every platform out there has high quality date libraries. It's a cliché that date & time operations are very complicated (leap years, leap seconds, time zones, daylight saving time and so on), every even marginally competent developer knows instinctly to stay away from writing their own date & time code and trust a library instead to do the heavy lifting. In addition, every tester should know that when dates are involved, they should try Feb 29th and see if it breaks anything. How can bugs like this slip through?

Re: Microsoft's Azure cloud down and out for 8 hours

#55

Post by ChuckMcM »

powertower wrote: post #19

This seems to be about some odd certificate issue, not the network, which caused Microsoft to take access to its service management system down. From the article itself: > It later added that less than 3.8 per cent of hosted services had been affected. If this was about Google or Apple, this submit would already have been flagged, taken off the front page, and several accounts would have been hell-banned. It will be…

"We have identified the root cause of this incident. It has been traced back to a cert issue triggered on 2/29/2012 GMT," the software giant said.

So here is a quiz for you, does you software know that this is a leap year? I'll speculate that someone's software didn't recognize it as such. Personally I'm always on the lookout for this sort of 'weird' thing because in the world of testing the edge cases are often poorly tested.

Re: Microsoft's Azure cloud down and out for 8 hours

#56

Post by TeHCrAzY »

ahrens wrote: post #44

We had problems today. We had the bad luck that one of our web roles crashed during the time the admin interface was down. That meant we couldn't restart it and neither could microsoft. We will be adding instances to the role to avoid similar problems. Otherwise, we are very happy with Azure.

Can you explain what your problem is? Azure will automatically restart instances that are dead afaik (unless you are saying that the automatic restarting was broken as well?).

Re: Microsoft's Azure cloud down and out for 8 hours

#57

Post by TeHCrAzY »

bry wrote: post #31

The most frustrating thing for me was the complete lack of any real communication from Microsoft. For awhile, even their status dashboard was down. I only found out about it after I got a PagerDuty alert and had to search Twitter (other people complaining about it) to confirm. We have an Azure CDN backed by a Compute Instance, and zero official notice from Microsoft about this still. I've learned more about the probl…

They finally pushed something out via Twitter, about an hour ago: http://twitter.com/#!/WindowsAzure/status/174954154362548224

Re: Microsoft's Azure cloud down and out for 8 hours

#58

Post by andrewem »

latchkey wrote: post #34

Heroku Dyno's were down for a lot of clients for several hours on saturday. One site that I use a lot, Intercom.io, was completely off the grid because of this. I was wondering why there was no news about it here. https://status.heroku.com/incident/308

It's unclear how big a deal this outage was, as I can never make heads or tails of what's really behind an article in the The Register.

That said, I run an app on Heroku and noticed that outage - Pingdom sent me a bunch of emails, which I saw after a day spent outside. The first thing I do when I get a Pingdom failure notice is check status.heroku.com, and sure enough they had an issue and were working on fixing it. I wasn't particularly expecting there to be mention of it on HN because, first, it wasn't an enormous outage, second, it was on the weekend, and third Heroku always does such a great job of posting about issues.

I don't know if they did so, but I'd expect Intercom.io and other affected sites to post information letting people know that the site is down and why. I haven't tried it yet but Heroku allows you to specify a custom error page to be served [1], which you could update on the fly to let people know what's going on. For other outages I have let users know via Twitter and Facebook, though obviously there's no substitute for a nice up-to-date static error page.

[1] http://devcenter.heroku.com/articles/custom_error_pages

Re: Microsoft's Azure cloud down and out for 8 hours

#59

Post by huggyface »

huggyface wrote: post #45
cooldeal wrote: post #41

Earlier quoted context omitted.

A whole bunch of universities and government agencies.

Such as? Microsoft's case studies for Azure are absolutely dismal. I note the other comment mentions Apple, which was a pre-release beta rumor based upon the IP that a beta iMessage lived at (since moved). It speaks volumes, I think, when such a disproven pre-release claim is still held as the example.

-1? For real? "A whole bunch" is not a credible statement of fact.
Post reply on HN