Microsoft had three staff at Australian data centre campus when Azure went out
1–10 of 52 posts
Re: Microsoft had three staff at Australian data centre campus when Azure went out
#2Guessing affected customers had to spend time and effort on top of ongoing high cloud bills
I've slept so much better since I began hosting, producing energy, and cooling on-prem
Re: Microsoft had three staff at Australian data centre campus when Azure went out
#3I know very little on the datacenter operations side of things - I guess 3 people is not a lot, but what is normal? How many operations people are at say AWS US-East-1? I presume it doesn't scale with number of servers, that would not scale well. What is a 'normal' level? 10? 100? It can't be more than 100, can it?
Re: Microsoft had three staff at Australian data centre campus when Azure went out
#4Guessing affected customers had to spend time and effort on top of ongoing high cloud bills I've slept so much better since I began hosting, producing energy, and cooling on-prem
The secret is hosting across failure boundaries so that a single outage like this does not impact you. Self-hosting is fine if you can afford the capex for two physically separate data centers (like really separate - like 100+ miles etc (or more!) to cope with natural disasters) and the staff to operate & maintain them 24/7. For many, this is not realistic.
For those that do need to use cloud, just make sure you are running your services in different failure zones.
Re: Microsoft had three staff at Australian data centre campus when Azure went out
#5Guessing affected customers had to spend time and effort on top of ongoing high cloud bills I've slept so much better since I began hosting, producing energy, and cooling on-prem
What about data center colocation? When you simply rent the energy, cooling, etc, but the hardware is yours? Do you think it's a nice middle ground?
Re: Microsoft had three staff at Australian data centre campus when Azure went out
#6I know very little on the datacenter operations side of things - I guess 3 people is not a lot, but what is normal? How many operations people are at say AWS US-East-1? I presume it doesn't scale with number of servers, that would not scale well. What is a 'normal' level? 10? 100? It can't be more than 100, can it?
I doubt any amount of staffing could address lack of specialism in dealing with power or air conditioning issues, both likely involve infrequent maintenance by external vendors. 20 people blowing up the phone to a vendor doesn't fix a problem any faster than 1 etc.
Re: Microsoft had three staff at Australian data centre campus when Azure went out
#7I know very little on the datacenter operations side of things - I guess 3 people is not a lot, but what is normal? How many operations people are at say AWS US-East-1? I presume it doesn't scale with number of servers, that would not scale well. What is a 'normal' level? 10? 100? It can't be more than 100, can it?
I've heard of the big-cos having to use bonafide robots for doing manual tasks in a data center like replacing broken drives or swapping tapes etc. I think there is still a bunch of manual tasks to be done.
That said I have no idea. When I worked (many many many many years ago) in a small DC that is perhaps the size of a 2bed apartment we had 4 guys scurrying about doing stuff (hands-on-keyboard, routing cables, replacing hardware etc). This was way before Docker & Kubernetes et al - physical iron and all that. I would assume that in modern DC ops you could run a football field sized DC with less than 10 people due to automation. But that said if part of the actual infrastructure like power or cooling fails, you need to have the right skill-set in place. If the cooler's failed and couldn't just be turned off and on again, we would have been out of luck in my old DC days and would need to call someone in and just hope the servers didn't fry in the meantime. Sounds like a similar deal here.
Re: Microsoft had three staff at Australian data centre campus when Azure went out
#8I know very little on the datacenter operations side of things - I guess 3 people is not a lot, but what is normal? How many operations people are at say AWS US-East-1? I presume it doesn't scale with number of servers, that would not scale well. What is a 'normal' level? 10? 100? It can't be more than 100, can it?
0 - 2 staff in a typical DC is not unusual at all, with people who are on-call usually within a 30 minute drive.
Larger DCs can and do have more staff on-site 24/7 and typically the amount of staff on-site at any given time is driven by SLAs.
I expect the DC in TFA to return to lower staff levels once they've worked on reducing their total "time to restart chiller" or reduced the amount of manual work involved in doing so.
Re: Microsoft had three staff at Australian data centre campus when Azure went out
#9Guessing affected customers had to spend time and effort on top of ongoing high cloud bills I've slept so much better since I began hosting, producing energy, and cooling on-prem
The secret is hosting across failure boundaries so that a single outage like this does not impact you. Self-hosting is fine if you can afford the capex for two physically separate data centers (like really separate - like 100+ miles etc (or more!) to cope with natural disasters) and the staff to operate & maintain them 24/7. For many, this is not realistic. For those that do need to use cloud, just make sure you are…
>like 100+ miles etc (or more!) to cope with natural disasters)
People talk about this often but this failure mode seems to never happen? When was the last time us-east-1 went down because of a natural calamity compared to some technical issue?
Re: Microsoft had three staff at Australian data centre campus when Azure went out
#10Guessing affected customers had to spend time and effort on top of ongoing high cloud bills I've slept so much better since I began hosting, producing energy, and cooling on-prem
The secret is hosting across failure boundaries so that a single outage like this does not impact you. Self-hosting is fine if you can afford the capex for two physically separate data centers (like really separate - like 100+ miles etc (or more!) to cope with natural disasters) and the staff to operate & maintain them 24/7. For many, this is not realistic. For those that do need to use cloud, just make sure you are…
> For those that do need to use cloud, just make sure you are running your services in different failure zones.
By which time you might as well just roll out your own kit in colocation or your own datacentres.
The cloud providers are nickle and dimers, they charge you for every little tiny thing.
Cloud might look cheap at cents-per-hour, but then you find you need X "services" to deliver your Service and so you are talking about exponential cents-per-hour (X cloud services times x cents-per-hour).
And then running your services across failure zones will of course cost you more beyond the basic double-cost, because most cloud providers charge by the GB for cross-zone traffic. So if you're doing cross-zone replication, that's gonna cost you a pretty penny.
Meanwhile, in your own colo/DC, you have predictable costs. And you can get redundant connections between sites for a flat rate, not some stupid per GB fee.