Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft had three staff at Australian data centre campus when Azure went out

itnews.com.au

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#21

Post by [deleted] »

gruez wrote: post #9
mattlondon wrote: post #4

Earlier quoted context omitted.

The secret is hosting across failure boundaries so that a single outage like this does not impact you. Self-hosting is fine if you can afford the capex for two physically separate data centers (like really separate - like 100+ miles etc (or more!) to cope with natural disasters) and the staff to operate & maintain them 24/7. For many, this is not realistic. For those that do need to use cloud, just make sure you are…

>like 100+ miles etc (or more!) to cope with natural disasters) People talk about this often but this failure mode seems to never happen? When was the last time us-east-1 went down because of a natural calamity compared to some technical issue?

[deleted]

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#22

Post by hotpotamus »

dagw wrote: post #17
traceroute66 wrote: post #12

Earlier quoted context omitted.

> Do you think it's a nice middle ground? It is. The cloud fanbois will tell you until their blue in the face that its not. I fully accept that the cloud is great for bursty workloads where you're doing nothing and then suddenly half the planet needs your service for a couple of days. That is clear. But if you've got a reasonably stable baseload running 24x7x365 and a few modest bursts here and there then honestly pe…

I believe the real secret reason the cloud is so popular among developers (based on 10+ years of experience) is that cloud providers are so much nicer and faster to deal with than your company IT department. Also on the price side, I'm not comparing the price of cloud vs colo, but the price of cloud vs what the company IT department charges my department for being allowed to use one of 'their' colo servers, and that…

I'm the ops guy on a small dev team, and I run a sort of hybrid setup for prod that does involve me working on hardware in a colo sometimes, though fairly rarely (I'd love to spend about half my time hauling servers around and cabling stuff so that I'm not stuck at a desk all day, but that's not the way it is).

The whole point of my job is to enable developers to deliver code that provides customers value. On that level I actually embrace the common "condescensions" (so-to-speak) that I'm tech support for developers or a YAML wrangler.

I actually had an experience recently where a developer asked to make some changes to our infrastructure. I pretty much developed our container orchestration system (based on Docker Swarm rather than Kubernetes - a choice our architect made that I've come to appreciate), so I walked him through how my IaC works, told him what he needed to change and then reviewed his pull request and applied the changes. I guess we're on a devops journey now if I want to put it in corpo-tech speak.

Anyway, I suppose a lot of IT departments/guys get lost in creating their "perfect" unassailable systems and forget that the big picture is that the job is to enable customers; most directly are likely to be the developers or other internal employees, but ultimately the end customer who's handing you money to solve their problems.

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#23

Post by nixgeek »

dgrin91 wrote: post #3

I know very little on the datacenter operations side of things - I guess 3 people is not a lot, but what is normal? How many operations people are at say AWS US-East-1? I presume it doesn't scale with number of servers, that would not scale well. What is a 'normal' level? 10? 100? It can't be more than 100, can it?

us-east-1 is many, many datacenters across its Availability Zones and tens of billions of investment for Amazon Web Services.

Across all the datacenters the number of operations personnel likely exceeds 100. Think of the unit of scale as a datacenter, with an availability zone potentially containing 10+ of those.

[1] https://www.datacenterfrontier.com/cloud/article/11427911/aw...

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#24

Post by gruez »

mattlondon wrote: post #13
gruez wrote: post #9

Earlier quoted context omitted.

>like 100+ miles etc (or more!) to cope with natural disasters) People talk about this often but this failure mode seems to never happen? When was the last time us-east-1 went down because of a natural calamity compared to some technical issue?

Not sure about us-east-1 specifically but there are frequently fairly large natural disasters in the US - there are always hurricanes and stuff, there was that flooding in new York not so long ago, earthquakes in California in the 90s, wildfires etc. And this is just in the US. Basically, don't put all your servers in NYC or all in SF or whatever, but put half in NYC and half in SF and that random hurricane/wildfire/…

>but there are frequently fairly large natural disasters in the US - there are always hurricanes and stuff, there was that flooding in new York not so long ago, earthquakes in California in the 90s, wildfires etc.

Right but what type of datacenter related incidents did they cause? Did us-east-1 go down because of hurricane sandy? Did us-west-1 go down because of wildfires? I don't seem to remember any datacenter outages caused by wide area natural disasters, whereas I can remember plenty caused by BGP/DNS/config shenanigans.

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#25

Post by nixgeek »

dmw_ng wrote: post #6
dgrin91 wrote: post #3

I know very little on the datacenter operations side of things - I guess 3 people is not a lot, but what is normal? How many operations people are at say AWS US-East-1? I presume it doesn't scale with number of servers, that would not scale well. What is a 'normal' level? 10? 100? It can't be more than 100, can it?

I doubt any amount of staffing could address lack of specialism in dealing with power or air conditioning issues, both likely involve infrequent maintenance by external vendors. 20 people blowing up the phone to a vendor doesn't fix a problem any faster than 1 etc.

Amazon goes to the extreme of putting its own custom firmware on switchgear because the choices that vendor makes in theirs doesn’t align with their objectives.

I don’t think AWS is blowing up a vendors phone when something goes wrong in one of their facilities.

[1] https://www.datacenterknowledge.com/archives/2017/04/07/how-...

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#26

Post by benterix »

dijit wrote: post #19

I guess people really have forgotten how to run datacenters. The only people who should be shocked in this thread are the people who have been hoodwinked into thinking operations is so hard you need thousands of staff. I know AWS/GCP/Azure like to charge us as if we were hiring an army of sysadmins, but the truth is that day-to-day DC ops does not require so many people. Hardware failures are more rare than you think…

The management's way of thinking is: "Well, let's just pay for the peace of mind." Except that this famous peace of mind never comes, because the cloud gets more and more complex each year and it's hard to keep up. Heck, even Amazon can't keep up: for example, officially they depreciate bucket policies but internally they are using it for example in the Cloud Formation templates for the Control Tower. But now it's too late to go back as most of the internet is running on the three major public clouds. You need a lot of determination and a good plan to free oneself from vendor lock-in. In larger orgs it's practically impossible.

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#27

Post by Scaevolus »

nixgeek wrote: post #25
dmw_ng wrote: post #6

Earlier quoted context omitted.

I doubt any amount of staffing could address lack of specialism in dealing with power or air conditioning issues, both likely involve infrequent maintenance by external vendors. 20 people blowing up the phone to a vendor doesn't fix a problem any faster than 1 etc.

Amazon goes to the extreme of putting its own custom firmware on switchgear because the choices that vendor makes in theirs doesn’t align with their objectives. I don’t think AWS is blowing up a vendors phone when something goes wrong in one of their facilities. [1] https://www.datacenterknowledge.com/archives/2017/04/07/how-...

Amazon doesn't make their own AC units or generators, so it's still likely they would need external support for a case like this.

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#28

Post by xulres »

mattlondon wrote: post #7
dgrin91 wrote: post #3

I know very little on the datacenter operations side of things - I guess 3 people is not a lot, but what is normal? How many operations people are at say AWS US-East-1? I presume it doesn't scale with number of servers, that would not scale well. What is a 'normal' level? 10? 100? It can't be more than 100, can it?

I've heard of the big-cos having to use bonafide robots for doing manual tasks in a data center like replacing broken drives or swapping tapes etc. I think there is still a bunch of manual tasks to be done. That said I have no idea. When I worked (many many many many years ago) in a small DC that is perhaps the size of a 2bed apartment we had 4 guys scurrying about doing stuff (hands-on-keyboard, routing cables, repl…

Smaller operations usually need more manpower.

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#30

Post by siva7 »

dijit wrote: post #19

I guess people really have forgotten how to run datacenters. The only people who should be shocked in this thread are the people who have been hoodwinked into thinking operations is so hard you need thousands of staff. I know AWS/GCP/Azure like to charge us as if we were hiring an army of sysadmins, but the truth is that day-to-day DC ops does not require so many people. Hardware failures are more rare than you think…

You actually need thousands of people for operations at major cloud provider scale (and much more for development), but it scales at some point where you only need people for hands-on tasks at satellite plants and the rest sits at HQ.
Post reply on HN