Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft had three staff at Australian data centre campus when Azure went out

itnews.com.au

Post reply on HN
Print view View original

31–40 of 52 posts

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#31

Post by haolez »

traceroute66 wrote: post #12
haolez wrote: post #5

Earlier quoted context omitted.

What about data center colocation? When you simply rent the energy, cooling, etc, but the hardware is yours? Do you think it's a nice middle ground?

> Do you think it's a nice middle ground? It is. The cloud fanbois will tell you until their blue in the face that its not. I fully accept that the cloud is great for bursty workloads where you're doing nothing and then suddenly half the planet needs your service for a couple of days. That is clear. But if you've got a reasonably stable baseload running 24x7x365 and a few modest bursts here and there then honestly pe…

I see. But the cloud is much more than just VMs. I don't know if I'd want to manage an equivalent to SQS on my own. Maybe I should try it out and see what happens :)

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#32

Post by jquast »

gruez wrote: post #24
mattlondon wrote: post #13

Earlier quoted context omitted.

Not sure about us-east-1 specifically but there are frequently fairly large natural disasters in the US - there are always hurricanes and stuff, there was that flooding in new York not so long ago, earthquakes in California in the 90s, wildfires etc. And this is just in the US. Basically, don't put all your servers in NYC or all in SF or whatever, but put half in NYC and half in SF and that random hurricane/wildfire/…

>but there are frequently fairly large natural disasters in the US - there are always hurricanes and stuff, there was that flooding in new York not so long ago, earthquakes in California in the 90s, wildfires etc. Right but what type of datacenter related incidents did they cause? Did us-east-1 go down because of hurricane sandy? Did us-west-1 go down because of wildfires? I don't seem to remember any datacenter outa…

I remember Hurricane Katrina shutting down lots of online services, and directnic battling to stay online https://www.datacenterknowledge.com/archives/2007/11/05/prov...

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#33

Post by TheNewsIsHere »

gruez wrote: post #24
mattlondon wrote: post #13

Earlier quoted context omitted.

Not sure about us-east-1 specifically but there are frequently fairly large natural disasters in the US - there are always hurricanes and stuff, there was that flooding in new York not so long ago, earthquakes in California in the 90s, wildfires etc. And this is just in the US. Basically, don't put all your servers in NYC or all in SF or whatever, but put half in NYC and half in SF and that random hurricane/wildfire/…

>but there are frequently fairly large natural disasters in the US - there are always hurricanes and stuff, there was that flooding in new York not so long ago, earthquakes in California in the 90s, wildfires etc. Right but what type of datacenter related incidents did they cause? Did us-east-1 go down because of hurricane sandy? Did us-west-1 go down because of wildfires? I don't seem to remember any datacenter outa…

> Did us-east-1 go Dow because of hurricane sandy?

Nope, but Sandy did a hell of a lot of damage to some key telecommunications infrastructure. Verizon lost multiple floors worth of equipment, cabling, and related infrastructure that served at least their customers across Manhattan.

Having geographical redundancy for mission critical workloads is a good investment if your business is making money. Networked computing is one of the few places we can actually “run away” from a physical source of problems. (Not forever, or universally, of course).

We’re based on the eastern seaboard. You bet we have failsafes in areas less susceptible to natural disaster.

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#35

Post by toast0 »

mtmail wrote: post #18
22c wrote: post #8

Earlier quoted context omitted.

0 - 2 staff in a typical DC is not unusual at all, with people who are on-call usually within a 30 minute drive. Larger DCs can and do have more staff on-site 24/7 and typically the amount of staff on-site at any given time is driven by SLAs. I expect the DC in TFA to return to lower staff levels once they've worked on reducing their total "time to restart chiller" or reduced the amount of manual work involved in doi…

Still we read how DCs are a job generator. E.g. "One hopes for hundreds of jobs for locals." https://cryptoquorum.com/oman-opens-cryptocurrency-mining-ce...

Yeah, that's counting construction jobs like the sibling said, or just unfounded optimism.

A datacenter with tight SLAs probably needs one 24/7 tech and one 24/7 security guard. That's acheivable with 5 jobs for each, so 10 jobs total. Maybe a couple more techs if the ticket volume is high. Never going to be hundreds of permanent jobs.

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#36

Post by toast0 »

gruez wrote: post #24
mattlondon wrote: post #13

Earlier quoted context omitted.

Not sure about us-east-1 specifically but there are frequently fairly large natural disasters in the US - there are always hurricanes and stuff, there was that flooding in new York not so long ago, earthquakes in California in the 90s, wildfires etc. And this is just in the US. Basically, don't put all your servers in NYC or all in SF or whatever, but put half in NYC and half in SF and that random hurricane/wildfire/…

>but there are frequently fairly large natural disasters in the US - there are always hurricanes and stuff, there was that flooding in new York not so long ago, earthquakes in California in the 90s, wildfires etc. Right but what type of datacenter related incidents did they cause? Did us-east-1 go down because of hurricane sandy? Did us-west-1 go down because of wildfires? I don't seem to remember any datacenter outa…

> Did us-east-1 go down because of hurricane sandy?

No, but I was at a company with all the production services in Reston, VA during that storm, and we would have been pretty screwed if Sandy made landfall in the DC area instead of continuing north.

Sandy's flooding in NYC wasn't great for some of the datacenters there, I seem to recall some having trouble, but most were fine.

BGP and DNS are certainly much better at causing disruption, and especially global disruption though.

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#37

Post by adambatkin »

benterix wrote: post #26
dijit wrote: post #19

I guess people really have forgotten how to run datacenters. The only people who should be shocked in this thread are the people who have been hoodwinked into thinking operations is so hard you need thousands of staff. I know AWS/GCP/Azure like to charge us as if we were hiring an army of sysadmins, but the truth is that day-to-day DC ops does not require so many people. Hardware failures are more rare than you think…

The management's way of thinking is: "Well, let's just pay for the peace of mind." Except that this famous peace of mind never comes, because the cloud gets more and more complex each year and it's hard to keep up. Heck, even Amazon can't keep up: for example, officially they depreciate bucket policies but internally they are using it for example in the Cloud Formation templates for the Control Tower. But now it's to…

I don't believe that S3 Bucket Policies are deprecated. They are powerful, effective, and consistent with almost everything else at AWS (Resource Policy). Perhaps you are thinking of ACLs?

Re: Microsoft had three staff at Australian data centre campus when Azure went out

#38

Post by dalyons »

dagw wrote: post #17
traceroute66 wrote: post #12

Earlier quoted context omitted.

> Do you think it's a nice middle ground? It is. The cloud fanbois will tell you until their blue in the face that its not. I fully accept that the cloud is great for bursty workloads where you're doing nothing and then suddenly half the planet needs your service for a couple of days. That is clear. But if you've got a reasonably stable baseload running 24x7x365 and a few modest bursts here and there then honestly pe…

I believe the real secret reason the cloud is so popular among developers (based on 10+ years of experience) is that cloud providers are so much nicer and faster to deal with than your company IT department. Also on the price side, I'm not comparing the price of cloud vs colo, but the price of cloud vs what the company IT department charges my department for being allowed to use one of 'their' colo servers, and that…

Also the vast array of managed services. Managed databases, message queues, infinite storage, data warehouses, caches , etc etc. Many of which are very complicated to host well yourself and operationize (failovers, monitoring , backups etc)

This idea that you can build a DC that competes on cost for rented cloud compute - it might be technically true but it’s mostly missing the point of why modern shops prefer the cloud.

Post reply on HN