[flagged]
I just did a due diligence with a company that only had servers in one data center. They were supremely confident that there was no way a whole DC could be impacted by an event.
We might want to regularly keep track of how important each server is
41–50 of 147 posts
Re: We might want to regularly keep track of how important each server is
#42Asset management is definitely a thing. Tag your environments, tag your apps, and provide your apps criticality ratings based on how important they are to running the business. Then it's a matter of a query to know which servers can be shut, and which absolutely must remain.
Having an application, process and hardware inventory is a must if you are going to have any hope of disaster recovery. Along with regular failovers to make sure you haven’t missed anything.
Re: We might want to regularly keep track of how important each server is
#43Earlier quoted context omitted.
At a FAANG, our services are cattle, but we still plan which services to keep running when we need to drain 50% of a DC. Latency is important. Money makers > Latency sensitive > Optional requests > Background requests > Batch traffic. Bootstrapping is important. If A depends on B, you might to drain A first, or A and B together.
The cattle metaphors really is a bad one. Anyone raising cattle should do the same thing, knowing which animals are the priority in case of draught, disease, etc. Hopefully one never has to face that scenario, but its much easier to pick up the pieces when you know where the priorities are whether you're having to power down servers or thin a herd.
Cattle are often interchangeable. You cull any that catch a disease (in some cases the USDA will cull the entire herd if just one catches something - bio security is a big deal) In the case of drought you pick a bunch to get rid of - based on market prices (If everyone else is you will try to keep yours because the market is collapsing - but this means managing feed and thus may mean culling more of the herd latter.)
Some cattle we can measure. Milk cows are carefully managed as to output - the farmer knows how much the milk each one is worth and so they can cull the low producers. However milk is so much more valuable than meat that they never cull based on drought - milk can always outbid meat for feed. If milk demand goes down the farmer might cull come - but often the farmer is under contract for X amount of milk and so they cannot manage prices.
Re: We might want to regularly keep track of how important each server is
#44Earlier quoted context omitted.
> Or you might want to have redundant cooling. Can you provide a cost centre or credit card for which they can bill this to? In case you didn't notice the domain, it is UToronto: academic departments aren't generally flush with cash. Further, you have to have physical space to fit the extra cooling equipment and pipes: not always easy or possible to do in old university buildings.
If you designed the system like this from the start or when replacing it anyways, N+1 redundancy might not me much more expensive than one big cooling unit. The systems can mostly share their ductwork and just have redundancy in the active components, so mostly the chillers. Of course these systems only get replaced every couple decades, if ever, so they are pretty much stuck with the setup they have.
> If you designed […]
University department IT is not designed, it grows over decades.
At some point some benefactor may pay for a new building and the department will move, so that could be a chance to actually design. But modern architecture and architects don't really go well with hosting lots of servers in what is ostensibly office space.
I've been involved in the build-out of buildings/office space on three occasions in my career, and trying to get a decent IT space pencilled has always been like pulling teeth.
Re: We might want to regularly keep track of how important each server is
#45It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
I'm pretty sure anyone in the industry that draws this distinction between cattle and pets has never worked with cattle and only knows of general ideas about the industrial cattle business.
Ranchers do eat their pets. They generally do love the cattle, but they also know at the end of a few years they get replaced - it is the cycle of life.
Re: We might want to regularly keep track of how important each server is
#46It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
Pets make sense sometimes. I also think there are still plenty of companies, large ones, with important services and data, that just don't operate in a way that allows the data center teams to do this either. I have some experience with both health insurance and life insurance companies for example where "this critical thing #8 that we would go out of business without" stillnloves solely on "this server right here". In university settings you have systems that are owned by a wide array of teams. These organizations aren't ready or even looking to implement a platform model where the underlying hardware can be generic.
Re: We might want to regularly keep track of how important each server is
#47Tracking servers is one thing, but tracking the dependency relationships among them is likely at least as important.
Re: We might want to regularly keep track of how important each server is
#48Re: We might want to regularly keep track of how important each server is
#49It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the…
The devops meme referenced, "cattle not pets", probably popularized by a book called "The Phoenix Project".
The real point is that pets are job security for the Unix herder. If you end up with one neckbeard running the joint that's even worse as a single point of failure.
Re: We might want to regularly keep track of how important each server is
#50Earlier quoted context omitted.
> Or you might want to have redundant cooling. Can you provide a cost centre or credit card for which they can bill this to? In case you didn't notice the domain, it is UToronto: academic departments aren't generally flush with cash. Further, you have to have physical space to fit the extra cooling equipment and pipes: not always easy or possible to do in old university buildings.
If you designed the system like this from the start or when replacing it anyways, N+1 redundancy might not me much more expensive than one big cooling unit. The systems can mostly share their ductwork and just have redundancy in the active components, so mostly the chillers. Of course these systems only get replaced every couple decades, if ever, so they are pretty much stuck with the setup they have.
> Of course these systems only get replaced every couple decades, if ever
This is despite the massive energy savings they could get if they replaced those older systems. Universities often are full of old buildings with terrible insulation heated/cooled by very old/inefficient systems. In 20 years they would be money ahead by tearing down most buildings on campus and rebuilding to modern standards - assuming energy costs don't go up which seems unlikely) But they consider all those old buildings historic and so won't.