Asset management is definitely a thing. Tag your environments, tag your apps, and provide your apps criticality ratings based on how important they are to running the business. Then it's a matter of a query to know which servers can be shut, and which absolutely must remain.
We might want to regularly keep track of how important each server is
11–20 of 147 posts
Re: We might want to regularly keep track of how important each server is
#12It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
When you are responsible for the full infrastructure, sequencing power down and power on in coordination with your UPS is a common solution. Network gear needs a few minutes to light up ports, core services like DNS and identity services might need to light up next, then storage, then hypervisors and container hosts, then you can actually start working on app dependencies.
This sort of sequencing leads itself naturally to having a plan for limited capacity “keep the lights on” workload shedding when facing a situation like the OP.
Not everyone has elected to pay Bezos double the price for things they can handle themselves, and this is part of handling it.
Re: We might want to regularly keep track of how important each server is
#13It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
Even if you're running "cattle", you still need to keep track of which systems are important, because to surprise of many, the full infrastructure is more like the ranch, and cattle is just part of it.
(and here I remind myself again to write the screed against "cattle" metaphor...)
Re: We might want to regularly keep track of how important each server is
#14It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
HPC admin here (and possibly managing a similar system topology with their room).
In heterogeneous system rooms, you can't stuff everything into a virtualization cluster with a shared storage and migrate things on the fly, thinking that every server (in hardware) is a cattle and you can just herd your VMs from host to host.
A SLURM cluster is easy. Shutdown all the nodes, controller will say "welp, no servers to run the workloads, will wait until servers come back", but storage systems are not that easy (ordering, controller dependencies, volume dependencies, service dependencies, etc.).
Also there are servers which can't be virtualized because they're hardware dependent, latency dependent, or just filling the server they are in, resource wise.
We also have some pet servers, and some cattle. We "pfft" to some servers and scramble for others due to various reasons. We know what server runs which service by the hostname, and never install pet servers without team's knowledge. So if something important goes down everyone at least can attend the OS or the hardware it's running on.
Even in a cloud environment, you can't move a VSwitch VM as you want, because you can't have the root of a fat SDN tree on every node. Even the most flexible infrastructure has firm parts to support that flexibility. It's impossible otherwise.
Lastly, not knowing which servers are important is a big no-no. We had "glycol everywhere" incidents and serious heatwaves, and all we say is, "we can't cool room down, scale down". Everybody shuts the servers they know they can, even if somebody from the team is on vacation.
Being a sysadmin is a team game.
Re: We might want to regularly keep track of how important each server is
#15It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
At a FAANG, our services are cattle, but we still plan which services to keep running when we need to drain 50% of a DC.
Latency is important. Money makers > Latency sensitive > Optional requests > Background requests > Batch traffic.
Bootstrapping is important. If A depends on B, you might to drain A first, or A and B together.
Re: We might want to regularly keep track of how important each server is
#16It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
When you are responsible for the full infrastructure, sequencing power down and power on in coordination with your UPS is a common solution. Network gear needs a few minutes to light up ports, core services like DNS and identity services might need to light up next, then storage, then hypervisors and container hosts, then you can actually start working on app dependencies. This sort of sequencing leads itself natural…
Double? Try 100x!!
Re: We might want to regularly keep track of how important each server is
#17It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
> Industry: don’t treat your systems like pets.
The industry has this narrative because it suits their desire to sell higher-margined cloud services. However in the real world, especially in academia as cks is, the reality is that many workloads are still not suitable for the cloud.
Re: We might want to regularly keep track of how important each server is
#18It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
I’m absolutely loving the term Unix herder and will probably adopt it :)
I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here:
- budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the hardware they’ve bought you. They won’t be as happy to dramatically increase your opex.
- storage is the ultimate pet. In a university IT department you’re going to have people who need access to tons and tons of speedy storage both short-term and long-term.
I’m smiling a little bit thinking about a job 10 years ago who adopted the cattle-not-pets mentality. The IT department decided they were done with their pets, moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array. There was a disk failure, but that’s ok, RAID-5 can handle that. And then the next day there was a second disk failure. Boom. Every single VM in the engineering department is gone including all of the data. It was all backed up to tape and slowly got restored but the blast radius was enormous.
Re: We might want to regularly keep track of how important each server is
#19It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
I think you're missing some aspects of cattle. You're still supposed to keep track of what happens and where. You still want to understand why and how each of the servers in the autoscaling group (or similar) behaves. The cattle part just means they're unified and quickly replaceable. Buy they still need to be well tagged, accounted for in planning, removed when they don't fulfil the purpose anymore, identified for billing, etc.
And also importantly: you want to make sure you have a good enough description for them that you can say "terraform/cloudformation/ansible: make sure those are running" - without having to find them on the list and do it manually.
Re: We might want to regularly keep track of how important each server is
#20Or you might want to have redundant cooling. Cooling system prices seem to scale fairly linearly with the cooling power above a few kW, so instead of one 100 kW system you could buy four 25 kW systems so a single failure won't be a disaster.
With that model, you’d probably want 5 instead of 4 (N+1), but the other thing to consider is if you can duct the cold air to where it needs to go when one or more of the units has failed.