21–30 of 147 posts
Or you might want to have redundant cooling. Cooling system prices seem to scale fairly linearly with the cooling power above a few kW, so instead of one 100 kW system you could buy four 25 kW systems so a single failure won't be a disaster.
It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
They did have to load shed, though: they just turned off the AC inverters. They figured anything critical in a central office was on DC power, and if you had something on AC, you were just going to have to wait until the backup-backup generator was installed.
0 - at the time, at least, CO battery backup was usually sized for 24 hours of runtime.
Or you might want to have redundant cooling. Cooling system prices seem to scale fairly linearly with the cooling power above a few kW, so instead of one 100 kW system you could buy four 25 kW systems so a single failure won't be a disaster.
This will mean just about 4 times the number of failures, too. And can 75% cooling still cool the server room anyway?
It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
At a FAANG, our services are cattle, but we still plan which services to keep running when we need to drain 50% of a DC. Latency is important. Money makers > Latency sensitive > Optional requests > Background requests > Batch traffic. Bootstrapping is important. If A depends on B, you might to drain A first, or A and B together.
Hopefully one never has to face that scenario, but its much easier to pick up the pieces when you know where the priorities are whether you're having to power down servers or thin a herd.
Or you might want to have redundant cooling. Cooling system prices seem to scale fairly linearly with the cooling power above a few kW, so instead of one 100 kW system you could buy four 25 kW systems so a single failure won't be a disaster.
> Or you might want to have redundant cooling. Can you provide a cost centre or credit card for which they can bill this to? In case you didn't notice the domain, it is UToronto: academic departments aren't generally flush with cash. Further, you have to have physical space to fit the extra cooling equipment and pipes: not always easy or possible to do in old university buildings.
Of course these systems only get replaced every couple decades, if ever, so they are pretty much stuck with the setup they have.
It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the…
It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
> proudly declares himself Unix herder
You know what herder's herd? Cattle. Not pets.
> wants to keep track of which systems are important.
I mean obviously? Industry does the same. Probably with automation, tools and tagging during provisioning.
The pet mentality is when you create a beautiful handcrafted spreadsheet showing which services run on the server named "Mjolnir" and which services run on the server named "Valinor". The cattle mentality is when you have the same in a distributed key-value database with UUIDs instead of fancyful server names.
Or the pet mentality is when you prefer to not shut down "Mjolnir" because it has more than 2 years of uptime or some other silly reason like that. (as opposed to not shutting it down because you know that you would loose more money that way than by risking it overheating and having to buy a new one.)
Earlier quoted context omitted.
I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the…
At the risk of no true Scotsman, that doesn’t sound like “cattle not pets“; when the cattle are sent to the slaughterhouse there isn’t any blast radius, there’s just more cattle taking over. You explicitly don’t have to replace them with exact clones of the original cattle from tape very slowly, you spin up a herd of more cattle in moments.
Data isn't "sent to the slaughterhouse". Ever.
Data can be annoying that way.