Turning off servers seems like the wrong call instead of transitioning servers into a lower powered state which can be exited once the power budget is available again.
We might want to regularly keep track of how important each server is
51–60 of 147 posts
Re: We might want to regularly keep track of how important each server is
#52Asset management is definitely a thing. Tag your environments, tag your apps, and provide your apps criticality ratings based on how important they are to running the business. Then it's a matter of a query to know which servers can be shut, and which absolutely must remain.
In moments of crisis, immediate measures like physical tagging can be crucial. Yet, a broader challenge looms: our dependency on air conditioning. In Toronto's winter, the missed opportunity to design buildings that work with the climate, rather than defaulting to a universal AC solution, underscores the need for thoughtful asset management tailored to specific environments.
Re: We might want to regularly keep track of how important each server is
#53Earlier quoted context omitted.
At the risk of no true Scotsman, that doesn’t sound like “cattle not pets“; when the cattle are sent to the slaughterhouse there isn’t any blast radius, there’s just more cattle taking over. You explicitly don’t have to replace them with exact clones of the original cattle from tape very slowly, you spin up a herd of more cattle in moments.
> when the cattle are sent to the slaughterhouse Data isn't "sent to the slaughterhouse". Ever. Data can be annoying that way.
I think the point is that systems that aren’t replaced easily shouldn’t be managed like they are, not … whatever it is you’re getting at here.
Re: We might want to regularly keep track of how important each server is
#54Or you might want to have redundant cooling. Cooling system prices seem to scale fairly linearly with the cooling power above a few kW, so instead of one 100 kW system you could buy four 25 kW systems so a single failure won't be a disaster.
This will mean just about 4 times the number of failures, too. And can 75% cooling still cool the server room anyway?
It means 5 times the number of failures as you intentionally put in an extra unit so that one can be taken offline at any time for maintenance (which itself will keep the whole system more reliable), and if one fails the whole keeps up. The cost is only slightly more to do this when there are 5 smaller units. Those smaller units could be standard off the shelf units as well, so it could be cheaper than a large unit that isn't mad in as large a quantity - this is a consideration that needs to be made case by case)
Even if you cheap out and only install 4 units, odds are your failure doesn't happen on the hottest day of the year and so 3 can keep up just fine. It is only when you are unlikely that you need to shut anything down.
Re: We might want to regularly keep track of how important each server is
#55Or you might want to have redundant cooling. Cooling system prices seem to scale fairly linearly with the cooling power above a few kW, so instead of one 100 kW system you could buy four 25 kW systems so a single failure won't be a disaster.
Won't 4 x 25 kW systems mean also 4x the installation cost?
Maybe, but costs are not linear and the nonlinear goes different ways for different parts of the install. The costs of the smaller systems installed could be cheaper than just the large system not installed if the smaller systems are standard parts.
Re: We might want to regularly keep track of how important each server is
#56It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the…
>>The IT department decided they were done with their pets, moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array. There was a disk failure, but that’s ok, RAID-5 can handle that.
Precisely why, when I was charged with setting up a 100 TB array for a law firm client at previous job, I went for RAID-6, even though it came with a tremendous write speed hit. It was mostly archived data that needed retention for a long period of time, so it wasn't bad for daily usage, and read speeds were great. Had the budget been greater, RAID 10 would've been my choice. (requisite reminder: RAID is not backup)
Not related, but they were hit with a million dollar ransomware attack (as in: the hacker group requested a million dollar payment), so that write speed limitation was not the bottleneck considering internet speed when restoring. Ahhh.... what a shitshow, the FBI got involved, and never worked for them again. I did warn them though: zero updates (disabled) and also disabled firewall on the host data server (windows) was a recipe for disaster. Within 3 days they got hit, and the boss had the temerity to imply I had something to do with it. Glad I'm not there anymore, but what a screwy opsec situation I thankfully no longer have to support.
Re: We might want to regularly keep track of how important each server is
#57It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
The issue here is not much the hardware, but the services that run on top of them.
I guess that many companies that use "current practices" have plenty of services that they don't even know about running on their clusters.
The main difference is that instead of the kind of issues that the link talks about, you have those services running year after year, using resources, for the joy of the cloud companies.
This happens even at Google [1]:
"There are several remarkable aspects to this story. One is that running a Bigtable was so inconsequential to Google’s scale that it took 2 years before anyone even noticed it, and even then, only because the version was old. "
[1] https://steve-yegge.medium.com/dear-google-cloud-your-deprec...
Re: We might want to regularly keep track of how important each server is
#58Cloud environments have an elegant solution in the form of "spot" or "pre-emptible" instances: if your workload can tolerate interruptions because it's not time sensitive or not terribly important, you can get a pretty steep discount.
Re: We might want to regularly keep track of how important each server is
#59Turning off servers seems like the wrong call instead of transitioning servers into a lower powered state which can be exited once the power budget is available again.
The right answer is turn them all off - anything important is in a redundant data center. But odds are they don't have that.
If a redundant data center isn't an option, then you should put more into ensuring the system is resilient - fireproof room (if a server catches on fire it can't spread to the next - there are a lot of considerations here that I don't know about that you need to figure out), plenty of backup power, redundant HVAC, redundant connections to the internet - and you should brainstorm other things that I didn't think of.
Re: We might want to regularly keep track of how important each server is
#60It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the…
We bought a 108-drive DAS some years ago, for a backup storage.
I needed to actively object the idea of a 108-wide RAID5 on that.
File systems get borked, RAID arrays can have multiple failures, admins can issue rm -rf in the wrong place.