Any server that must not fail is just not important enough.
(Build your services with HA in mind, so you don‘t have to worry about a situation like this one.)
We might want to regularly keep track of how important each server is
101–110 of 147 posts
Re: We might want to regularly keep track of how important each server is
#102Earlier quoted context omitted.
In moments of crisis, immediate measures like physical tagging can be crucial. Yet, a broader challenge looms: our dependency on air conditioning. In Toronto's winter, the missed opportunity to design buildings that work with the climate, rather than defaulting to a universal AC solution, underscores the need for thoughtful asset management tailored to specific environments.
Toronto's climate and winters is dramatically changing, the universal AC solution is almost mandatory due to the climate not being as cold in this area as it once was.
do you have a source for that? my source[1] appears the average temp hasn't changed much in the past quarter century:
Re: We might want to regularly keep track of how important each server is
#103Earlier quoted context omitted.
In moments of crisis, immediate measures like physical tagging can be crucial. Yet, a broader challenge looms: our dependency on air conditioning. In Toronto's winter, the missed opportunity to design buildings that work with the climate, rather than defaulting to a universal AC solution, underscores the need for thoughtful asset management tailored to specific environments.
I upvoted, but I agree so much, I had to comment, too. I wonder how long it’d take to recoup the loss of retrofitting such a system. Despite this story today, this type of problem must be rare. I imagine most of the savings would be found in the electric bill, and it’d probably take a lot of years to recoup the cost.
It's pretty common for hyperscalers actually:
https://betterbuildingssolutioncenter.energy.gov/showcase-pr...
https://greenmountain.no/data-centers/cooling/
I vaguely remember some other whole building DC designs that used a central vent which opened externally based on external climate for some additional free cooling. Can't find the reference now though. But geothermal is pretty common for sure.
Re: We might want to regularly keep track of how important each server is
#104Similar thing (catastrophic aircon failure due to a flood in a crap colocated DC) happened to us too before we shifted to AWS. Photos from the colo were pretty bizarre - fans balanced on random boxes, makeshift aircon ducting made of cardboard and tape, and some dude flailing an open fire door back and forth all day to get a little bit of fresh air in. Bizarre to see in 2010-ish with multi million dollar customers. W…
> some dude flailing an open fire door back and forth all day to get a little bit of fresh air in
That's hilarious (probably for you as well, in hindsight). Do you feel comfortable naming and shaming this DC, so we know to avoid it?
Re: We might want to regularly keep track of how important each server is
#105Earlier quoted context omitted.
Toronto's climate and winters is dramatically changing, the universal AC solution is almost mandatory due to the climate not being as cold in this area as it once was.
do you have a source for that? my source[1] appears the average temp hasn't changed much in the past quarter century: https://toronto.weatherstats.ca/metrics/temperature.html
Average temp probably isn’t what you need here - peak temperature and length of high temperature conditions would be more important when figuring out if you need to have artificial cooling available.
Re: We might want to regularly keep track of how important each server is
#106Earlier quoted context omitted.
I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the…
>>The IT department decided they were done with their pets, moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array. There was a disk failure, but that’s ok, RAID-5 can handle that. Precisely why, when I was charged with setting up a 100 TB array for a law firm client at previous job, I went for RAID-6, even though it came with a tremendous write speed hit. It was mostly archived data that ne…
> the boss had the temerity to imply I had something to do with it.
What was your response? I feel like mine would be "you are now accusing me of a severe crime, all further correspondence will be through my lawyer, good luck".
Re: We might want to regularly keep track of how important each server is
#107One place I worked did a backup power test and when they came back from the diesels to the grid the entire datacentre lost power due to a software bug for about 10 seconds. It caused a massive outage.
The problem was a lot of the machines pulled their OS image from central storage servers and there was no where near enough IO to load everything and they had to prioritise what to bring up first to lighten the load and stop everything thrashing. It was a complete nightmare even though the front end to take sales were well isolated from the backend. Working out what was most important across an entire corporation took as long as the problem resolving slowly by just bringing things up randomly.
Nowadays you would just run multiple datacentres or cloud HA and we have SSDs but I just can't see such an architecture understanding being possible for any reasonably large company. The cost of keeping it and the dependencies up to date would be huge and it would always be out of date. More documentation isn't the solution, its to have multiple sites.
Re: We might want to regularly keep track of how important each server is
#108If machines aren't important, why do you have them at all? Because it's an university and we don't care about justifying costs?
It's possible to have two important things and yet for one to be more important than the other.
Re: We might want to regularly keep track of how important each server is
#109One place I worked did a backup power test and when they came back from the diesels to the grid the entire datacentre lost power due to a software bug for about 10 seconds. It caused a massive outage. The problem was a lot of the machines pulled their OS image from central storage servers and there was no where near enough IO to load everything and they had to prioritise what to bring up first to lighten the load and…
That brings back memories of a similar setup with hundreds of Windows servers booting over the network. We had regular “brownouts” even during the day just because the image streaming servers couldn’t handle the IOPS. Basic maintenance would slow down the servers for ten thousand users and generate support tickets.
I jumped up and down and convinced management to buy one of the first enterprise SSDs on the market. It was a PCIe card form factor and cost five digits for a tiny amount of storage.
We squeezed in the images using block-level deduplication and clever copy scripts that would run the compaction routine after each file was copied.
The difference was staggering. Just two of those cards made hundreds of other servers run like greased lightning. Boot times dropped to single digit seconds instead of minutes. Maintenance changes could be done at any time with zero impact on users. The whole cluster could be rebooted all at once with only a slight slowdown. Fun times.
Re: We might want to regularly keep track of how important each server is
#110Earlier quoted context omitted.
I upvoted, but I agree so much, I had to comment, too. I wonder how long it’d take to recoup the loss of retrofitting such a system. Despite this story today, this type of problem must be rare. I imagine most of the savings would be found in the electric bill, and it’d probably take a lot of years to recoup the cost.
It's pretty common for hyperscalers actually: https://betterbuildingssolutioncenter.energy.gov/showcase-pr... https://greenmountain.no/data-centers/cooling/ I vaguely remember some other whole building DC designs that used a central vent which opened externally based on external climate for some additional free cooling. Can't find the reference now though. But geothermal is pretty common for sure.
You may be thinking about Yahoo’s approach from 2010?
> The Yahoo! approach is to avoid the capital cost and power consumption of chillers entirely by allowing the cold aisle temperatures to rise to 85F to 90F when they are unable to hold the temperature lower. They calculate they will only do this 34 hours a year which is less than 0.4% of the year.
https://perspectives.mvdirona.com/2011/03/yahoo-compute-coop...