131–140 of 147 posts
Earlier quoted context omitted.
The devops meme referenced, "cattle not pets", probably popularized by a book called "The Phoenix Project". The real point is that pets are job security for the Unix herder. If you end up with one neckbeard running the joint that's even worse as a single point of failure.
If you end up with one cloud guru running your cloud, that's maybe worse as a single point of failure. AWS system admins may be more fungible these days than unix sysadmins. Maybe.
Asset management is definitely a thing. Tag your environments, tag your apps, and provide your apps criticality ratings based on how important they are to running the business. Then it's a matter of a query to know which servers can be shut, and which absolutely must remain.
> provide your apps criticality ratings based on how important they are to running the business In a decentralized, self-service model, you can add "deal with convincing a stakeholder their app is anything less than most-critical." Although it usually works itself out if higher-criticality imposes ongoing time commitments on them as well (aka stick).
"Everything is critical" should sound a million alarm bells in the minds of "enterprise architects" but most I've discussed this with are blissfully unaware.
Earlier quoted context omitted.
In moments of crisis, immediate measures like physical tagging can be crucial. Yet, a broader challenge looms: our dependency on air conditioning. In Toronto's winter, the missed opportunity to design buildings that work with the climate, rather than defaulting to a universal AC solution, underscores the need for thoughtful asset management tailored to specific environments.
Several data centres in Toronto (including the massive facilities at 151 Front Street West where most of the internet for the province passes through) make use of the deep lake cooling loop that takes water pumped in from Lake Ontario to cool equipment before moving on to other uses. Water is pumped in from a sufficient depth such that the temperature is fairly constant year round.
Would love to know if a data centre could get paid for rejecting it’s heat to the system during what is heating time for other users.
Years ago during a week-long power outage, a telephone central office where we had some equipment suffered a generator failure. The telephone company had a backup plan (they kept one generator on a trailer in the city for such a contingency,) and they had battery capacity[0] for their critical equipment to last until the generator was hooked up. They did have to load shed, though: they just turned off the AC inverter…
Well that’s more forward thinking than AT&T’s Nashville CO when it got bombed. They just depended on natgas grid for their generators if grid electric went out. They underestimated the correlation between energy grids.
When natgas got cutoff and the UPSs died, they had no ability to hook in roll-up generators and had to hastily install connection points (or hardline them). And no standby contracts for them.
Earlier quoted context omitted.
Honestly that was pretty much the situation. In our defence, he offered. It had hit hour 6 of both the primary and the backup aircon being down, on a very hot day - everyone was way beyond blame and the NOC staff were basically up for any creative solution they could find.
Wait, you didn't mean "he repositioned it a couple levels down on the rack to make some room above so he could unscrew the cover and crack it a bit open it like a grand piano"?
Though he did damage the lid in the process, so who knows how he got it open.
Earlier quoted context omitted.
In moments of crisis, immediate measures like physical tagging can be crucial. Yet, a broader challenge looms: our dependency on air conditioning. In Toronto's winter, the missed opportunity to design buildings that work with the climate, rather than defaulting to a universal AC solution, underscores the need for thoughtful asset management tailored to specific environments.
I know someone who did that in the Yukon during the winter, just monitor temperatures and crack a window when it got too hot. Seems like a great solution except that they were in a different building so they had to trudge through the snow to close the window if it got too cold.
didn’t think I’d see Yukon here :)
Earlier quoted context omitted.
Honestly, big issue with the cattle metaphor is that the individual services you run on servers are very much often not interchangeable. A DNS service is not NTP is not mail gateway is not application load balancer is not database etc etc etc At best, multiple replicas of those are cattle. And while you can treat the servers underlying them as interchangeable, that doesn't change the fact the services you run on them…
I think it's a way of thinking about things, rather than a true/false description. e.g. VMware virtual hosts make good cattle - in some setups I have worked on the hosts are interchangeable, move virtual machines between them without downtime. In others the hosts have different storage access, different connectivity and it matters which combination of hosts are online/offline together, and which VMs need the special…
What grinds my gears is that over years I found it a thought limiting meme - it effectively swings a metaphor too hard into one direction, and some early responses under original article IMO present quite well the issue. It's not like people are stupid - but metaphors like this exist to make shortcuts for thinking and discussion, and for last few years I've seen that it short-circuits the discussion too hard, making people either stop thinking about certain interdependencies, or stopping noticing that there are still systems they treat like "pets", just named differently and in different scope, but now mentally pushing out how fragile they can be.
Earlier quoted context omitted.
This will mean just about 4 times the number of failures, too. And can 75% cooling still cool the server room anyway?
It means 5 times the number of failures as you intentionally put in an extra unit so that one can be taken offline at any time for maintenance (which itself will keep the whole system more reliable), and if one fails the whole keeps up. The cost is only slightly more to do this when there are 5 smaller units. Those smaller units could be standard off the shelf units as well, so it could be cheaper than a large unit t…
> Even if you cheap out and only install 4 units, odds are your failure doesn't happen on the hottest day of the year
Earlier quoted context omitted.
Several data centres in Toronto (including the massive facilities at 151 Front Street West where most of the internet for the province passes through) make use of the deep lake cooling loop that takes water pumped in from Lake Ontario to cool equipment before moving on to other uses. Water is pumped in from a sufficient depth such that the temperature is fairly constant year round.
I think the system just has an isolated loop that heat exchanges with the incoming municipal water supply. Unsure if the whole system cools the loop glycol further or not, but ultimately there’s still a compressor-based aircon system sitting somewhere, probably at each building, that they’re depending on. They’re just not rejecting heat to the air (as much?). Would love to know if a data centre could get paid for rej…
This is definitely a thing in other parts of the world https://www.datacenterdynamics.com/en/news/stockholm-data-pa...
> IP-Only, Interxion and Advania Data Centers are building data centers on the Kista site, which is connected to Stockholm's district heating system so tenants get paid for their waste heat, which is used to warm local homes and businesses
Earlier quoted context omitted.
You may be thinking about Yahoo’s approach from 2010? > The Yahoo! approach is to avoid the capital cost and power consumption of chillers entirely by allowing the cold aisle temperatures to rise to 85F to 90F when they are unable to hold the temperature lower. They calculate they will only do this 34 hours a year which is less than 0.4% of the year. https://perspectives.mvdirona.com/2011/03/yahoo-compute-coop...
No, what I was remembering was a building design for datacenters, but I can't find a reference. Maybe it was only conceptual. The design was to pull in cold exterior air, pass thru the dehumidifiers to bring some of the moisture levels down, and vent heat from a high rise shaft out the top. All controlled to ensure humidity didn't get wrecked.