Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We might want to regularly keep track of how important each server is

utcc.utoronto.ca

Post reply on HN
Print view View original

111–120 of 147 posts

Re: We might want to regularly keep track of how important each server is

#111

Post by _dan »

SonOfLilit wrote: post #93
_dan wrote: post #83

Similar thing (catastrophic aircon failure due to a flood in a crap colocated DC) happened to us too before we shifted to AWS. Photos from the colo were pretty bizarre - fans balanced on random boxes, makeshift aircon ducting made of cardboard and tape, and some dude flailing an open fire door back and forth all day to get a little bit of fresh air in. Bizarre to see in 2010-ish with multi million dollar customers. W…

> have the on site engineer drop it down enough to crack the lid open Took me four reads to find an alternative way to read it other than "we asked some guy that doesn't even work for us to throw it on the ground repeatedly until the cover cracks open", like that Zoolander scene.

Honestly that was pretty much the situation.

In our defence, he offered. It had hit hour 6 of both the primary and the backup aircon being down, on a very hot day - everyone was way beyond blame and the NOC staff were basically up for any creative solution they could find.

Re: We might want to regularly keep track of how important each server is

#112

Post by quickthrower2 »

This is where the cloud kicks ass. Run multiple nodes with geo redundancy (where based on various concerns: cost, contracts, legal). But nodes should cross data centres. Maybe if one city gets nuked (literally or a fire/power outage) you still have uptime. Use Kubernetes maybe.

Re: We might want to regularly keep track of how important each server is

#113

Post by iwontberude »

Isn't this the point of decoupling your compute and datastores using CSI with disaggregated storage Kubernetes? So long as you keep your datastores available, whatever compute you can manage to attach it from Kubernetes can run whatever you truly need at capacities that you can handle with that level of hardware. Similarly, you could scale down the workloads on all the machines so they generated less heat without turning anything off at the expense of performance.

Re: We might want to regularly keep track of how important each server is

#114

Post by philsnow »

b112 wrote: post #16
luma wrote: post #12

Earlier quoted context omitted.

When you are responsible for the full infrastructure, sequencing power down and power on in coordination with your UPS is a common solution. Network gear needs a few minutes to light up ports, core services like DNS and identity services might need to light up next, then storage, then hypervisors and container hosts, then you can actually start working on app dependencies. This sort of sequencing leads itself natural…

Double? Try 100x!!

If you’re running a couple ec2 instances in one AZ then yeah it’s closer to 100x, but if you wanted to replicate the durability of S3, it would cost you a lot in terms of redundancy (usually “invisible” to the customer) and ongoing R&D and support headcount.

Yes, even when you add it all up, Amazon still charges a premium even over that all-in cost. That’s sweat equity.

Re: We might want to regularly keep track of how important each server is

#115

Post by spydum »

thakoppno wrote: post #110
spydum wrote: post #103

Earlier quoted context omitted.

It's pretty common for hyperscalers actually: https://betterbuildingssolutioncenter.energy.gov/showcase-pr... https://greenmountain.no/data-centers/cooling/ I vaguely remember some other whole building DC designs that used a central vent which opened externally based on external climate for some additional free cooling. Can't find the reference now though. But geothermal is pretty common for sure.

You may be thinking about Yahoo’s approach from 2010? > The Yahoo! approach is to avoid the capital cost and power consumption of chillers entirely by allowing the cold aisle temperatures to rise to 85F to 90F when they are unable to hold the temperature lower. They calculate they will only do this 34 hours a year which is less than 0.4% of the year. https://perspectives.mvdirona.com/2011/03/yahoo-compute-coop...

No, what I was remembering was a building design for datacenters, but I can't find a reference. Maybe it was only conceptual. The design was to pull in cold exterior air, pass thru the dehumidifiers to bring some of the moisture levels down, and vent heat from a high rise shaft out the top. All controlled to ensure humidity didn't get wrecked.

Re: We might want to regularly keep track of how important each server is

#116

Post by antod »

_heimdall wrote: post #25
cowsandmilk wrote: post #5

It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.

I'm pretty sure anyone in the industry that draws this distinction between cattle and pets has never worked with cattle and only knows of general ideas about the industrial cattle business.

Likewise, anyone talking about civil engineering and bridges.

At least back in the day Slashdot was fully aware of how broken their ubiquitous car analogies were and played it up.

Re: We might want to regularly keep track of how important each server is

#117

Post by tetha »

_dan wrote: post #83

Similar thing (catastrophic aircon failure due to a flood in a crap colocated DC) happened to us too before we shifted to AWS. Photos from the colo were pretty bizarre - fans balanced on random boxes, makeshift aircon ducting made of cardboard and tape, and some dude flailing an open fire door back and forth all day to get a little bit of fresh air in. Bizarre to see in 2010-ish with multi million dollar customers. W…

> Similar thing (catastrophic aircon failure due to a flood in a crap colocated DC) happened to us too before we shifted to AWS. Photos from the colo were pretty bizarre - fans balanced on random boxes, makeshift aircon ducting made of cardboard and tape, and some dude flailing an open fire door back and forth all day to get a little bit of fresh air in. Bizarre to see in 2010-ish with multi million dollar customers.

I'd have considered calling a few friends from the fire brigade or the catastrophe protection there.

It's not an emergency, yes. However, if you want a situation for your trainees to figure out how to ventilate a building with the force of a thousand gasoline driven fans without anyone complaining and no danger to any person... well be my guest because I can't hear you anymore. Those really big fans are loud AF, seriously.

And, on a more serious note, you could show those blokes how a DC works. Where power goes, what components do, how to handle uncontrolled fire in areas. Would be a major benefit to the local fire fighters.

Re: We might want to regularly keep track of how important each server is

#118

Post by SonOfLilit »

_dan wrote: post #111
SonOfLilit wrote: post #93

Earlier quoted context omitted.

> have the on site engineer drop it down enough to crack the lid open Took me four reads to find an alternative way to read it other than "we asked some guy that doesn't even work for us to throw it on the ground repeatedly until the cover cracks open", like that Zoolander scene.

Honestly that was pretty much the situation. In our defence, he offered. It had hit hour 6 of both the primary and the backup aircon being down, on a very hot day - everyone was way beyond blame and the NOC staff were basically up for any creative solution they could find.

Wait, you didn't mean "he repositioned it a couple levels down on the rack to make some room above so he could unscrew the cover and crack it a bit open it like a grand piano"?

Re: We might want to regularly keep track of how important each server is

#119

Post by RexM »

organsnyder wrote: post #91
macintux wrote: post #90

Earlier quoted context omitted.

I find it’s much less stressful to rescue situations where it wasn’t your fault to begin with. Absent the ability to point fingers at a vendor, crises like that are a miserable experience for me.

Being the hero always feels better than cleaning up your own messes.

Why not both?
Post reply on HN