Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We might want to regularly keep track of how important each server is

utcc.utoronto.ca

Post reply on HN
Print view View original

71–80 of 147 posts

Re: We might want to regularly keep track of how important each server is

#71

Post by light_hue_1 »

bluGill wrote: post #50
wongarsu wrote: post #27

Earlier quoted context omitted.

If you designed the system like this from the start or when replacing it anyways, N+1 redundancy might not me much more expensive than one big cooling unit. The systems can mostly share their ductwork and just have redundancy in the active components, so mostly the chillers. Of course these systems only get replaced every couple decades, if ever, so they are pretty much stuck with the setup they have.

> Of course these systems only get replaced every couple decades, if ever This is despite the massive energy savings they could get if they replaced those older systems. Universities often are full of old buildings with terrible insulation heated/cooled by very old/inefficient systems. In 20 years they would be money ahead by tearing down most buildings on campus and rebuilding to modern standards - assuming energy c…

> In 20 years they would be money ahead by tearing down most buildings on campus and rebuilding to modern standards - assuming energy costs don't go up which seems unlikely) But they consider all those old buildings historic and so won't.

It has nothing to do with considering those building historic.

The problem is unless someone wants to donate a $50-100M, new buildings don't happen. And big donors want to donate to massive causes "Build a new building to cure cancer!" not "This building is kind of crappy, let's replace it with a better one".

It doesn't matter that over 50 years something could be cheaper if there's no money to fix it now.

Re: We might want to regularly keep track of how important each server is

#72

Post by monkeywork »

outofpaper wrote: post #52
spydum wrote: post #11

Asset management is definitely a thing. Tag your environments, tag your apps, and provide your apps criticality ratings based on how important they are to running the business. Then it's a matter of a query to know which servers can be shut, and which absolutely must remain.

In moments of crisis, immediate measures like physical tagging can be crucial. Yet, a broader challenge looms: our dependency on air conditioning. In Toronto's winter, the missed opportunity to design buildings that work with the climate, rather than defaulting to a universal AC solution, underscores the need for thoughtful asset management tailored to specific environments.

Toronto's climate and winters is dramatically changing, the universal AC solution is almost mandatory due to the climate not being as cold in this area as it once was.

Re: We might want to regularly keep track of how important each server is

#73

Post by gregmac »

throw0101a wrote: post #10
tlb wrote: post #2

Or you might want to have redundant cooling. Cooling system prices seem to scale fairly linearly with the cooling power above a few kW, so instead of one 100 kW system you could buy four 25 kW systems so a single failure won't be a disaster.

> Or you might want to have redundant cooling. Can you provide a cost centre or credit card for which they can bill this to? In case you didn't notice the domain, it is UToronto: academic departments aren't generally flush with cash. Further, you have to have physical space to fit the extra cooling equipment and pipes: not always easy or possible to do in old university buildings.

This kind of thing is like insurance. Maybe IT failed to state the consequences of not having redundancy, maybe people in control of the money failed to understand.. or maybe the risks were understood and accepted.

Either way, by not paying for the insurance (redundant systems) up front the organization is explicitly taking on the risk.

Whether the cost now is higher is impossible to say as an outsider, but there's a lot of expenses: paying a premium for emergency repairs/replacement; paying salaries to a bunch of staff who are unable to work at full capacity (or maybe at all); a bunch of IT projects delayed because staff is dealing with an outage; and maybe downstream ripple effects, like classes cancelled or research projects in jeopardy.

I've never worked in academics, but I know people that do and understand the budget nonsense they go through. It doesn't change the reality, though, which is systems fail and if you don't plan for that you'll pay dearly.

Re: We might want to regularly keep track of how important each server is

#74

Post by tonyarkles »

throw0101b wrote: post #39
tonyarkles wrote: post #18

Earlier quoted context omitted.

I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the…

> You have a capex budget […] As someone who has worked IT in academia: no, you do not. :)

I'm laughing (and crying) with you, not at you.

From my past life in academia, you're totally right. But that kind of reinforces the point... you do occasionally get some budget for servers and then have to make them last as long as possible :). Those one-time expenses are generally more palatable than recurring cloud storage costs though.

Re: We might want to regularly keep track of how important each server is

#75

Post by evilduck »

dotancohen wrote: post #21
tlb wrote: post #2

Or you might want to have redundant cooling. Cooling system prices seem to scale fairly linearly with the cooling power above a few kW, so instead of one 100 kW system you could buy four 25 kW systems so a single failure won't be a disaster.

This will mean just about 4 times the number of failures, too. And can 75% cooling still cool the server room anyway?

Four service degradations vs. one huge outage event. Pick your poison.

Re: We might want to regularly keep track of how important each server is

#76

Post by tonyarkles »

jodrellblank wrote: post #28
tonyarkles wrote: post #18

Earlier quoted context omitted.

I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the…

At the risk of no true Scotsman, that doesn’t sound like “cattle not pets“; when the cattle are sent to the slaughterhouse there isn’t any blast radius, there’s just more cattle taking over. You explicitly don’t have to replace them with exact clones of the original cattle from tape very slowly, you spin up a herd of more cattle in moments.

> you spin up a herd of more cattle in moments

Where does the decade of data they've been collecting come from when you "spin up a new herd"?

Re: We might want to regularly keep track of how important each server is

#77

Post by p_l »

bluGill wrote: post #43
_heimdall wrote: post #26

Earlier quoted context omitted.

The cattle metaphors really is a bad one. Anyone raising cattle should do the same thing, knowing which animals are the priority in case of draught, disease, etc. Hopefully one never has to face that scenario, but its much easier to pick up the pieces when you know where the priorities are whether you're having to power down servers or thin a herd.

Cattle are often interchangeable. You cull any that catch a disease (in some cases the USDA will cull the entire herd if just one catches something - bio security is a big deal) In the case of drought you pick a bunch to get rid of - based on market prices (If everyone else is you will try to keep yours because the market is collapsing - but this means managing feed and thus may mean culling more of the herd latter.)…

Honestly, big issue with the cattle metaphor is that the individual services you run on servers are very much often not interchangeable.

A DNS service is not NTP is not mail gateway is not application load balancer is not database etc etc etc

At best, multiple replicas of those are cattle.

And while you can treat the servers underlying them as interchangeable, that doesn't change the fact the services you run on them are not.

Re: We might want to regularly keep track of how important each server is

#78

Post by throwawaaarrgh »

If your machines are all hypervisors you could migrate important VMs to a couple hosts and turn off the rest. You could also possibly throttle the vcpus, which would slow down the VMs but allow you to run the machines cooler, or more VMs per machine. Finally the ones with long running jobs could just be snapshotted and powered down and restored later, resuming their computation.

There's a reason us old fogies were so excited when virtual machines got increasingly robust. We could use them to solve problems quickly that used to be nearly impossible.

Re: We might want to regularly keep track of how important each server is

#79

Post by hinkley »

j33zusjuice wrote: post #64
ethbr1 wrote: post #35

Earlier quoted context omitted.

> provide your apps criticality ratings based on how important they are to running the business In a decentralized, self-service model, you can add "deal with convincing a stakeholder their app is anything less than most-critical." Although it usually works itself out if higher-criticality imposes ongoing time commitments on them as well (aka stick).

That seems like a poorly run company. Idk. Maybe we’ve worked in very different environments, but devs have almost always been aware of the criticality of the app, so convincing people wasn’t hard. In most places, the answer hinges on “is it customer facing?” and/or “does it break a core part of our business?” If the answer is no to both, it’s not critical, and everyone understands that. There’s always some weird out…

I maintain a couple of apps that are pretty much free to break or to revert back to an older build without much consequence, except for one day a week, when half the team uses them instead of just me.

Any other day I can use them to test new base images, new coding or deployment techniques, etc. I just have to put things back by the end of our cycle.

Re: We might want to regularly keep track of how important each server is

#80

Post by naniwaduni »

p_l wrote: post #77
bluGill wrote: post #43

Earlier quoted context omitted.

Cattle are often interchangeable. You cull any that catch a disease (in some cases the USDA will cull the entire herd if just one catches something - bio security is a big deal) In the case of drought you pick a bunch to get rid of - based on market prices (If everyone else is you will try to keep yours because the market is collapsing - but this means managing feed and thus may mean culling more of the herd latter.)…

Honestly, big issue with the cattle metaphor is that the individual services you run on servers are very much often not interchangeable. A DNS service is not NTP is not mail gateway is not application load balancer is not database etc etc etc At best, multiple replicas of those are cattle. And while you can treat the servers underlying them as interchangeable, that doesn't change the fact the services you run on them…

Cattle often aren't interchangeable too. Not gonna have a great time milking the bulls.
Post reply on HN