We might want to regularly keep track of how important each server is

Good plan! I think this is a relatively common practice within some corners of the telecom world.

At university (Western WA in B'ham), I worked for our campus resnet, which had extensive involvement with other networking groups on campus. They ran layers 3 and below on the resnets, we took DNS+DHCP, plus egress, and everything through to layers 8 and 9.

The core network gear was co-located in a few musty basements along with the telephone switches. DC and backup power was available, but severely limited under certain failure scenarios.

All of the racked networking gear in the primary space was labeled with red and green dots. Green was first to go in any load-shedding scenario. Think: redundant LAN switches, switches carrying local ports, network monitoring servers, other +1 redundant components, etc.

I'm not sure if the scheme was ever required in real life, but do know it was based on hard-earned experiences like the author here.

Similar thing (catastrophic aircon failure due to a flood in a crap colocated DC) happened to us too before we shifted to AWS. Photos from the colo were pretty bizarre - fans balanced on random boxes, makeshift aircon ducting made of cardboard and tape, and some dude flailing an open fire door back and forth all day to get a little bit of fresh air in. Bizarre to see in 2010-ish with multi million dollar customers.

We ended up having to strategically shut servers down as well, but the question of what's critical, where is it in the racks, and what's next to it was incredibly difficult to answer. And kinda mind-bending - we'd been thinking of these things as completely virtualised resources for years, suddenly having to consider their physical characteristics as well was a bit of a shock. Just shutting down everything non-critical wasn't enough - there were still now critical non-redundant servers next to each other overheating.

All we had to go on was an outdated racktables install, a readout of the case temperature for each, and a map of which machine was connected to which switch port which loosely related to position in the rack - none completely accurate. In the end we got the colo guys to send a photo of the rack front and back and (though not everything was well labelled) we were able to make some decisions and get things stable again.

In the end one server that was critical but we couldn't get to run cooler we got lucky with - we were able to pull out the server below and (without shutting it down) have the on site engineer drop it down enough to crack the lid open and get some cool air into it to keep it running (albeit with no redundancy and on the edge of thermal shutdown).

We came really close to a major outage that day that would have cost us dearly. I know it sounds like total shambles (and it kinda was) but I miss those days.

>devs have almost always been aware of the criticality of the app

I'm sure that developers are aware of the how important their stuff is to their immediate customer, but they're almost never aware of the relative criticality vis-a-vis stuff they don't own or have any idea about.

Not sure if the goal was just to make an amusing comparison, but these are actually two completely different concerns.

Building your systems so that they don't depend on permanent infrastructure and snowflake configurations is an orthogonal concern from understanding how to shed load in a business-continuity crisis.

> budgets often work differently.

Very differently. Instead of a system you continually iterate on for its entire lifetime, if you're in a more regulated research area you might build it once, get it approved, and then it's only critical updates for the next five (or more!) years while data is collected.

Not many of the IT principles developed for web app startups apply in the research domain. They're less like cattle or pets and more like satellites which have very limited ability to be changed after launch.