Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We might want to regularly keep track of how important each server is

utcc.utoronto.ca

Post reply on HN
Print view View original

91–100 of 147 posts

Re: We might want to regularly keep track of how important each server is

#91

Post by organsnyder »

macintux wrote: post #90
_dan wrote: post #83

Similar thing (catastrophic aircon failure due to a flood in a crap colocated DC) happened to us too before we shifted to AWS. Photos from the colo were pretty bizarre - fans balanced on random boxes, makeshift aircon ducting made of cardboard and tape, and some dude flailing an open fire door back and forth all day to get a little bit of fresh air in. Bizarre to see in 2010-ish with multi million dollar customers. W…

I find it’s much less stressful to rescue situations where it wasn’t your fault to begin with. Absent the ability to point fingers at a vendor, crises like that are a miserable experience for me.

Being the hero always feels better than cleaning up your own messes.

Re: We might want to regularly keep track of how important each server is

#92

Post by ElevenLathe »

tonyarkles wrote: post #76
jodrellblank wrote: post #28

Earlier quoted context omitted.

At the risk of no true Scotsman, that doesn’t sound like “cattle not pets“; when the cattle are sent to the slaughterhouse there isn’t any blast radius, there’s just more cattle taking over. You explicitly don’t have to replace them with exact clones of the original cattle from tape very slowly, you spin up a herd of more cattle in moments.

> you spin up a herd of more cattle in moments Where does the decade of data they've been collecting come from when you "spin up a new herd"?

It's stored in another system that is probably treated as a pet, hopefully by somebody way better at it with a huge staff (like AWS). Even if it's a local NetApp cluster or something, you can leave the state to the storage admins rather than random engineers that may or may not even be with the company any more.

Re: We might want to regularly keep track of how important each server is

#93

Post by SonOfLilit »

_dan wrote: post #83

Similar thing (catastrophic aircon failure due to a flood in a crap colocated DC) happened to us too before we shifted to AWS. Photos from the colo were pretty bizarre - fans balanced on random boxes, makeshift aircon ducting made of cardboard and tape, and some dude flailing an open fire door back and forth all day to get a little bit of fresh air in. Bizarre to see in 2010-ish with multi million dollar customers. W…

> have the on site engineer drop it down enough to crack the lid open

Took me four reads to find an alternative way to read it other than "we asked some guy that doesn't even work for us to throw it on the ground repeatedly until the cover cracks open", like that Zoolander scene.

Re: We might want to regularly keep track of how important each server is

#94

Post by [deleted] »

gregmac wrote: post #73
throw0101a wrote: post #10

Earlier quoted context omitted.

> Or you might want to have redundant cooling. Can you provide a cost centre or credit card for which they can bill this to? In case you didn't notice the domain, it is UToronto: academic departments aren't generally flush with cash. Further, you have to have physical space to fit the extra cooling equipment and pipes: not always easy or possible to do in old university buildings.

This kind of thing is like insurance. Maybe IT failed to state the consequences of not having redundancy, maybe people in control of the money failed to understand.. or maybe the risks were understood and accepted. Either way, by not paying for the insurance (redundant systems) up front the organization is explicitly taking on the risk. Whether the cost now is higher is impossible to say as an outsider, but there's a…

[deleted]

Re: We might want to regularly keep track of how important each server is

#95

Post by bluGill »

naniwaduni wrote: post #80
p_l wrote: post #77

Earlier quoted context omitted.

Honestly, big issue with the cattle metaphor is that the individual services you run on servers are very much often not interchangeable. A DNS service is not NTP is not mail gateway is not application load balancer is not database etc etc etc At best, multiple replicas of those are cattle. And while you can treat the servers underlying them as interchangeable, that doesn't change the fact the services you run on them…

Cattle often aren't interchangeable too. Not gonna have a great time milking the bulls.

But if you are milking you don't have bulls. Maybe you have one (though almost everyone uses artificial insemination these days). Worrying about milking bulls is like worrying about the NetWare server - once common but has been obsolete since before many reading this were even born.

Of course the pigs, cows, and chickens are not interchangeable. Nor are corn, hay, soybeans.

Re: We might want to regularly keep track of how important each server is

#96

Post by jodrellblank »

tonyarkles wrote: post #76
jodrellblank wrote: post #28

Earlier quoted context omitted.

At the risk of no true Scotsman, that doesn’t sound like “cattle not pets“; when the cattle are sent to the slaughterhouse there isn’t any blast radius, there’s just more cattle taking over. You explicitly don’t have to replace them with exact clones of the original cattle from tape very slowly, you spin up a herd of more cattle in moments.

> you spin up a herd of more cattle in moments Where does the decade of data they've been collecting come from when you "spin up a new herd"?

If you are using clustered storage (Ceph, for example) instead of a single RAID5 array, ideally the loss of one node or one rack or one site doesn't lose your data it only loses some of the replicas. When you spin up new storage nodes, the data replicates from the other nodes in the cluster. If you need 'the university storage server' that's a pet. Google aren't keeping pet webservers and pet mailbox servers for GMail - whichever loadbalanced webserver you get connected to will work with any storage cluster node it talks to. Microsoft aren't keeping pet webservers and mailbox servers for Office365, either. If they lose one storage array, or rack, or one DC, your data isn't gone.

'Cattle' is the idea that if you need more storage, you spin up more identikit storage servers and they merge in seamlessly and provide more replicated redundant storage space. If some break, you replace them with identikit ones which seamlessly take over. If you need data, any of them will provide it.

'Pets' is the idea that you need the email storage server, which is that HP box in the corner with the big RAID5 array. If you need more storage, it needs to be expansion shelves compatible with that RAID controller and its specific firmware versions which needs space and power in the same rack, and that's different from your newer Engineering storage server, and different to your Backup storage server. If the HP fails, the service is down until you get parts for that specific HP server, or restore that specific server's data to one new pet.

And yes, it's a model not a reality. It's easier to think about scaling your services if you have "two large storage clusters" than if you have a dozen different specialist storage servers each with individual quirks and individual support contracts which can only be worked on by individual engineers who know what's weird and unique about them. And if you can reorganise from pets to cattle, it can free up time, attention, make things more scalable, more flexible, make trade offs of maintenance time and effort.

Re: We might want to regularly keep track of how important each server is

#98

Post by letsdothisagain »

j33zusjuice wrote: post #64
ethbr1 wrote: post #35

Earlier quoted context omitted.

> provide your apps criticality ratings based on how important they are to running the business In a decentralized, self-service model, you can add "deal with convincing a stakeholder their app is anything less than most-critical." Although it usually works itself out if higher-criticality imposes ongoing time commitments on them as well (aka stick).

That seems like a poorly run company. Idk. Maybe we’ve worked in very different environments, but devs have almost always been aware of the criticality of the app, so convincing people wasn’t hard. In most places, the answer hinges on “is it customer facing?” and/or “does it break a core part of our business?” If the answer is no to both, it’s not critical, and everyone understands that. There’s always some weird out…

Welcome to University IT, where organizational structures are basically feudal (by law!). Imagine an organization where your president can't order a VP to do something, and you have academia :)

Re: We might want to regularly keep track of how important each server is

#99

Post by carbotaniuman »

j33zusjuice wrote: post #53
logifail wrote: post #30

Earlier quoted context omitted.

> when the cattle are sent to the slaughterhouse Data isn't "sent to the slaughterhouse". Ever. Data can be annoying that way.

I think the point is that systems that aren’t replaced easily shouldn’t be managed like they are, not … whatever it is you’re getting at here.

Sadly you can't just spool down data either - data isn't fungible!

Re: We might want to regularly keep track of how important each server is

#100

Post by letsdothisagain »

outofpaper wrote: post #52
spydum wrote: post #11

Asset management is definitely a thing. Tag your environments, tag your apps, and provide your apps criticality ratings based on how important they are to running the business. Then it's a matter of a query to know which servers can be shut, and which absolutely must remain.

In moments of crisis, immediate measures like physical tagging can be crucial. Yet, a broader challenge looms: our dependency on air conditioning. In Toronto's winter, the missed opportunity to design buildings that work with the climate, rather than defaulting to a universal AC solution, underscores the need for thoughtful asset management tailored to specific environments.

I know someone who did that in the Yukon during the winter, just monitor temperatures and crack a window when it got too hot. Seems like a great solution except that they were in a different building so they had to trudge through the snow to close the window if it got too cold.
Post reply on HN