Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We might want to regularly keep track of how important each server is

utcc.utoronto.ca

Post reply on HN
Print view View original

61–70 of 147 posts

Re: We might want to regularly keep track of how important each server is

#61

Post by datadrivenangel »

verticalscaler wrote: post #49
tonyarkles wrote: post #18

Earlier quoted context omitted.

I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the…

The devops meme referenced, "cattle not pets", probably popularized by a book called "The Phoenix Project". The real point is that pets are job security for the Unix herder. If you end up with one neckbeard running the joint that's even worse as a single point of failure.

If you end up with one cloud guru running your cloud, that's maybe worse as a single point of failure.

AWS system admins may be more fungible these days than unix sysadmins. Maybe.

Re: We might want to regularly keep track of how important each server is

#62

Post by justsomehnguy »

fishtacos wrote: post #56
tonyarkles wrote: post #18

Earlier quoted context omitted.

I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the…

>>The IT department decided they were done with their pets, moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array. There was a disk failure, but that’s ok, RAID-5 can handle that. Precisely why, when I was charged with setting up a 100 TB array for a law firm client at previous job, I went for RAID-6, even though it came with a tremendous write speed hit. It was mostly archived data that ne…

> even though it came with a tremendous write speed hit

Only on a writes < stripe. If your writes are bigger then you can have way more speed than RAID10 on the same set, limited only by the RAID controller CPU.

Re: We might want to regularly keep track of how important each server is

#63

Post by datadrivenangel »

spydum wrote: post #11

Asset management is definitely a thing. Tag your environments, tag your apps, and provide your apps criticality ratings based on how important they are to running the business. Then it's a matter of a query to know which servers can be shut, and which absolutely must remain.

Good documentation and metadata like this is necessary for corporations to truly be organized.

Re: We might want to regularly keep track of how important each server is

#64

Post by j33zusjuice »

ethbr1 wrote: post #35
spydum wrote: post #11

Asset management is definitely a thing. Tag your environments, tag your apps, and provide your apps criticality ratings based on how important they are to running the business. Then it's a matter of a query to know which servers can be shut, and which absolutely must remain.

> provide your apps criticality ratings based on how important they are to running the business In a decentralized, self-service model, you can add "deal with convincing a stakeholder their app is anything less than most-critical." Although it usually works itself out if higher-criticality imposes ongoing time commitments on them as well (aka stick).

That seems like a poorly run company. Idk. Maybe we’ve worked in very different environments, but devs have almost always been aware of the criticality of the app, so convincing people wasn’t hard. In most places, the answer hinges on “is it customer facing?” and/or “does it break a core part of our business?” If the answer is no to both, it’s not critical, and everyone understands that. There’s always some weird outlier, “well, this runs process B to report on process A, and sends a custom report to the CEO …”, but hopefully those exceptions are rare.

Re: We might want to regularly keep track of how important each server is

#66

Post by j33zusjuice »

outofpaper wrote: post #52
spydum wrote: post #11

Asset management is definitely a thing. Tag your environments, tag your apps, and provide your apps criticality ratings based on how important they are to running the business. Then it's a matter of a query to know which servers can be shut, and which absolutely must remain.

In moments of crisis, immediate measures like physical tagging can be crucial. Yet, a broader challenge looms: our dependency on air conditioning. In Toronto's winter, the missed opportunity to design buildings that work with the climate, rather than defaulting to a universal AC solution, underscores the need for thoughtful asset management tailored to specific environments.

I upvoted, but I agree so much, I had to comment, too. I wonder how long it’d take to recoup the loss of retrofitting such a system. Despite this story today, this type of problem must be rare. I imagine most of the savings would be found in the electric bill, and it’d probably take a lot of years to recoup the cost.

Re: We might want to regularly keep track of how important each server is

#67

Post by j33zusjuice »

radiowave wrote: post #47

Tracking servers is one thing, but tracking the dependency relationships among them is likely at least as important.

I’m really glad we realized that before disaster struck. We have a project in-progress to do exactly this. It’d even better if SWE wrote ADRs (or whatever) that document all this stuff up front, but … well, there are only so many battles anyone can fight, right?

Re: We might want to regularly keep track of how important each server is

#69

Post by fishtacos »

justsomehnguy wrote: post #62
fishtacos wrote: post #56

Earlier quoted context omitted.

>>The IT department decided they were done with their pets, moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array. There was a disk failure, but that’s ok, RAID-5 can handle that. Precisely why, when I was charged with setting up a 100 TB array for a law firm client at previous job, I went for RAID-6, even though it came with a tremendous write speed hit. It was mostly archived data that ne…

> even though it came with a tremendous write speed hit Only on a writes < stripe. If your writes are bigger then you can have way more speed than RAID10 on the same set, limited only by the RAID controller CPU.

Due to network limitations and contract budgeting, I never got the chance to upgrade them to 10 Gb, but can confirm I could hit 1000 Mbps (100+ MB/s) on certain files on RAID-6. It sadly averaged out to about 55-60 MB/s writes (HDD array, Buffalo), which again, for this use case was acceptable, but below expectations. I didn't buy the unit, I didn't design the architecture it was going into, merely a cog in the support machinery.

Re: We might want to regularly keep track of how important each server is

#70

Post by spydum »

j33zusjuice wrote: post #64
ethbr1 wrote: post #35

Earlier quoted context omitted.

> provide your apps criticality ratings based on how important they are to running the business In a decentralized, self-service model, you can add "deal with convincing a stakeholder their app is anything less than most-critical." Although it usually works itself out if higher-criticality imposes ongoing time commitments on them as well (aka stick).

That seems like a poorly run company. Idk. Maybe we’ve worked in very different environments, but devs have almost always been aware of the criticality of the app, so convincing people wasn’t hard. In most places, the answer hinges on “is it customer facing?” and/or “does it break a core part of our business?” If the answer is no to both, it’s not critical, and everyone understands that. There’s always some weird out…

agree. ethbr1 is 100% right about this being a problem; if politics is driving your criticality rating, it's probably being done wrong. it should be as simple as your statement, being mindful of some of those downstream systems that aren't always obviously critical (until they are unavailable for $time)

edit: whoops, maybe I read the meaning backward, but both issues exist!

Post reply on HN