Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We might want to regularly keep track of how important each server is

utcc.utoronto.ca

Post reply on HN
Print view View original

121–130 of 147 posts

Re: We might want to regularly keep track of how important each server is

#121

Post by rezonant »

SonOfLilit wrote: post #118
_dan wrote: post #111

Earlier quoted context omitted.

Honestly that was pretty much the situation. In our defence, he offered. It had hit hour 6 of both the primary and the backup aircon being down, on a very hot day - everyone was way beyond blame and the NOC staff were basically up for any creative solution they could find.

Wait, you didn't mean "he repositioned it a couple levels down on the rack to make some room above so he could unscrew the cover and crack it a bit open it like a grand piano"?

I suppose the case could've been locked-- if so you get a bonus penetration test out of it.

Re: We might want to regularly keep track of how important each server is

#122

Post by cogman10 »

People like to rag on Kubernetes for it's complexity, but this is the exact sort of scenario where k8s really does shine.

The answer to "which machines are important" is "only enough to provide the resources to run everything". You can kill whatever nodes you like just so long as there is enough of them to keep the cluster health and k8s will simply migrate workloads where they need to be.

That being said, storage is still an issue. Perhaps NAS is the one place where you might mark "these are the important machines".

Re: We might want to regularly keep track of how important each server is

#123

Post by rjbwork »

cogman10 wrote: post #122

People like to rag on Kubernetes for it's complexity, but this is the exact sort of scenario where k8s really does shine. The answer to "which machines are important" is "only enough to provide the resources to run everything". You can kill whatever nodes you like just so long as there is enough of them to keep the cluster health and k8s will simply migrate workloads where they need to be. That being said, storage is…

And what if you only have 33% of your nominal cluster capacity available because the AC goes out in your server room?

Now which containers should your cluster stop running?

You haven't actually solved anything with this, you've just changed the abstraction layer at which you need to make decisions. Probably an improvement, but does not obviate some kind of load shedding heuristic.

Re: We might want to regularly keep track of how important each server is

#124

Post by zetsurin »

rjbwork wrote: post #123
cogman10 wrote: post #122

People like to rag on Kubernetes for it's complexity, but this is the exact sort of scenario where k8s really does shine. The answer to "which machines are important" is "only enough to provide the resources to run everything". You can kill whatever nodes you like just so long as there is enough of them to keep the cluster health and k8s will simply migrate workloads where they need to be. That being said, storage is…

And what if you only have 33% of your nominal cluster capacity available because the AC goes out in your server room? Now which containers should your cluster stop running? You haven't actually solved anything with this, you've just changed the abstraction layer at which you need to make decisions. Probably an improvement, but does not obviate some kind of load shedding heuristic.

You have: you've got clearly labelled workloads you can decide which ones are important and not, and easily turn down the ones that aren't important.

Re: We might want to regularly keep track of how important each server is

#125

Post by jodrellblank »

p_l wrote: post #77
bluGill wrote: post #43

Earlier quoted context omitted.

Cattle are often interchangeable. You cull any that catch a disease (in some cases the USDA will cull the entire herd if just one catches something - bio security is a big deal) In the case of drought you pick a bunch to get rid of - based on market prices (If everyone else is you will try to keep yours because the market is collapsing - but this means managing feed and thus may mean culling more of the herd latter.)…

Honestly, big issue with the cattle metaphor is that the individual services you run on servers are very much often not interchangeable. A DNS service is not NTP is not mail gateway is not application load balancer is not database etc etc etc At best, multiple replicas of those are cattle. And while you can treat the servers underlying them as interchangeable, that doesn't change the fact the services you run on them…

I think it's a way of thinking about things, rather than a true/false description. e.g. VMware virtual hosts make good cattle - in some setups I have worked on the hosts are interchangeable, move virtual machines between them without downtime. In others the hosts have different storage access, different connectivity and it matters which combination of hosts are online/offline together, and which VMs need the special connectivity.

The regular setups are easier to understand, nicer to work on. The irregular ones are a trip hazard, they need careful setup, more careful maintenance, more detailed documentation, more aware monitoring. But there's probably ways they could be made regular, if the unique connectivity was moved out to a separate 'module' e.g. at the switch layer, or if the storage had been planned differently, sometimes with more cost, sometimes just with different design up-front.

Along these lines, yes DNS is not NTP but you could have a 'cattle' template Linux server which can run your DNS or NTP or SMTP relay which can be script deployed, and then standard DNS/NTP/SMTP containers deployed on top. Or you could build a new Linux server by hand and deploy a new service layer by hand, every time, each one slightly different depending how rushed you are and what verison of installers are conveniently available and whether the same person does the work following the latest runbook or an outdated one or going from memory. You could deploy a template OpnSense VM which can front DNS or NTP or SMTP instead of having to manually login to a GUI firewall interface and add rules for the new service by hand.

'Cattle not pets' is a call to standardise, regularise, modularize, template, script, automate; to move towards those ways of doing things. Servers are now software which can be copypasted in a way they weren't 10-30 years ago, at least in my non-FAANG world. To me it doesn't mean every server has to mean nothing to you, or every server is interchangeable, it means consider if thinking that wasy can help.

Re: We might want to regularly keep track of how important each server is

#126

Post by jodrellblank »

_heimdall wrote: post #25
cowsandmilk wrote: post #5

It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.

I'm pretty sure anyone in the industry that draws this distinction between cattle and pets has never worked with cattle and only knows of general ideas about the industrial cattle business.

https://largest.org/geography/largest-cattle-ranches-in-the-... says Deseret Ranches has 44,000 cattle.

Google tells me beef cattle are slaughtered after 2 years. Split 2 years among 44,000 cattle and you get to spend at most 24 minutes with each one, if you dedicate 2 years of your life to nothing else but that, not even sleep, travel, eating. If you let them live their natural life expectancy of 20 years, you get 240 minutes with each cow - two hours in its 20 year life.

"I care about my cattle", yes, I don't think "cattle" is supposed to mean "stop caring about the things you work on". "I know each and every one as well as the family dog I've had for ten years", no. That's not possible. You raise them industrially and kill them for profit/food, that's a different dynamic than with Spot.

Re: We might want to regularly keep track of how important each server is

#127

Post by bcrl »

outofpaper wrote: post #52
spydum wrote: post #11

Asset management is definitely a thing. Tag your environments, tag your apps, and provide your apps criticality ratings based on how important they are to running the business. Then it's a matter of a query to know which servers can be shut, and which absolutely must remain.

In moments of crisis, immediate measures like physical tagging can be crucial. Yet, a broader challenge looms: our dependency on air conditioning. In Toronto's winter, the missed opportunity to design buildings that work with the climate, rather than defaulting to a universal AC solution, underscores the need for thoughtful asset management tailored to specific environments.

Several data centres in Toronto (including the massive facilities at 151 Front Street West where most of the internet for the province passes through) make use of the deep lake cooling loop that takes water pumped in from Lake Ontario to cool equipment before moving on to other uses. Water is pumped in from a sufficient depth such that the temperature is fairly constant year round.

Re: We might want to regularly keep track of how important each server is

#128

Post by cogman10 »

rjbwork wrote: post #123
cogman10 wrote: post #122

People like to rag on Kubernetes for it's complexity, but this is the exact sort of scenario where k8s really does shine. The answer to "which machines are important" is "only enough to provide the resources to run everything". You can kill whatever nodes you like just so long as there is enough of them to keep the cluster health and k8s will simply migrate workloads where they need to be. That being said, storage is…

And what if you only have 33% of your nominal cluster capacity available because the AC goes out in your server room? Now which containers should your cluster stop running? You haven't actually solved anything with this, you've just changed the abstraction layer at which you need to make decisions. Probably an improvement, but does not obviate some kind of load shedding heuristic.

But that's not what this was about. It was about "hey, which of these boxes had the ldap running on it? We need to make sure that gets shifted somewhere else!"

K8S let's you say "ok, we don't have enough capacity to run everything so let's shut down the Bitcoin deployment to free up capacity".

There's no leg work or bookkeeping to figure out what was running where, instead it's "what do I need to run and what can I shut down or tune down". All from the comfort of the room with AC.

And if you're really cleaver, you went ahead and gave system critical pods elevated property. [1]

[1] https://kubernetes.io/docs/concepts/scheduling-eviction/pod-...

Re: We might want to regularly keep track of how important each server is

#129

Post by _heimdall »

jodrellblank wrote: post #126
_heimdall wrote: post #25

Earlier quoted context omitted.

I'm pretty sure anyone in the industry that draws this distinction between cattle and pets has never worked with cattle and only knows of general ideas about the industrial cattle business.

https://largest.org/geography/largest-cattle-ranches-in-the-... says Deseret Ranches has 44,000 cattle. Google tells me beef cattle are slaughtered after 2 years. Split 2 years among 44,000 cattle and you get to spend at most 24 minutes with each one, if you dedicate 2 years of your life to nothing else but that, not even sleep, travel, eating. If you let them live their natural life expectancy of 20 years, you get 2…

I believe this is just a great example of why cattle shouldn't be raised in such high volume industrial processes.

Have you ever been around cattle? Or helped them calve? Or slaughtered one for meat?

I know every one of my animals and understand the herd dynamics, from who the lead cow is to who is the asshole that is the one often starting fights and annoying the others.

We shouldn't be throwing so many animals into such a controlled and confined system that they are reduced to numbers on a spreadsheet. We shouldn't raise an animal for slaughter after dedicating at most 24 minutes to them.

Re: We might want to regularly keep track of how important each server is

#130

Post by RHSman2 »

_dan wrote: post #83

Similar thing (catastrophic aircon failure due to a flood in a crap colocated DC) happened to us too before we shifted to AWS. Photos from the colo were pretty bizarre - fans balanced on random boxes, makeshift aircon ducting made of cardboard and tape, and some dude flailing an open fire door back and forth all day to get a little bit of fresh air in. Bizarre to see in 2010-ish with multi million dollar customers. W…

I wanna watch this movie
Post reply on HN