Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We might want to regularly keep track of how important each server is

utcc.utoronto.ca

Post reply on HN
Print view View original

141–147 of 147 posts

Re: We might want to regularly keep track of how important each server is

#141

Post by thelastparadise »

naniwaduni wrote: post #80
p_l wrote: post #77

Earlier quoted context omitted.

Honestly, big issue with the cattle metaphor is that the individual services you run on servers are very much often not interchangeable. A DNS service is not NTP is not mail gateway is not application load balancer is not database etc etc etc At best, multiple replicas of those are cattle. And while you can treat the servers underlying them as interchangeable, that doesn't change the fact the services you run on them…

Cattle often aren't interchangeable too. Not gonna have a great time milking the bulls.

> Not gonna have a great time milking the bulls.

Depends on who's doin' the milkin'.

Re: We might want to regularly keep track of how important each server is

#142

Post by gorjusborg »

tonyarkles wrote: post #76
jodrellblank wrote: post #28

Earlier quoted context omitted.

At the risk of no true Scotsman, that doesn’t sound like “cattle not pets“; when the cattle are sent to the slaughterhouse there isn’t any blast radius, there’s just more cattle taking over. You explicitly don’t have to replace them with exact clones of the original cattle from tape very slowly, you spin up a herd of more cattle in moments.

> you spin up a herd of more cattle in moments Where does the decade of data they've been collecting come from when you "spin up a new herd"?

In the 'no pets' model, think of your compute nodes as CPU/memory/OS only (storage is accessible via network) and it starts to make sense.

Re: We might want to regularly keep track of how important each server is

#143

Post by marcus0x62 »

Scoundreller wrote: post #134
marcus0x62 wrote: post #23

Years ago during a week-long power outage, a telephone central office where we had some equipment suffered a generator failure. The telephone company had a backup plan (they kept one generator on a trailer in the city for such a contingency,) and they had battery capacity[0] for their critical equipment to last until the generator was hooked up. They did have to load shed, though: they just turned off the AC inverter…

> had some equipment suffered a generator failure. The telephone company had a backup plan (they kept one generator on a trailer in the city for such a contingency Well that’s more forward thinking than AT&T’s Nashville CO when it got bombed. They just depended on natgas grid for their generators if grid electric went out. They underestimated the correlation between energy grids. When natgas got cutoff and the UPSs d…

To be fair, the CO I described[0] is a local switching center. The Nashville “CO” was the area long lines/tandem/long distance site. Now, I’m not defending the lack of a backup plan, but tandems tend to be far larger than a typical local/class 5 CO.

0 - which like the Nashville long lines building also happens to be a pre-divestiture AT&T site, although at the time it was owned by BellSouth. It is also in Tennessee, although in Memphis, not Nashville.

Re: We might want to regularly keep track of how important each server is

#144

Post by jodrellblank »

_heimdall wrote: post #129
jodrellblank wrote: post #126

Earlier quoted context omitted.

https://largest.org/geography/largest-cattle-ranches-in-the-... says Deseret Ranches has 44,000 cattle. Google tells me beef cattle are slaughtered after 2 years. Split 2 years among 44,000 cattle and you get to spend at most 24 minutes with each one, if you dedicate 2 years of your life to nothing else but that, not even sleep, travel, eating. If you let them live their natural life expectancy of 20 years, you get 2…

I believe this is just a great example of why cattle shouldn't be raised in such high volume industrial processes. Have you ever been around cattle? Or helped them calve? Or slaughtered one for meat? I know every one of my animals and understand the herd dynamics, from who the lead cow is to who is the asshole that is the one often starting fights and annoying the others. We shouldn't be throwing so many animals into…

"cattle not pets" is about computer servers, it's not about ethical treatment of living creatures. Whether or not living animals should be numbers on a spreadsheet, servers can be without ethical concerns[1]. "I treat my cattle like pets" is respectable, but not relevant - unless you are also saying "therefore you should treat your servers like pets", which you would need to expand on.

> "I know every one of my animals"

And you are still dodging the part where there is a point at which you could not do that if you had more and more animals. You can choose not to have more animals, but a company cannot avoid having more servers if they are to digitise more workflows, serve more customers, offer more complex services, have higher reliability failovers, DR systems, test systems, developer systems, monitoring and reporting and logging and analysis of all of the above - and again the analogy is not saying "the way companies treat cows is the right way to treat cows", it's saying "the ruthless, ROI-focused, commodity way companies actually do treat cows is a more scalable and profitable way to think about the computer servers/services that you currently think about like family pets".

[1] at least, first order ones in a pre-AI age. Energy use, pollution, habitat damage, etc. are another matter.

Re: We might want to regularly keep track of how important each server is

#145

Post by rjbwork »

cogman10 wrote: post #128
rjbwork wrote: post #123

Earlier quoted context omitted.

And what if you only have 33% of your nominal cluster capacity available because the AC goes out in your server room? Now which containers should your cluster stop running? You haven't actually solved anything with this, you've just changed the abstraction layer at which you need to make decisions. Probably an improvement, but does not obviate some kind of load shedding heuristic.

But that's not what this was about. It was about "hey, which of these boxes had the ldap running on it? We need to make sure that gets shifted somewhere else!" K8S let's you say "ok, we don't have enough capacity to run everything so let's shut down the Bitcoin deployment to free up capacity". There's no leg work or bookkeeping to figure out what was running where, instead it's "what do I need to run and what can I s…

I see. I guess there's two components - prioritization for loadshedding, and knowing how to actually turn things off to shed the load. You're saying k8s makes the latter very easy, and it was actually about that latter exercise of mapping workload to infra, not enumeration and prioritization of services.

Re: We might want to regularly keep track of how important each server is

#146

Post by kiitos »

mgaunard wrote: post #97

If machines aren't important, why do you have them at all? Because it's an university and we don't care about justifying costs?

Machines are incidental implementation details of the applications which run on them. The applications matter, the machines don't.

Re: We might want to regularly keep track of how important each server is

#147

Post by monkeywork »

gosub100 wrote: post #102
monkeywork wrote: post #72

Earlier quoted context omitted.

Toronto's climate and winters is dramatically changing, the universal AC solution is almost mandatory due to the climate not being as cold in this area as it once was.

do you have a source for that? my source[1] appears the average temp hasn't changed much in the past quarter century: https://toronto.weatherstats.ca/metrics/temperature.html

From the site you cited:

* Feb 1 2014 MAX 5.5 °C HR MEAN -7.8 °C MEAN M/M -8.3 °C MIN -21.3 °C

* Feb 1 2024 MAX 15.7 °C HR MEAN 1.3 °C MEAN M/M 1.3 °C MIN -8.2 °C

* Jan 1 2014 MAX 7.5 °C HR MEAN -8.3 °C MEAN M/M -8.6 °C MIN -24.2 °C

* Jan 1 2024 MAX 6.5 °C HR MEAN -2.3 °C MEAN M/M -2.1 °C MIN -15.5 °C

* Dec 1 2013 MAX 15.6 °C HR MEAN -4.0 °C MEAN M/M -4.2 °C MIN -17.8 °C

* Dec 1 2023 MAX 13.1 °C HR MEAN 2.7 °C MEAN M/M 2.7 °C MIN -4.9 °C

Looking at the MEANs shows the real story - as well as seeing that the Mins are getting nowhere close to what they used to.

Also I live here, snow volumes have been low enough that I no longer need a snowblower. I used to build a backyard rink and haven't been able to properly last couple of years because the weather is too mild and I can't enough solid time that it'll be frozen. Public outdoor rinks (that aren't artificially chilled) suffer the same fate and are rarely if ever available.

Even in Ottawa where at this time a decade ago people would have been skating on the Canal for weeks by now - it's still not frozen over and open for the public.

Post reply on HN