Live data from Hacker News

Old school Linux administration – my next homelab generation

scholz.ruhr

91–100 of 115 posts

Re: Old school Linux administration – my next homelab generation

#91

Can someome explain "Pets" and "Cattle"? Not seen this useage before, think I get it but want to be sure. Pets are servers for love and fun, Cattle are servers exploited for loveless meat & milk?

This post has the slide that seared into my memory when I first heard the analogy in 2012-2013 or so:

https://www.engineyard.com/blog/pets-vs-cattle/

Direct link to slide screencap here:

https://www.engineyard.com/wp-content/uploads/2022/02/vCloud...

Re: Old school Linux administration – my next homelab generation

#92
post #75

As a person who manages a big fleet of servers containing both pets and cattle, the upkeep of the pets is nowhere near the cloud-lovers drum-up. A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). Also, not installing a n+3 Kubernetes cluster with an external storage backend reduces overheads and n…

> A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). For my earlier home setups, this was actually part of the problem! My servers and apps were so zero-touch, that by the time I needed to do anything, I'd forgotten everything about them! Now, I could have meticulously documented everything, but..…

If you make a habit to perform all changes via CI (ansible/chef/salt, maybe terraform if applicable) you get this for free too. See your playbooks as "dockerfiles".

Re: Old school Linux administration – my next homelab generation

#93
post #82

I can see the appeal. I've been working on containerizing apps at work. There's a lot of complication there. But I also really really want to be able to run updates whenever I want. Not just during a maintenance window when it's ok for services to be offline. And containers are the most likely route to get me there. At home, well, I enjoy my work, so I containerize all my home stuff as well. And I have a plan to swit…

if you have more than one pod running at the same time, otherwise will be downtime if the container spawns somewhere else or am i missing something?

I assume you mean at work?

Yes, we'd have more than one instance of the app running. Ideally 3 or 5 instances. And they'd be load balanced. So I could update the app one instance at a time with no downtime. At least that is the goal.

Re: Old school Linux administration – my next homelab generation

#94
post #85
post #82

I can see the appeal. I've been working on containerizing apps at work. There's a lot of complication there. But I also really really want to be able to run updates whenever I want. Not just during a maintenance window when it's ok for services to be offline. And containers are the most likely route to get me there. At home, well, I enjoy my work, so I containerize all my home stuff as well. And I have a plan to swit…

Docker especially with Portainer is pretty serviceable for an at home setup. Once the line has been crossed with "I should be backing this up", "I should have a dev copy vs the one i'm using", the existing setups can port quite well/into something like ProxMox to keep everything in a homelab setup running relatively like an appliance with a minimum of manual system administration, maintenance and upgrading.

If Docker Swarm was a bit better, I'd have all my test instances at work running a Docker Swarm, Portainer, and Traefik stack. Unfortunately Swarm has some quirks that make running stateful apps a bit difficult.

For home use, I've been experimenting with Portainer. It seems to work well for apps I'm not developing and am just running.

Re: Old school Linux administration – my next homelab generation

#95

As a person who manages a big fleet of servers containing both pets and cattle, the upkeep of the pets is nowhere near the cloud-lovers drum-up. A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). Also, not installing a n+3 Kubernetes cluster with an external storage backend reduces overheads and n…

Where I work maintaining server is a big pain in the ass because of the ever growing security and regulatory compliance requirements. The rules makes patching things an exercise in red tape frustration. Last year when we’ve been asked to redeploy our servers because the OS version was being added to a nope list we decided to migrate to Kubernetes so we don’t have to manage servers anymore (the nodes are a control pla…

>Now we just build our stuff using wathever curated image is there that can be used and ship it without worrying about OS patches.

So you basically replaced the "we regularly have to update the OS" with "we regularly have to pull the newest image". It's possible because I have a Linux admin background but I don't see that big of a difference here. Oh and just using whatever curated image is there doesn't necessarily provide you with a secure environment. [0]

[0] https://snyk.io/blog/top-ten-most-popular-docker-images-each...

Re: Old school Linux administration – my next homelab generation

#96

HA is overrated, i'd much rather go for a low mean time to repair. Backups, reinstall, ansible playbook is my way to go if hardware fails, which is quite rare to be honest. HA goes beyond hardware in terms of "electricity, internet connection, storage, location etc.., IMO people quite often underestimate what it means to have real HA --> second location with identical setup to shift workload or even have active-activ…

> HA is overrated, i'd much rather go for a low mean time to repair.

Those are almost entirely independent domains

the only point where they connect is that

> full down to back online if hardware is not damaged is less than 15 min.

automation makes both easier to deal with.

"Mean time to repair" is all fine if you don't have to drive 30 minutes to datacenter to swap out stuff.

Even if its cloudy cloud and you have backups, restores can take a lot of time.

Also you will want HA when it's your internet gateway to die

There are also levels of implementation. Making fully redundant set of loadbalancers is relatively easy. But any stateful app is much, much harder. In case of applications active-passive type of setups are also much easier than full on active active redundancy, especially if app wasn't written for it.

Re: Old school Linux administration – my next homelab generation

#97

As a person who manages a big fleet of servers containing both pets and cattle, the upkeep of the pets is nowhere near the cloud-lovers drum-up. A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). Also, not installing a n+3 Kubernetes cluster with an external storage backend reduces overheads and n…

Our biggest project (with a bunch of different infrastructure stuff) is also one with the least time spent per instance). We have months where no ops even logged on their VMs.

Our highest maintenance stuff is entirely "the dev fucked up"/"the dev is clueless". Stuff like server deciding to dump gigabytes of logs per minute or run out of disk space coz of some code error. Only difference compared to k8s is that the dead app would signal dev first, not actual monitoring we had.

And honorable mention for WordPress "developers" that make the first place in "most issues per server" every single year. And we have very few WP compared to everything else. Stuff like "dev uploaded some plugins, got instantly hacked (partially, we force using outgoing proxy and that stopped full compromise), we reverted it, he uploaded same stuff and got hacked again". So, nothing to do with actual servers.

Stuff like setting up automation to deploy ceph cluster took some time... once. Now it takes nothing. We even managed to plug it to k8s cluster.

> A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel).

could even not be touched if you set up automated updates and reboots, the red tape is usually a problem, like dumbly written support deals needing notification for every restart even if service is in HA. Or some dumbo selling a service with SLA but only on single machine...

Re: Old school Linux administration – my next homelab generation

#98
post #75

Earlier quoted context omitted.

> A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). For my earlier home setups, this was actually part of the problem! My servers and apps were so zero-touch, that by the time I needed to do anything, I'd forgotten everything about them! Now, I could have meticulously documented everything, but..…

We have found out that, while some applications are installed much easier with Docker, operating them becomes much more harder on the long run. NextCloud is a prime example. Adding some extensions (apps) on NextCloud becomes almost impossible when installed via Docker. We have two teams with their own NextCloud installations. One is installed on bare metal, and other one is a Docker setup. The bare metal one is much…

At that point I believe that if software wasn't written Docker-first it will just be plainly shit to manage as Docker container.

Docker containers more often than not are a crutch for app that have absolute shitshow install process to just bake that process in Dockerfile instead of making it more sensible in the first place.

Re: Old school Linux administration – my next homelab generation

#99

Earlier quoted context omitted.

Where I work maintaining server is a big pain in the ass because of the ever growing security and regulatory compliance requirements. The rules makes patching things an exercise in red tape frustration. Last year when we’ve been asked to redeploy our servers because the OS version was being added to a nope list we decided to migrate to Kubernetes so we don’t have to manage servers anymore (the nodes are a control pla…

>Now we just build our stuff using wathever curated image is there that can be used and ship it without worrying about OS patches. So you basically replaced the "we regularly have to update the OS" with "we regularly have to pull the newest image". It's possible because I have a Linux admin background but I don't see that big of a difference here. Oh and just using whatever curated image is there doesn't necessarily…

Hi. Sorry if I didn’t made this clear. By curated I meant images that were security scanned and approved before being published on the company internal registry. Those images shouldn’t have the issues listed on the link you posted.

Re: Old school Linux administration – my next homelab generation

#100
post #98

Earlier quoted context omitted.

We have found out that, while some applications are installed much easier with Docker, operating them becomes much more harder on the long run. NextCloud is a prime example. Adding some extensions (apps) on NextCloud becomes almost impossible when installed via Docker. We have two teams with their own NextCloud installations. One is installed on bare metal, and other one is a Docker setup. The bare metal one is much…

At that point I believe that if software wasn't written Docker-first it will just be plainly shit to manage as Docker container. Docker containers more often than not are a crutch for app that have absolute shitshow install process to just bake that process in Dockerfile instead of making it more sensible in the first place.

For a home lab/home prod, time to install is not necessarily the worst thing to optimise for. For example, I’m not going to spend several hours each on 8-10 different alternative applications, just to install them so I can get a feel for which one I want to keep.

Docker is an amazing timesaver, and probably the only reason why I run anything more ambitious than classic LAMP + NFS on my home lab.

Post reply on HN